final report rocky mountain power wyoming residential home energy

Rocky Mountain Power Wyoming Residential Home Energy Savings Evaluation

Final Report

October 14, 2011

Prepared for:

Rocky Mountain Power

Prepared by:

The Cadmus Group, Inc. / Energy Services 720 SW Washington Street, Suite 400 Portland, OR 97205

503.228.2992

Principal Investigators:

Katie Parkinson Hossein Haeri

Jeana Swedenburg

Rocky Mountain Power Wyoming HES 2009–2010 Final Report October 14, 2011

The Cadmus Group, Inc. / Energy Services i

Table of Contents

Glossary of Terms ................................................................................................................1

Executive Summary .............................................................................................3 Overview of Evaluation Activities .......................................................................................3

Key Findings ........................................................................................................................4

Summary and Recommendations ........................................................................................6

Introduction ..........................................................................................................8 Program Description ............................................................................................................8

Evaluated Gross and Net Savings Methodology .................................................................9

Impact Evaluation .............................................................................................. 17 Lighting ..............................................................................................................................17

Appliances, HVAC, and Weatherization ...........................................................................37

Process Evaluation Findings ........................................................................... 56 Program Implementation and Delivery ..............................................................................56

Marketing ...........................................................................................................................62

Quality Assurance ..............................................................................................................67

Customer Response ............................................................................................................68

Communication ..................................................................................................................71

Document Review ..............................................................................................................72

Summary and Recommendations ......................................................................................73

Cost-Effectiveness ............................................................................................ 78

Appendices ........................................................................................................ 82


The Cadmus Group, Inc. / Energy Services 1

Glossary of Terms Analysis of Covariance (ANCOVA)

An ANCOVA model is an ANOVA model with a continuous variable added.

Analysis of Variance (ANOVA)

An ANOVA model explains the variation in the independent variable based on a series of characteristics (expressed as binary variables with values of zero or one).

Evaluated Gross Savings

The total savings found by the evaluator attributable to the program before adjusting for freeridership or spillover. It is most often calculated at the measure level.

Evaluated Net Savings

The savings “net” of what would have occurred in the absence of the program.

Freeridership

Participants that would have adopted the measures offered under the program in absence of the program. This is often expressed as the freeridership rate, or fraction of evaluated gross savings that can be classified as freeridership.

In-Service Rate (ISR)

Sometimes also called the “installation rate,” is the fraction of incented measures that were actually installed and operating.

Net-to-Gross (NTG)

Is the ratio of net savings to gross savings. This ratio is calculated as:

1

P-Value

A p-value indicates the probability that a given estimate or finding is due to chance. A p-value less than 0.10 indicates that we can say with 90 percent confidence that the finding is valid.

Realization Rate Realization rate is calculated by comparing evaluated net savings to the reported gross savings.

R2 (and adjusted R2)

The R2 represents the portion of variance in a regression equation explained by a model and takes values between zero and one. The adjusted R2 factors in the degrees of freedom in the regression model, which is a function of the number of independent variable included in the model.



Spillover

Spillover savings are reductions in energy consumption caused by the presence of the energy efficiency program, but which the program does not directly fund. Like freeridership, this is often expressed as a fraction of evaluated gross savings (or spillover rate).

T-Test

Is used to indicate the statistical significance of the observed impacts. A t-test with a p-value less than 0.10 indicates that we can say with 90 percent confidence that the two estimates are different.

Variance Inflation Factor (VIF)

The VIF is a measure of correlation between explanatory variables. A VIF of one indicates no correlation, while higher VIFs indicate greater correlation.



Executive Summary

Rocky Mountain Power offers the Home Energy Savings (HES) Program in Wyoming, Utah, Idaho, Washington, and Northern California. In 2009, Rocky Mountain Power first offered the HES Program in Wyoming. The HES Program provides residential customers with incentives to help facilitate the purchase of energy-efficient products and services through upstream (manufacturer and retailer) and downstream (customer) incentive mechanisms. During the 2009 and 2010 program years, Rocky Mountain Power reported total gross savings acquired through the programs of 8,060,948 kWh. Over the first two years of the program, participation exceeded 26,000 customers. The largest of the Wyoming Rocky Mountain Power residential programs, the HES Program contributed 66 percent of the Wyoming program’s savings and 98 percent of residential portfolio-wide participants.

The HES Program offers energy efficiency measures in four categories:

Lighting: Upstream incentives for manufacturers to reduce retail prices on compact florescent lamps (CFLs), and incentives to customers for light fixtures and ceiling fans.

Appliances: Customer incentives for clothes washers, dishwashers, refrigerators, and high-efficiency electric storage water heaters.

Heating, ventilation, and air conditioning (HVAC): Customer incentives for high-efficiency heating and cooling equipment and services, duct sealing, and evaporative cooling equipment.

Windows and insulation: Customer incentives for attic, wall, and floor insulation, and high-efficiency windows.

Rocky Mountain Power contracted with The Cadmus Group, Inc., (Cadmus) to conduct process and impact evaluations of the Home Energy Savings (HES) Program in Wyoming for program years 2009 and 2010. The impact evaluation assessed energy impacts and program cost-effectiveness. The process evaluation assessed program delivery and efficacy, bottlenecks, barriers, best practices, and opportunities for improvements. This document presents the results of these evaluations.

Overview of Evaluation Activities The evaluation of the HES Program consisted of primary and secondary data collection activities, informing the impact and process evaluation components. The impact evaluation estimated two key components: gross savings and the net-to-gross ratio (NTG). The gross savings calculations included adjustments for the installation rate and engineering inputs. NTG—the combination of freeridership and spillover—discounted savings from units that would have been installed in the program’s absence, and credited the program for unaccounted savings achieved through the program’s influence.

The process evaluation investigated topics such as participant satisfaction; implementation and delivery processes; marketing methods; quality assurance; and other qualitative issues.



Key Findings Launched in 2009, the HES Program provides incentives for over 20 energy-saving measures typically found in residential households. Cadmus’ evaluation focused on the top 10 measures which collectively contributed to over 98 percent of the HES Program savings. CFLs, accounted for 89 percent of the total HES Program savings and as a result were a primary focus of the evaluations.

Key Impact Findings Key impact evaluation findings include the following:

Appliances: Incented appliances experienced a 100 percent installation rate. Evaluated gross savings realization rates ranged from 56 percent (refrigerators) to 187 percent (clothes washers). Savings realization rates above 100 percent were results of changes in assumptions regarding electricity usage and fuel type saturation. The HES Program non-lighting measures had a 76 percent NTG ratio (Table 36, Table 44).

HVAC: Incented HVAC equipment experienced a 100 percent installation rate. Evaluated gross savings realization rates ranged from 98 percent (CAC TXV and Install) to 510 percent (HVAC tune up) due to changes in usage, home size, and fuel type saturation assumptions input into the energy modeling software. The HES Program non-lighting measures had a 76 percent NTG ratio estimate (Table 39, Table 44).

Windows and Insulation: Incented windows and insulation experienced a 100 percent installation rate. Evaluated gross savings realization rates ranged from 111 percent (attic insulation) to 312 percent (floor insulation) due to changes in assumptions regarding home size and fuel type saturation used in energy modeling. The HES non-lighting measures had a 76 percent NTG ratio estimate (Table 39, Table 44).

Lighting: The incented CFL estimated installation rate was 67 percent based on reported behavior from surveys on storage and removals practices. The HES lighting component experienced a 76 percent evaluated gross savings realization rate and an 83 percent net-to-gross (NTG) ratio (Table 1).

Table 1, below, summarizes evaluated savings estimates in comparison to reported values.

Table 1. 2009 and 2010 CFL Lighting HES Program Savings*

Measure Group Units

Reported Gross

Savings (kWh)

Evaluated Gross

Savings (kWh)

Gross Realization

Rate

Evaluated Net Savings

(kWh)

Precision at 90%

Confidence** Upstream Lighting 214,364 7,170,603 5,460,861 76% 4,532,515 ±8.5% Non-Lighting 5,116*** 894,012 1,411,403 158% 1,072,666 ±9.7%

Totals 8,064,615 6,872,264 85% 5,605,181 *Throughout the report, totals in tables may not add up exactly due to rounding. ** Appendix B provides a description of methodology for how precision is calculated. ***Each incented insulation or window home is one unit



Key Process Evaluation Findings Key process evaluation findings include the following:

Wyoming-specific barriers, such as a dispersed population and prevalence of independent retailers, caused a longer than expected ramp-up time for the HES Program. However, increased field staff and increased marketing efforts began to overcome these obstacles.

Of the 254 in-territory lighting customers surveyed, 78 percent recognized the terms “compact fluorescent bulb” or “CFL”; 50 percent were familiar with light emitting diodes (LED) bulbs.

Over half (54 percent) of HES participants learned of the HES Program through a retailer, followed by bill inserts (13 percent) and print media (7 percent).

Customers did not associate their upstream lighting products with Rocky Mountain Power’s HES Program incentives. Although the bulk of HES Program savings accrued through the program’s lighting component, very few lighting customers knew Rocky Mountain Power’s HES Program provided the CFL discounts.

Participants rarely accessed HES Program information online: only 21 percent of insulation customers, 18 percent of appliance customers, and 8 percent of lighting customers had visited the HES Website.

HES Program satisfaction generally ran high. All surveyed customers reported high satisfaction levels regarding program incentives, purchased measures, and overall program experience. In fact, 91 percent of participating appliance/window participants were “very” or “somewhat” satisfied with their overall experience with the HES Program.

Cost-Effectiveness Results As shown in Table 2, the HES Program was cost-effectiveness across the evaluation period (2009 – 2010) for four of the five primary cost tests; total resource cost (TRC), the PacifiCorp total resource cost (PTRC), the participant cost test (PCT), and the utility cost test (UCT) perspectives. The program was not cost-effective from the rate impact measure (RIM) perspective, which measures the impact of programs on customer rates. Most programs do not pass the RIM test due to the adverse impact of lost revenue.

Table 2. 2009-2010 Evaluated Program Cost-Effectiveness Summary

Cost-Effectiveness Test

Levelized

Costs Benefits Net

Benefit / Cost Ratio $ / kWh Benefits

Total Resource + Conservation Adder (PTRC) $0.055 $2,131,526 $3,122,975 $991,449 1.47 Total Resource No Adder (TRC) $0.055 $2,131,526 $2,839,068 $707,542 1.33 Utility (UCT) $0.029 $1,112,945 $2,839,068 $1,726,123 2.55 Ratepayer Impact (RIM) $0.110 $4,278,165 $2,839,068 ($1,439,097) 0.66 Participant (PCT) $0.042 $1,819,961 $4,006,986 $2,187,025 2.20



Table 3 and Table 4 show the HES Program cost-effectiveness for the 2009 and 2010 program years, respectively.

Table 3. 2009 Evaluated Program Cost-Effectiveness Summary


Levelized

Costs Benefits Net


Total Resource + Conservation Adder (PTRC) $0.051 $799,725 $1,132,188 $332,463 1.42 Total Resource No Adder (TRC) $0.051 $799,725 $1,029,262 $229,536 1.29 Utility (UCT) $0.027 $425,066 $1,029,262 $604,195 2.42 Ratepayer Impact (RIM) $0.107 $1,685,598 $1,029,262 ($656,336) 0.61 Participant (PCT) $0.038 $653,916 $1,552,531 $898,614 2.37

Table 4. 2010 Evaluated Program Cost-Effectiveness Summary


Levelized

Costs Benefits Net


Total Resource + Conservation Adder (PTRC) $0.058 $1,430,354 $2,138,105 $707,751 1.49 Total Resource No Adder (TRC) $0.058 $1,430,354 $1,943,732 $513,378 1.36 Utility (UCT) $0.030 $738,782 $1,943,732 $1,204,950 2.63 Ratepayer Impact (RIM) $0.112 $2,784,417 $1,943,732 ($840,685) 0.70 Participant (PCT) $0.045 $1,252,332 $2,636,085 $1,383,753 2.10

Summary and Recommendations Following a slower-than-expected start in 2009, Rocky Mountain Power implemented several changes in 2010 to program operations, delivery structures, and marketing approaches which led to a significant improvement in participation and savings in year two of the program. Conclusions and recommendations have been drawn from process evaluation interviews, surveys, and other analysis conducted. While Cadmus’ process evaluation found several aspects of HES Program operations and delivery have improved, Cadmus believes the program may benefit from additional changes as the program matures and continues adapting to the Wyoming market. Based on the findings of this evaluation, Cadmus makes the following observations and recommendations:

Although the bulk of HES Program savings were attributable to the program’s lighting component, very few lighting customers knew Rocky Mountain Power provided the CFL discounts.

o Recommendation: Provide more in-store marketing materials focused on lighting, reinforcing the message that the utility discounts qualifying lighting products at the register.

Lighting retailers do not plan to educate customers about the 2007 Energy Independence and Security Act (EISA).

o Recommendation: Consider providing EISA educational materials to customers, specifying utility-supported, high-efficiency lighting options.

Trade allies serves as the program’s most valuable tool to increase program awareness.



o Recommendation: Continue to recruit new trade allies to broaden program awareness throughout the service territory.

The need for new equipment most often motivates customers to purchase qualified appliance measures.

o Recommendation: Offer an additional incentive to customers for early replacement of equipment.

All surveyed customers reported high satisfaction levels regarding program incentives, purchased measures, and overall experience with the program.

For more detail, please see Summary and Recommendations in the Process Evaluation Findings section of this report.



Introduction

Program Description In 2009, HES launched in Wyoming, and, according to program stakeholders, provided the first demand-side management (DSM) program offered in the state.

Portland Energy Conservation, Inc. (PECI) implemented the HES Program, which provided incentives to residential Rocky Mountain Power customers purchasing qualifying, high-efficiency appliances and weatherization measures. Prescriptive incentives offered included the following measures:

Clothes washers,

Dishwashers,

Water heaters,

Refrigerators,

Insulation,

Windows

Air conditioning units,

Duct sealing, and

Fluorescent fixtures and ceiling fans

To encourage dealers to promote energy-efficient equipment incentives and to properly size, install, and maintain equipment, Rocky Mountain Power also offered dealer incentives for qualifying central air conditioning, duct sealing, and heat pump measures bought or installed through the HES Program.

The HES Program included an upstream lighting component, applying incentives for eligible CFLs at the manufacturer level, and discounting for end-use customers purchasing high-efficiency lighting options.

Table 5 lists HES Program measures, corresponding baseline technologies and customer and HVAC dealer incentive amounts.



Table 5. HES Program Participation and Incentive by Measure

Measure Energy-Efficient Standards Unit Incentive

Levels Dealer Spiff

Clothes Washers Clothes Washer-Tier One (1.72–1.99 MEF) Units $50

Clothes Washer-Tier Two (2.0 + MEF) Units $75 Dishwasher ENERGY STAR Dishwasher Units $20 Electric Water Heater 40+ Gallons (EF 0.93 or higher) Units $50 Evaporative Cooler Permanently Installed Units $100 Refrigerator ENERGY STAR Refrigerator Units $20

Insulation Insulation: Attic (R-19 +) Square Feet Up to $0.35 Insulation: Floor (R-19 +) Square Feet $0.35

Insulation: Wall (R-11+ or fill cavity) Square Feet $0.35 Windows Windows Square Feet $1.00

Central Air Conditioner

CAC/HP Tune up Projects $100 $25 CAC (15 SEER) Units $250 $25

CAC TXV and Install Projects $50 $75 CAC TXV and Sizing Projects $50 $25

Duct Sealing Duct Sealing-Electric Projects $150 $25

Duct Sealing-Gas Projects $150 $25

Heat Pumps Heat Pump Conversion (8.2+ HSPF) Projects $350 $25

Heat Pump Upgrade (8.2+ HSPF) Projects $250 $25 Ceiling Fans Ceiling Fans Units $20 Fixtures Fixtures Units $20

CFLs CFLs-Spiral Lamps $0.75–$2.62

CFLs-Specialty Lamps $1.81–$2.78

Evaluated Gross and Net Savings Methodology In this report, we present two saving values: evaluated gross and net savings. To determine evaluated net savings, Cadmus applied four steps to reported gross program savings (Table 6). Reported gross savings are defined as the electricity savings (kWh) reported to Cadmus by Rocky Mountain Power.

Table 6. Impact Steps

Savings Estimate Step Action

Evaluated Gross Savings 1 Validate Accuracy of Data in Participant Database 2 Perform Engineering Review to Validate Saving Calculations 3 Adjust Gross Savings with Actual Installation Rate

Evaluated Net Savings 4 Apply Net-to-Gross Adjustments

Step one (verify participant database) included a review of the program tracking database to ensure participants and reported savings matched the 2009 and 2010 annual reports.

Step two (perform an engineering review) included a review of measure saving assumptions, equations, and inputs.

Step three (adjust gross savings with the installation rate) determined the number of measures program participants installed (and kept installed). This value was determined through a



telephone survey and the installation and persistence rate (referred to as in-service rate or ISR) in calculating evaluated gross savings.

Together, the first three steps make up the evaluated gross savings. The fourth step (applying net adjustments) was then applied to determine evaluated net savings.

Cadmus’ evaluation included the following data collection activities:

Management Staff Interviews: Cadmus conducted an in-depth interview PacifiCorp’s HES Program manager, in July 2011.

Program Partner Interviews: In August 2011, Cadmus interviewed two program management staff from PECI. PECI provided program implementation, incentive processing, and verification services for the HES Program.

Participant Telephone Survey (Appliances, Windows & HVAC): Cadmus conducted 298 interviews with customers receiving incentives from Rocky Mountain Power for clothes washers, refrigerators, dishwashers, windows, fixtures, heat pumps, ceiling fans and electric water heaters.

Participant Telephone Survey (Insulation): Cadmus conducted 44 interviews with customers receiving incentives from Rocky Mountain Power for installing wall, attic, or floor insulation.

Insulation Participant Verification Site Visit: Cadmus performed 69 site visits with insulation participants, verifying the amounts, quantities, and locations of installed insulation.

Participant Retailer/Contractor Survey: Cadmus conducted 30 interviews with trade allies (29 retailers and one contractor) supplying and installing discounted CFLs, appliances, windows, or insulation through HES. Many trade allies answered questions about multiple measures, resulting in 14 completed sections for lighting, 22 sections for appliances/windows, and seven sections for weatherization.

Nonparticipant Retailer/Contractor Survey: Cadmus conducted 30 interviews with retailers and contractors (22 retailers and 8 contractors) not participating in HES during 2009 and 2010 who supply and install CFLs, appliances, windows, and/or insulation.

In-territory Lighting Survey: Cadmus performed 254 interviews with Rocky Mountain Power customers purchasing CFLs and LEDs during the 2009 and 2010 program years.

Out-of-Territory Lighting Survey (Leakage): Cadmus performed 70 interviews with residents outside of Rocky Mountain Power’s territory, quantifying incentivized CFLs leaving the service territory.

Marketing Materials Review: Cadmus reviewed marketing and communications developed to promote participation and to educate target audiences about HES Program details. The review included specific marketing elements regarding: general look and feel; brand and message consistency; program accessibility; and online and interactive properties.



Data collection instruments can be found in Appendix A.

Sample Design and Data Collection Methods Cadmus developed samples, seeking to achieve precision of ±10 percent at the 90 percent confidence level for individual estimates at the measure level. Sample sizes were determined assuming a coefficient of variation (CV) of 0.5.1 For small population sizes, a finite population adjustment factor was applied. The final disposition of samples for various data collection activities is shown in Table 7.

Table 7. Sample Disposition for Various Data Collection Activity

Data Collection Activity Population Sample Achieved Surveys

Management Staff Interviews N/A N/A 1 Program Partner Interviews N/A N/A 2 Participant Telephone Survey (Appliances, Windows & HVAC) 4,843 368 298 Participant Telephone Survey (Insulation) 181 50 44 Insulation Participant Verification Site Visits 181 67 69 Participant Retailer/Contractor Survey 78 30 30 Nonparticipant Retailer/Contractor Survey N/A 30 30 In-Territory Lighting Survey N/A 250 254 Out-of-Territory Lighting Survey N/A 70 70

For nearly all data collection, Cadmus drew samples using either a simple or stratified random sampling.2

Management and Program Partner Interviews Cadmus interviewed a census of the Rocky Mountain Power HES Program staff and program partners provided by Rocky Mountain Power.

Participant Telephone Survey (Appliances, Windows, HVAC & Insulation) Cadmus stratified the participant telephone survey (appliances, windows, HVAC & insulation) by measure to ensure statistically meaningful results for each measure.

1 A measure of the dispersion of data points in a data series. 2 Simple random samples are drawn from the entire population, whereas stratified random samples are drawn

randomly from sub-populations (strata) and then weighted to extrapolate to the greater population.



Table 8. Participant Telephone Survey (Appliances, Windows, HVAC & Insulation) Sample Sizes

Measure Population Sample Achieved Surveys Clothes Washer 2,658 100 104 Refrigerator 925 70 70 Dishwasher 1,109 100 102 Windows 60 33 12 Fixtures 39 25 3 Heat Pump Upgrade 2 3 0 Ceiling Fans 20 16 1 Electric Water Heater 29 21 6 Insulation 181 70 44 Total 5,024 418 342

Table 9 details the screening process for eligible participants. The 342 participants were randomly selected from 4,973 unique participants with Wyoming mailing addresses, valid phone numbers and valid Rocky Mountain Power customer numbers.

Table 9. Participant Telephone Survey (Appliances, Windows, HVAC & Insulation) Sample

Total Total Records 5,311

No Phone Number 324 Measure Quantity Equals Zero 14

Eligible for call list 4,973 Completed Surveys 342 Response Rate* 7% Cooperation Rate** 18%

* Response rate is defined as the number of customers who completed a survey divided by the number of eligible participants in the call list.

** Cooperation rate is defined as the number of customers who completed a survey divided by the number of customers reached by phone.

Insulation Participants Verification Site Visits As shown in Table 10, Cadmus stratified the Insulation Participant Verification Site Visits by insulation type to ensure adequate sampling for all measure types.

Table 10. Insulation Site Visit Sample Size

Location Population Sample Achieved Site Visits Attic 159 48 62 Floor 9 8 3 Wall 13 11 4 Total 181 67 69



Retailer/Contractor Surveys In nearly all cases, Cadmus staff drew random samples, with sampled units having equal probabilities of being chosen. However, the team weighted the probability of selecting a given retailer for the survey’s CFL section, based on their total CFL sales. This ensured capturing a sufficient number of large retailers in the sample while retaining the desired statistical properties of a random sample.

Cadmus selected appliance and window retailers for interviews based on their products and business types, ensuring they adequately represented the greater population. This approach, intended solely for qualitative analysis, offered an advantage over random sampling of samples too small to produce statistically valid estimates.

Table 11 and Table 12 detail the screening process for eligible participants. The 30 participant and 30 nonparticipants were randomly selected from 78 and 563 unique Wyoming retailers respectively.

Table 11. Retailer Participant Survey Sample

Total Total Records 101

No Phone Number 19 Duplicate Records (by customer number and phone number) 4

Eligible participants in call list 78 Completed Surveys 30 Response Rate* 30% Cooperation Rate** 30%



Table 12. Retailer Nonparticipant Survey Sample

Total Total Records 626

No Phone Number 17 Duplicate Records (by customer number and phone number) 3 Ineligible Retailer for HES Program 43

Eligible participants in call list 563 Completed Surveys 30 Response Rate* 5% Cooperation Rate** 19%



Table 13 and Table 14 show the responses by retailer or contractor, indicating which sections were answered by each.



Table 13. Retailer Participant Surveys

Company/ Store Lighting Appliances Weatherization Ace Hardware X X X Ace Hardware X X Ace Hardware X Ace Hardware X X Ace Hardware X Best Buy X Brown's Western Appliance X Discount Grocery X Dollar Store X Gambles X Gizmo's Rents X Gizmo's/Cost Plus Appliance X Home Depot X X X Home Depot X X X Insulation Inc X John Paras Furniture & Appliance-Rock Springs X Kusel's Inc X Letz's TV and Appliance X Roger's Home Entertainment Center X Rushmore Furniture Co., Inc. X Sam's Club X Sears X Sears X Sears X Sears X True Value of Laramie X Walmart X X X Walmart X X Walmart X X Walmart X X



Table 14. Retailer Nonparticipant Surveys

Company/ Store Lighting Appliances Weatherization A To Z Value Hardware X Advanced Air Systems X X Alco Discount X X B & B Appliance X Big Horn Heating and Cooling Inc X C K Hardware X X Casper Tin Shop X Days Heating and Air Conditioning X Dollar Tree Stores Inc X Dollar Tree Stores Inc X Douglas Grocery X F & R Insulation Inc X Family Dollar Stores X Family Dollar Stores X Family Dollar Stores X Family Dollar Stores X Family Dollar Stores X Howshar Hardware X X Kmart Corporation X X Kn Energy Appliance Store X Knecht Home Center X X X Laramie Heating and Shtmtl X Leitheads Appliance Center X Lintons Big R Stores X X Moore Insulation Inc X Newcastle Hardware X Nielsen Plumbing & Heating X Pamida Discount Center 076 X Pokes Mercantile LLC X Sears X

In-Territory Lighting Survey Cadmus drew the in-territory lighting survey sample from a random list of Wyoming Rocky Mountain Power residential customers provided by Rocky Mountain Power. Surveyors screened respondents to identify recent CFL purchasers for the survey.

Table 15 details the screening process for eligible participants. The 254 participants were randomly selected from 9,135 unique customers with Wyoming mailing addresses, valid phone numbers and valid Rocky Mountain Power customer numbers.



Table 15. In-Territory Lighting Survey Sample

Total Total Records 10,000

Duplicate records (by customer number and phone number) 865 Eligible participants in call list 9,135 Completed Surveys 254 Response Rate* 3% Cooperation Rate** 10%



Marketing Materials Review The process evaluation included Cadmus’ review of marketing and communications developed to promote participation and educate target audiences on HES Program details. As appropriate, Cadmus also integrated findings on marketing approach and effectiveness into analysis from program staff interviews and customer surveys.

Data inputs used for marketing and messaging review included:

Collateral (e.g., promotional material, advertising, and educational pieces),

Presentation decks,

Online promotional elements, and

Marketing media mix and timing

Where applicable, the review included specific comments regarding the following.

General look and feel,

Brand and message consistency,

Program accessibility, and

Stakeholder criteria, including:

Incentive forms

Web-based marketing and educational collateral

Searchable retailer listings

Online processes availability

The marketing review also included a qualitative evaluation of online resources available from Rocky Mountain Power and comparison with other interactive resources.3

3 The online review assumed Rocky Mountain Power.net as an initial entry point for HES Program participants.



Impact Evaluation

This section provides the impact findings for the HES Program. These findings are based on an analysis of the data using several methods, including:

Participant and nonparticipant surveys,

Billing analysis,

Engineering reviews,

Site visits, and

Secondary research

Each of these data collection elements contributed to gross or net savings estimates. Table 16 summarizes the evaluation activities and goals of each effort.

Table 16. Summary of Evaluation Approach

Action

Impact

Process Gross

Savings NTG Participant Surveys (Appliance, HVAC, and Weatherization Measures) X X X In-Territory Lighting Surveys X X Out-of-Territory Lighting Surveys X Participant Retailer/Contractor Surveys X X Nonparticipant Retailer/Contractor Surveys X Billing Analysis X Insulation Participant Verification Site Visits X Stakeholder Interviews (Management Staff and Implementers) X Secondary Research X Secondary Data Analysis X

As noted, HES offered a number of different products and measures, which required different evaluation methods. To address the complexities and details of each individual measure group, the impact findings are organized into two sections:

1. Lighting

2. Appliances, HVAC, and Weatherization

Lighting During the 2009–2010 program years, Rocky Mountain Power incented over 214,000 CFL bulbs through nine different retailers in 30 stores. The bulbs contributed to 89 percent of the total HES savings and, as shown in Table 17, included standard and specialty CFL bulbs.



Table 17. Incented CFL Bulbs by Type

Bulb Type Incented Bulbs Percent of Total Spiral (Standard) 215,648 96 A-Lamp 1,658 1 Candelabra 144 0 Daylight 5,032 2 Reflector 2,228 1 T2 Base 48 0 Globe 570 0 Total 225,328 100 Source: 2009–2010 WY HES PECI tracking data.

Generally, CFL buy-down programs offer an effective alternative to traditional mail-in incentives due to their ease of participation, widespread accessibility, and low administrative costs. For such programs, utility incentives pass through manufacturers to retailers, who reduce bulb prices to the end consumer. The programs motivate retailer participation through reduced bulb prices without losses in their profits. At the customer level, participation may be so seamless that participants do not even know they purchased an incentivized bulb or participated in a utility program.

Upstream programs, however, offer particular evaluation challenges. Calculating metrics, such as installation rates and attributions, traditionally relies on finding participants and incentivized products; in this instance, however, purchasers may not be aware of their participation in a utility-sponsored program.

Consequently, the calculation of various inputs to the CFL lighting component required the use of primary and secondary data collection activities, as shown in Table 18. Lighting trends reported in the in-territory lighting surveys of Rocky Mountain Power’s Wyoming residential customers served as a proxy for HES lighting participants in lieu of verifiable participation data.

Table 18. Wyoming Lighting Activities

Activity N Metric Result Participant Retailer/Contractor Surveys 23 NTG, Willingness to Pay Net Savings In-Territory Lighting Surveys 254 Installation Rate, Installation Location, Hours-of-Use Gross Savings Out-of-Territory Lighting Surveys 70 Leakage Leakage Secondary Research N/A NTG Net Savings Secondary Data Analysis N/A Hours-of-Use Gross Savings



Evaluated Gross Savings Approach – Lighting Four different parameters informed the calculation of gross savings for the lighting component: in-service rate, delta watts, hours-of-use (HOU), and the wasted heat factor. The following algorithm was used to calculate gross lighting savings:

∆ ∗ ∗ ∗ 365 ∗

1,000

Where:

ΔWatts = Wattage of baseline bulb (ENERGY STAR CFL)

ISR = In-service rate, or percentage of units incented that get installed

HOU = Hours-of-use; annual lighting operating hours

WHF = Waste heat factor for energy to account for HVAC interaction affects (heating and cooling)

The annual savings algorithm was based on industry-standard engineering practices, consistent with the methodology used by the Northwest Regional Technical Forum (RTF) for calculating energy use and savings for residential lighting. Each component of the methodology is discussed in detail below.

In-Service Rate The in-service rate (also known as the installation rate) was determined through the in-territory lighting surveys of 254 recent CFL purchasers. The survey asked those who purchased CFLs during 2009 or 2010 a series of questions to determine whether the purchased CFLs had been installed, and, if so, in which rooms. As shown in Table 19 and Table 20, 67 percent of 2009 and 2010 bulbs were installed, with the most common installation locations living spaces (such as family and living rooms) and bedrooms. This evaluation did not include bulbs in storage as part of the in-service rate input, as they had not been installed during the 2009–2010 program period and, as such, did not contribute to first-year program savings.

Table 19. CFL Installation Rate (n=253)

Bulbs Percent of Total Installed 2,192 66.5 In storage 696 21.1 Discarded or given away 407 12.4 Total 3,295 100



Table 20. CFL Installation Locations (n=214)

Bulbs Percent of Total* Living space 490 30.4 Bedroom 416 25.8 Bathroom 223 13.8 Kitchen 216 13.4 Other 136 8.4 Basement 78 4.8 Outdoor 54 3.3 Total 1,613 100

* Percents may not add to 100 percent due to rounding.

Delta Watts Delta Watts represent the wattage difference between a baseline fixture and an equivalent CFL efficient fixture. For the HES Program, specific CFL products may be sold by participating Wyoming retailers. Rocky Mountain Power provided 2009–2010 CFL sales data by SKU4 number (model number and bulb type) for the 76 products eligible at the nine retail outlets. Sales data indicated sales of 225,3285 incented CFLs. Product sales data included CFL wattages, though lumen data or light outputs for bulbs was not available.

To determine per-bulb savings, the baseline incandescent wattage for each CFL bulb sold was estimated. As shown in Table 21, the baseline wattage was established using the comparable light output of the purchased CFL. Groups of lumen ranges (bins) were developed based on the Energy Independence and Security Act (EISA)6 of 2007.

Table 21. Lumen Bins by Baseline Wattage and Estimated CFL Wattage

Lumens Bins Baseline Wattage

(Wbase) Estimated CFL

Wattage (Weff) Bins 310–749 40 6–11 750–1,049 60 12–16 1,050–1,489 75 17–22 1,490–2,600 100 23–38

Analysis of a list of eligible ENERGY STAR CFL products provided estimates of the CFL Wattage bin for each associated lumen bin.

Three models classified as reflector type lamps did not follow the lumen bin classifications described above. Reflectors can be described as flood lights providing a direct path of light. The three eligible product wattages were 14, 15, and 16 Watts. A comparable baseline wattage for

4 SKU stand for Stock Keeping Unit, the unique make and model indicator for a specific retailer. 5 Sales in the tracking database differed from that reported in annual reports due to different reporting and

tracking calendars. For the purpose of this evaluation, the CFLs in the annual report were verified. 6 Congress signed EISA into law on December 19, 2007. The new law contains provisions for phasing in more

efficient incandescent lamps based on rated lumens. For example, a 100 Watt incandescent lamp with a rated lumen range of 1,490 to 2,600 will be required to have a minimum of 72 Watts, effective January 1, 2012.



reflectors (R30 or BR30 type) is 65 watts, based on manufacturer literature and as shown in Table 22.7

Table 22. Reflector Lumen Bins by Baseline Wattage and CFL Wattage

Lumens Bins Baseline Wattage

(Wbase) CFL Wattage (Weff) 650–750 65 R14, R15, and R16

Of the 76 eligible products included in the HES Program, 46 CFL SKU numbers were verified online for each retailer (including the reflectors) and each model’s rated lumens were recorded. For the remaining 30 CFL products, lumens were estimated based on an analysis of eligible ENERGY STAR CFL products.

ENERGY STAR Analysis A downloaded list of ENERGY STAR-qualified CFL bulb products which was last updated on May 24, 2011 was used for this analysis. The database consisted of 5,245 CFL products and their associated Wattage and lumens. The list required data cleaning to remove or update: inconsistencies in the database; missing values; decimal places; outliers; and incorrect entries. This cleaning removed or updated nine entries, resulting in a “cleaned” database of 5,243 CFL products.

The final database also included 117 three-way CFL bulb types. The analysis used the middle wattages, as specified by manufacturers.

The analysis broke out the ENERGY STAR CFL product list into lumen bins, specified by the EISA lumen requirements, and was extrapolated to the higher lumens bins. Table 23 shows the number of CFL products by lumen bin according to the ENERGY STAR database.

As shown in Figure 1, the baseline wattage followed a consistent trend: as CFL wattage increased, the comparable baseline wattage also increased. R14, R15, and R16, which represented reflector-type lamps, had different baseline wattages than the other product types. The reported baseline wattage and delta wattage was based on a Rocky Mountain Power HES 2009–2010 savings analysis.

Table 23. ENERGY STAR Product Counts by Lumen Bin

Lumens Bins ENERGY STAR Product Counts Less than 310 75 310–749 925 750–1,049 1,980 1,050–1,489 865 1,490–2,600 1,328 Greater than 2600 70

Total 5,243

7 The 65 Watt baseline was based on manufacturer specifications and product literature from GE, Philips, and

Westinghouse.



For CFL wattages where multiple ENERGY STAR products existed, lumens varied significantly. For example, 381 CFL products were 20 Watts with lighting outputs ranging from 850 to 2,150 lumens. To address these variations, median lumens were calculated for each bulb wattage.

As shown in Figure 1, the calculated trend line exhibited a relatively linear pattern. Based on the trend of median lumens and the specified lumen bins, the lumens for the 30 remaining CFLs products were estimated. For each incented CFL, baseline wattage was established using the comparable light output of the purchased CFL.

Figure 1. Median Lumens of CFL Wattage

Table 24 represents all the eligible 2009-2010 CFL products that were purchased as a part of the HES Program and their associated wattages. The evaluated and reported delta wattages show the differences in assumptions by eligible CFL product. The reported baseline incandescent wattages were found in documentation provided by PacifiCorp. The evaluated baseline wattages were determined by the analysis as described in the report.



Table 24. Evaluated and Reported Delta Wattage of 2009–2010 CFLs and Baseline Wattages

Eligible 2009–2010 CFL Wattages

Evaluated Baseline Wattage

(Wbase) Evaluated Delta

Watts (ΔW)

Reported Baseline Wattage

(Wbase) Reported Delta

Watts (ΔW) 9 40 31 40 31 10 40 30 50 40 11 40 29 50 39 13 60 47 50 37 14 60 46 60 46

R14 65 51 60 46 15 60 45 60 45

R15 65 50 60 45 R16 65 49 60 44 18 75 57 75 57 19 75 56 85 66 20 75 55 75 55 23 100 77 100 77 26 100 74 100 74 27 100 73 100 73

Cadmus recommends this approach to determine an equivalent baseline by the equivalent lumens of each lamp as it is consistent with EISA of 2007. For program evaluations for 2012 and beyond, EISA of 2007 has established equivalent baseline to follow.

Waste Heat Factor The waste heat factor (WHF) represents the portion of annual lighting energy with an interactive effect, whether lost or gained, with heating and cooling equipment. Rocky Mountain Power’s deemed savings value for CFL light fixtures used default inputs from the Northwest Power Planning Council’s Sixth Power Plan, one of which included a lighting analysis for space conditioning interaction adjustments. Deemed savings values did not include interactive effects for CFLs and ENERGY STAR ceiling fans. The inputs were modified to calculate the interactive effect for Wyoming, and that adjustment was applied to all interior lighting measures.

Rocky Mountain Power’s “Energy Decisions Fact Sheets” 2006 data for Wyoming was used to determine the saturation of heating and cooling equipment types, where the calculated interaction factor in the ENERGY STAR savings calculator for heat pumps was based on the Casper, Wyoming values for full-load heating and cooling hours.8 As most HVAC systems frequently operate at partial load, the calculation doubled the full-load hours and then divided by the total hours in a year to determine a percentage. See the following example:

Full-load heating hours: 2,620

Space heating interaction = 2,620 x 2 / 8,760 = 59.8 percent

Full-load cooling hours: 439

Space heating interaction = 439 x 2 / 8,760 = 10.0 percent

8 http://www.energystar.gov/ia/business/bulk_purchasing/bpsavings_calc/ASHP_Sav_Calc.xls



Customizing the interactive effect calculations resulted in a 9.7 percent reduction of interior lighting electricity savings due to combined heating and cooling effects. Program CFLs could be installed in interior or exterior applications. As the interaction factor does not apply to exterior lighting applications, the calculations included applying a weighted average of interior and exterior installations to the interaction factor. The resulting 9.4 percent WHF, used in the CFL savings analysis, addresses all CFL applications.

Hours-of-Use The average hours-of-use (HOU) was calculated using ANCOVA model coefficients, drawn from combined, multistate, multiyear data from recent CFL HOU metering studies.9 This model expressed average HOU as a function of room type, existing CFL saturation, and the presence of children in the home. Appendix C provides a more detailed exploration of the impact methodology used to estimate CFL HOU.

All the necessary independent variables were estimated, except for existing CFL saturation, using response data from the participant telephone surveys, which were summarized in Table 20. For CFL saturations, data from comparison households in service areas with relatively young CFL programs were averaged,10 as these data were not available for Rocky Mountain Power’s Wyoming service area.

The weekend HOU was calculated using parameter estimates from the ANCOVA model. The weighted average of these two values provided the average annual HOU of 2.25 hours per bulb (see Table 25).

Table 25. HOU by Day Type

Day HOU Weight

Weekday 2.36 69.3% Weekend 2.02 30.7% Overall 2.25

Lighting Findings Table 26 presents the resulting evaluated gross savings by bulb wattage. Evaluated per unit savings included HOUs, delta Watts, WHF, and in-service rates as discussed above. Rocky Mountain Power’s reported per-unit savings, based on program analysis documentation, included a 0.8 installation service rate. Neither value contained adjustments for leakage or NTG.

9 2010 Evaluation, Measurement, and Verification Report, March 15, 2011, Cadmus, prepared for Dayton Power

and Light

10 Albee, K., et al. One Analysis to Rule Them All and In the Darkness Give Them CFLs. Proceedings of the 2011 International Energy Program Evaluation Conference, Boston, MA.



Table 26. Evaluated and Reported Per Unit CFL Savings by Bulb Wattage for 2009–2010

Eligible 2009–2010 CFL Wattages

Evaluated Per Unit Gross Savings (kWh)

Reported Per Unit Savings (kWh)

9 15.35 27.12 10 14.86 35.04 11 14.36 34.16 13 23.28 32.40 14 22.78 40.32

R14 25.26 40.32 15 22.29 39.44

R15 24.76 39.44 R16 24.27 38.56 18 28.23 49.92 19 27.74 57.84 20 27.24 48.16 23 38.14 67.44 26 36.65 64.80 27 36.16 63.92

The HES Program realized 23.99 kWh in annual evaluated per unit gross savings, weighted by the total number of CFLs sold, with total program evaluated gross savings of 5,460,861 kWh annually (as shown in Table 27). A review of Rocky Mountain Power’s documentation indicated filed reported savings of 7,170,603 kWh annually, including a 0.8 combined NTG and leakage assumption factor.

Table 27. Evaluated and Reported Program CFL Savings for 2009–2010

Reported Number CFLs Purchased*

Reported Program Gross Savings

(kWh)

Evaluated Program Gross Savings

(kWh) Gross Savings

Realization Rate 214,364 7,170,603 5,460,861 76%

* Total CFLs reported in the 2009 and 2010 Rocky Mountain Power filing.

Estimating Net Savings – Lighting Upstream energy-efficiency programs, such as the lighting component of the HES program, present several evaluation challenges. By design, such programs are largely invisible to consumers, and many customers may be unaware that they took part in the program. Evaluations of upstream programs implemented elsewhere have found that the majority of customer participants were unaware of their participation status.

The relatively low cost of light bulbs further complicates the NTG analysis of upstream lighting programs. Consumers can accurately recall details about buying light bulbs (e.g., how many individual light bulbs and packages purchased, when the purchase occurred) for only a short time after the purchase. This applies to incandescent bulbs as well as CFLs, especially as consumers become familiar with CFLs and no longer view them as novelty items.

In addition to sales of program-discounted CFLs, utility marketing and outreach efforts often lead to sales of larger numbers of non-program CFLs. This spillover effect especially occurs



when retailers reduce the price of non-program CFLs to keep them competitive with incented lamps. Non-program CFL sales (i.e., sales of non-discounted CFLs during program promotions and CFL sales made outside of program promotional periods) can occur at participating retailers as well as at nonparticipating retailers. Limiting the NTG analysis to the few consumers who recall purchasing a program-discounted CFL can significantly underestimate program impacts.

The NTG for CFLs was estimated using three different approaches. First, participating retailers and contractors were interviewed to obtain their estimate of net program impacts. Second, the secondary literature was searched for estimates, and third, willingness-to-pay research was conducted to estimate a demand curve for CFLs, from which a freerider rate was inferred.

Participant Retailer/Contractor Surveys The HES Program lamps’ NTG was estimated using responses from in-depth participating retailer interviews. Of 30 participating retailers interviewed across various distribution channels, 14 addressed the lighting component of the HES Program, with six answering the required battery of NTG questions. The representative group of retailers providing data included: Ace Hardware, Discount Grocery, Family Dollar, Home Depot, Sam’s Club, and Wal-Mart. Responses represented 44 percent of 2009–2010 HES incented lamp sales and 30% of participating stores and contractors.

Store representatives were asked a series of questions designed to estimate the percentages of all CFLs they would have sold in the HES Program’s absence, as well as the percentages of their total CFL sales incented through the HES Program during 2009 and 2010. The participant retailer/contractor survey accounted for freeridership and spillover with questions addressing participating retailers’ lift in total CFL sales resulting from the HES Program (i.e., CFLs attributable to the HES Program, including non-program CFLs). Appendix D provides interview guides for each of these groups.

The NTG battery of questions included:

1. “If the HES incentives were not available over the past year, do you think your [stores] sales of standard ENERGY STAR CFL bulbs would have been about the same, lower, or higher?”

2. “Over the past two years, by what percent would your [stores] sales of standard ENERGY STAR CFLs have been [lower/higher] without the HES Program?”

3. “During 2009 and 20010, what percent of your [stores] total CFL sales would you estimate were CFLs purchased through the HES Program?”

In assessing responses to the above questions, NTG was estimated as follows:

1. As question 2 and 3 responses were recorded in percentile ranges, calculations used the midpoint of each range.

2. The HES Program tracking database provided program lamp sales data by store. This included an estimated number of CFLs sold through the HES Program by retailer.

3. The following equation provided estimated total CFL sales by retailer:



% #3

4. The following equation provided estimated sales, by retailer, in the HES Program’s absence:

1 % #2

5. The following equation provided estimated lift or CFL sales attributable to the HES Program by retailer:

/

6. The following equation estimated NTG by retailer:

To ensure accuracy and reliability to question 1 and 2 responses, survey administrators confirmed responses to these questions by asking: “Just to confirm, your sales of standard ENERGY STAR CFLs would have been [insert %] [lower/higher] if the [/Rocky Mountain Power] program was not available?”

Individual NTG ratios were weighted by the distributions of program lamps sold by each of the six retailers providing useable NTG responses. For example, Home Depot was weighted by the percentage of program lamps they sold through the HES Program. This weighting approach ensured that the final NTG estimate reflects distributions of program CFLs, and that high-volume retailers were more heavily weighted in the final NTG calculation. To calculate the weight for each store, each store’s program lamp sales were calculated as a percentage of total lamps sold by all retailers, then were divided by the sum percentage of all six stores’ lamp sales relative to the program lamp total. See Table 28.

Table 28. Interviewed Retailer Program Lamp Sales and Weights

Retailer Contributing to NTG Retailer City

Total Program Lamp Sales Weight

Ace Hardware Evanston 1,426 0.014 Wal-Mart Evanston 8,888 0.090 Sam’s Club Casper 61,443 0.620 Home Depot Casper 26,443 0.267 Ace Hardware Cody 623 0.006 Family Dollar Green River 298 0.003

Total 99,121 Source: Questions D14 and D15 of the retailer participant survey.

. As shown in Table 29, the resulting mean store-weighted NTG estimate is 1.18.

P22170

Highlight

P22170

Highlight

P22170

Highlight

P22170

Highlight

P22170

Highlight

P22170

Highlight



Table 29. Responses to NTG Questions and Weighted NTG Estimate

Retailer

Response of Whether CFL Sales Would be lower/higher

or the Same Without HES Program

Estimated Program Lamps Sales as a

Percent of Total Lamp Sales Lift NTG

Retailer 1 15% lower 15% 1,426 100% Retailer 2 0% same 55% 0 0 % Retailer 3 55% lower 95% 35,572 58% Retailer 4 15% lower 5% 79,329 300% Retailer 5 55 % lower 45% 761 122% Retailer 6 0% same 95% 0 0%

Weighted NTG 118% Source: Questions D5 and D7 of the retailer participant survey.

Potential Bias and Uncertainty There are potential sources of bias that contribute to the uncertainty around the store-weighted NTG estimate, including:

The small sample of market actor responses results in a wide range of NTG estimates (see Table 29). Responses from this small sample may not sufficiently represent all stores of the same name or all stores within each retail distribution channel.

Program lamp sales for the six retailers contributing to NTG represent less than half (44 percent) of the total lamps sold through the HES Program (225,328).

Some retailers could not provide estimates of program lamp sales or discuss how their sales would have been impacted without the HES incentives (known as non-response bias).

Therefore, uncertainty around the 1.18 NTG estimate is large, resulting from the small sample and potential biases from non-response.

Secondary Data Review For a second estimate of NTG, Cadmus reviewed the literature on upstream lighting programs as there is no single way to estimate NTG for upstream residential lighting programs that is broadly considered best. Utilities across the United States have employed a number of different methodologies to derive NTG ratios, often using a combination of methodologies. These include:

Participant and nonparticipant retailer interviews. Interviews with corporate- and store-level retailers include questions regarding retailers’ total monthly or annual CFLs sales, monthly or annual program sales, and changes observed in CFL sales and buying patterns due to the program. Retailer interviews often also ask about changes in customer awareness and CFL stocking patterns.

Consumer telephone surveys. Consumer telephone surveys query a random sample of the sponsoring utility’s customers about their recent light bulb purchases. Surveys may include questions about the quantity of CFLs recently purchased, quantity of incandescent and other light bulbs recently purchased, consumers’ awareness of and



experience with different types of energy-efficient lighting, and consumers’ recollection of sponsoring utility identification.

Revealed preference intercept surveys. Revealed preference intercept surveys query consumers about their lighting product preferences based on their actual purchasing behaviors. These are administered in stores, at the time of light bulb purchases.

Willingness to pay (WTP) assessments. WTP assessments describe lighting product features to survey respondents, and then ask respondents how much they would be willing to pay for products with various feature combinations. These assessments provide more theoretical than revealed preferences, in that they rely on respondents’ hypothetical purchasing decisions (rather than the in-store, time-of-purchase decisions captured by revealed preference intercept surveys).

Conjoint/price elasticity analysis. In conjoint analysis, survey respondents choose between different light bulbs (e.g., A-line, flood, incandescent) which are characterized by six or fewer distinct attributes (e.g., bulb type, price, lifetime, price promotion, brand, light color, recommendation). A conjoint software program (e.g., Sawtooth) determines price elasticity by simulating participants’ willingness-to-pay for CFLs with different attribute configurations at various price points. To estimate the NTG ratio from such a model, evaluators calculate elasticity associated with flood and A-line CFLs using estimated market shares at the average non-discounted price and at the average fully-discounted price. Both price points are estimated using a hedonic pricing regression model. The ratio between these market shares provides the freeridership value. The NTG ratio then equals one minus the freeridership value.

Multistate regression analysis. This approach pools data collected through customer telephone and in-home audit lighting surveys administered in multiple program and non-program areas across the U.S. into a single regression model. Pooled data are used in an equation predicting CFL purchases and NTG ratios by controlling for factors affecting CFL sales, such as income, education, homeownership status, home size, electricity rates, and concentrations of big-box stores.

Secondary research. Secondary research studies NTG estimates derived by residential lighting programs elsewhere in the U.S., selecting the most appropriate NTG ratio for the utility the research is conducted for. Secondary research activities include reviewing applicable past evaluations and conference papers, contacting utilities currently offering programs, and searching industry evaluation databases.

Table 30 summarizes the secondary research findings related to methodologies used and NTG ratios derived from other recent upstream residential lighting programs across the U.S. For utilities using multiple NTG approaches and with available data, the table shows the NTG for each approach as well as the final NTG the utility selected for the overall program.

Rocky Mountain Power Wyoming HES 2009–2010 Draft Report September 9, 2011


Table 30. Summary of Secondary Research Results

Program Sponsor State

Program Years

Evaluated Year of

Program Overall NTG

NTG Ratio by Methodology

Customer Telephone

Survey

Supplier Telephone Interview

Secondary Research

WTP Assessment

Revealed Preference

Conjoint/ Price

Elasticity Analysis

Multistate Regression

Ameren Illinois IL 2010 PY2 0.83 0.83 Ameren Missouri MO 2010 PY2 0.96 0.96 Efficiency Maine ME 2003-2006 PY1–PY4 0.94 0.94 Massachusetts ENERGY STAR

MA 2009–2010 PY8–PY9 0.45 Spiral: 0.39 Specialty: 0.49

Spiral: 0.49 Specialty: 0.31

Spiral: 0.37 Specialty:

0.36

Specialty: 0.59 Spiral: 0.53 Specialty:

0.45 PG&E CA 2006-2008 PY3–PY5 0.49 X* 0.49 SCE CA 2006-2008 PY3–PY5 0.64 X* 0.64 SDG&E CA 2006-2008 PY3–PY5 0.48 X* 0.48 PPL Electric (PA) PA 2010-2011 PY2 0.85 0.85 Rocky Mountain Power–UT

UT 2006-2008 PY1–PY3 PY1 = 0.840 PY2 = 0.822 PY3 = 0.868

PY1 = 0.840 PY2 = 0.822 PY3 = 0.868

X* X*

Rocky Mountain Power–WA

WA 2006-2008 PY1–PY3 PY1 = 0.919 PY2 = 0.894 PY3 = 0.807

PY1 = 0.919 PY2 = 0.894 PY3 = 0.807

X* X*

Southwestern Public Service Company

NM 2009 PY1 0.81 0.81

Wisconsin Focus on Energy

WI 2007-2010 PY1-PY3 PY1 = 0.75 PY2 = 0.67 PY3 = 0.62

PY1 = 0.75 PY2 = 0.67 PY3 = 0.62

Xcel Energy CO 2008-2009 PY3–PY4 1.0 0.738 0.601 0.54-1.97 1.65 Unspecified mid-Atlantic utility

N/A 2009–2010 PY1–PY2 0.80 0.80

Unspecified SW utility

N/A 2009–2010 PY1 0.75 0.75

Unspecified SW utility

N/A 2010-2011 PY2 0.79 0.79

* Secondary approach; NTG value not available.



Secondary sources indicated NTG ratios for other upstream residential lighting programs ranging from 0.45 to 1.0. When these other programs and NTG ratios were studied to identify the most appropriate NTG ratio for Rocky Mountain Power, it was first recognized the HES Program has operated for roughly three years. Given the HES Program’s relative youth, and as NTG ratios for more mature upstream lighting programs tend to be lower than NTG ratios for newer programs, the research focus was narrowed to other relatively new programs. Therefore, California, Massachusetts, and Wisconsin were disregarded due to their relatively long-standing CFL promotional programs.

Also, in light of the program’s duration, NTG ratios of programs’ first-year operations may not be appropriate for new programs such as HES. Given these differences, NTG ratios in programs’ third year (where available) or second year were used. Averaging the remaining values, a 0.86 NTG ratio was derived for the Rocky Mountain Power upstream residential lighting program based on reviews of secondary data sources.

Lighting Customer Willingness to Pay (In-Territory Lighting Surveys) A total of 254 in-territory lighting surveys were conducted in August 2011, drawn at random from a Rocky Mountain Power list of 9,135 Wyoming residential customers. The survey asked respondents a battery of questions designed to determine their willingness to pay for CFLs in absence of HES Program mark-downs. After determining how many CFLs participants purchased in 2009 and 2010, participants were asked to indicate whether they would:

1. Generally purchase more CFLs, fewer CFLs, or the same number of CFLs at various un-incented hypothetical price levels.

2. The quantity of CFLs they would hypothetically purchase at various un-incented prices.

Specifically, customers were asked how many lamps they would purchase at three prices: $4.40, $11, and $1.10. These prices are two times, five times, and one-half the observed un-incented price of CFLs. One hundred and forty-three respondents gave answers at all three prices.

CFL demand was assumed to be inversely related to price, indicating that participants would purchase more CFLs at lower prices. To estimate participant willingness to pay for un-incented lamps, a demand curve for survey participants was estimated, relating hypothetical prices and quantities. Figure 2 illustrates the program lamp demand function based on responses from the in-territory lighting surveys. The Y-axis shows prices and the X-axis shows quantities of lamps purchased at each price. An equation describing the relationship between price and quantity is also shown in the figure.



Figure 2. Demand Schedule for Hypothetical Lamp Purchases

To estimate the number of lamps purchased at the average program price per lamp (net lamps) and the number of lamps purchased without the program incentive (freeridership), the number of lamps that would be purchased at the average incented price of $1.19 and at the average un-incented price of $2.20 was estimated. The resulting numbers are 2,232 and 2,846, respectively, as shown in Figure 3.

Figure 3. Modeled CFL Quantities for FR Estimation

The number of lamps to the left of the vertical line from the un-incented price ($2.20)—in this case 2,232 lamps—are freerider (FR) lamps that would have been purchased without the incentive. Only lamps to the right of this value and to the left of the incented lamp price represent program effects. Thus the equation for FR is as follows:

$4.40

$11.00

$1.10

y = 20.499e-0.001x

R² = 0.9996

$0.00

$2.00

$4.00

$6.00

$8.00

$10.00

$12.00

0 500 1000 1500 2000 2500

Hyp

othe

tical

Pric

e

Quantity of CFLs

$0.00

$2.00

$4.00

$6.00

$8.00

$10.00

$12.00

0 500 1,000 1,500 2,000 2,500 3,000 3,500

Pric

e

Quantity of CFLs= Modeled Q CFLs

Quantity at avg. non-incented price

Quantity at avg. incented price



1 _ _

_

Where:

Qcflavg_incented = 2,846; quantity of CFLs purchased at the average price of incented lamps

Qcflavg_unincented = 2,232; quantity of CFLs purchased at the average price of un-incented lamps

Responses to the in-territory lighting survey produced a 78 percent FR estimate.

The estimate derived in this way is the freerider rate, not the net effect. Programs could produce a significant degree of spillover that is unaccounted for in this approach. The primary way upstream programs produce spillover is by reducing the price of lamps sold without incentives. The widespread availability of CFL incentives has reduced the price of un-incented lamps as well as incented lamps. Thus, the observed un-incented CFL price of $2.20 is substantially less than recent prices in other markets. For instance, recent research in Maryland indicates an un-incented CFL price of $4.53. Other research indicates an un-incented price between $3.37 and $3.50. These higher prices better reflect the cost of a CFL in the absence of program incentives. As the un-incented price estimate rises, the FR rate declines because fewer lamps would have been purchased in the absence of the program. At $3.37 for an un-incented lamp, the FR rate is 63 percent. At $4.53, the FR rate is 53 percent. It is not possible to quantify the impact of programs on un-incented lamps with the data available. $4.00 is a reasonable value; at this cost, the FR rate estimated from WTP data is 57 percent, for a NTG value of 0.43.

Statistical Significance and Uncertainty Random digit dial customer phone surveys avoid bias through the very randomness of the selection process. With every sample, however, random error occurs, reflecting the people selected to participate in the study. For instance, this study’s sample reported a willingness to purchase 969 CFLs at twice the current price. Had a different group of people been sampled, it could be expected, by the random circumstance of those falling into the sample, that the CFL willingness to purchase totals could be somewhat larger or smaller. Using classical sampling theory, the likely boundaries within which error lies were estimated.

A 90 percent confidence interval was constructed for random error around the sum of CFLs purchased at each hypothetical price level.11 Table 31 shows error due to sampling for the sum of purchased CFLs at each price. The relative precision of the estimates ranges from 14.6 to 23.3 percent, indicating that the estimate of NTG from this approach does not have a high degree of stability. A NTG value of 0.43, for instance, is within the 90 percent confidence interval of the observed data.

11 LED estimates had greater variability and a smaller sample size, and therefore much lower relative precision.



Table 31. 90 Percent Confidence Interval and Summary Statistics for the CFL WTP Study (n = 143)

Price Sum of CFLs Purchased

Precision at 90% Confidence

Twice the avg. price 1,235 ±14.6% Five times the avg. price 478 ±23.3% Half the avg. price 2,297 ±16.8%

Source: Questions F1-F6 of the out-of-territory lighting survey. The random error reported in Table 31 does not include systematic measurement error that may be associated with this WTP methodology. For instance, if some respondents experienced social pressure to report their CFL purchases, they may estimate making more purchases at high prices than they would actually make. If this bias in part of the sample is not offset by an opposite bias to under-report purchases by other respondents, the result is a systematic measurement error. It is not known from the data available whether this kind of bias is present in the data, or if it is present, its size.

NTG Findings Efforts to identify NTG for the HES lighting program produced three values:

Retailer/contractor surveys: 1.18

Secondary literature review: 0.86

WTP research: 0.43

Each approach had limitations. The retailer surveys, for instance, were few in number and respondents had difficulty responding to the questions related to NTG, with fewer than half providing useful information. What is more, the variability across responses was large, with a single high estimate of program effect at a large retailer dramatically increasing the overall estimate. The secondary literature review also provided a wide range of estimates, underscoring the inherent difficulty in estimating NTG, and comparability to the HES program varies, as well. The WTP research required respondents to answer difficult, hypothetical questions. Once again, only 58 percent of respondents could provide useful information. Moreover, we know from our research around the country that the WTP estimates NTG are often lower than estimates derived from other methodologies.

Given the inherent uncertainty in estimating NTG, an approach that triangulates the methods reduces the effect of unknown error associated with each. In principal, Cadmus should combine the three estimates weighted by some measure of each estimate’s certainty, such as its variance. This would decrease the power of the retailer survey in the overall NTG estimate because of the wide disagreement among retailers. Cadmus cannot directly estimate variance for the WTP estimate, however, so we cannot bring it into this scheme. In such a case, equal weighting is a reasonable approach. An approach that equally weights each of the three estimates gives a blended NTG value of 0.83. The fact that this is close to the estimate of the secondary literature review (0.86), which itself blends numerous estimates, is reassuring that, whatever the true NTG value, Cadmus has avoided extreme error.



CFL Leakage

Background Before selecting stores for participation, PECI contracted with Buxton, a market research firm, to identify likely CFL customers for retailers within Rocky Mountain Power’s service area. PECI used Buxton’s proprietary tool, Micro Analyzer, to identify stores with high proportions of likely CFL purchasers.12 This tool defined profiles for each store, including a drive-time based polygon of likely customers around each retailer and its respective mix of 66 consumer segments. PECI then mapped these profiles to the Wyoming Rocky Mountain Power service areas to determine proportions of likely customers belonging to each area utility. PECI targets participant retailers where 90% of customers within a 10 minute drive time are within the Rocky Mountain Power service territory. The final dataset for analysis contained the proportion of likely customers by utility for each retailer in Rocky Mountain Power’s service area.

Methodology In order to quantify the impact of CFL leakage, defined as the proportion of incented CFLs purchased by non-Rocky Mountain Power customers, an analysis was conducted using the market research data and primary out-of-territory lighting survey data. Combining the market data and sales data received from PECI, likely leakage values were estimated by mapping the proportion of total sales by store to the estimated proportion of likely CFL purchasers not served by Rocky Mountain Power. Likely leakage by store was then defined as the product of the proportion of total incented CFL sales and the proportion of non-Rocky Mountain Power likely purchasers for each store. That is, for each store, ‘i’:

∑

∗

Once likely leakage had been calculated for each store, leakage was aggregated to the zip code level. For zip codes with likely leakage, the out-of-territory lighting survey was conducted, which was a random digit dial survey of non-Rocky Mountain Power customers that purchased CFLs in the past two years. A summary of these data are shown in Table 32.

12 A brief overview of Buxton’s database and analytics was found on their Website:

http://www.buxtonco.com/pdf/product/Retail_MKSolutions_brochure.pdf



Table 32. Likely CFL Leakage by Store

Store ID

Percent of Likely Shoppers that Are Rocky Mountain Power

Customers

Percent of Incented Bulb

Sales Potential Leakage

Potential Leakage with Imputation

141243 50% 6.9% 3.4% 3.44% 145717 50% 5.1% 2.5% 2.53% 1129961 50% 3.1% 1.5% 1.55% 377842 Missing* 5.4% N/A 0.86% 1084761 50% 1.4% 0.7% 0.68% 1014542 50% 1.3% 0.6% 0.64% 1047180 50% 1.1% 0.5% 0.54% 1075141 50% 0.7% 0.3% 0.34% 1043222 50% 0.3% 0.1% 0.14% 119 50% 0.3% 0.1% 0.13% 576747 Missing* 0.1% N/A 0.02% 559910 Missing* 0.1% N/A 0.02% 608173 Missing* 0.1% N/A 0.02% 272 Missing* 0.0% N/A 0.01% 1025938 33% 0.0% 0.0% 0.00% 642546 100% 27.3% 0.0% 0.00% 600319 100% 15.0% 0.0% 0.00% 600149 100% 11.7% 0.0% 0.00% 146120 100% 6.0% 0.0% 0.00% 161744 100% 4.8% 0.0% 0.00% 145612 100% 3.9% 0.0% 0.00% 834940 100% 2.0% 0.0% 0.00% 1077670 100% 0.8% 0.0% 0.00% 1126340 100% 0.7% 0.0% 0.00% 332180 100% 0.6% 0.0% 0.00% 690421 100% 0.6% 0.0% 0.00% 289126 100% 0.3% 0.0% 0.00% 722410 100% 0.1% 0.0% 0.00% 601420 100% 0.1% 0.0% 0.00% 1475834 100% 0.1% 0.0% 0.00% 1052911 50% 0.0% 0.0% 0.00% 1098321 99% 0.0% 0.0% 0.00% 532890 50% 0.0% 0.0% 0.00% Total 100% 9.98% 10.91%

*”Missing” indicates that the Buxton analysis was not run on that particular store

For a small number of stores, the Buxton dataset did not contain data on likely customers. In these cases, Cadmus used data from stores within the same zip code. If there was not data for other store with a matching zip code, then a sales-weighted average proportion of likely non-Rocky Mountain Power customers from the remaining data was used. This is reflected in the rightmost column of the table above. Based on Cadmus’ imputation, the potential leakage for the HES program was 10.9 percent; slightly higher than PECI’s target of 10 percent.

Based on the data available from Buxton market research, the average CFLs per household is expected to be approximately equal between Rocky Mountain Power and non-Rocky Mountain Power customers in these leakage-prone areas. That is, the two populations should be equally



likely to purchase CFLs. Cadmus tested this hypothesis using a survey sample of 70 non-Rocky Mountain Power customers and comparing them with the CFL participant surveys data.

To compare the two populations, Cadmus evaluated the statistical significance of the difference in mean CFL purchases using a t-test. In this test, the null hypothesis assumes the two means are not statistically different, whereas the alternate hypothesis assumes they are. If the test results in a p-value below 0.10, the two means can be said to differ with 90 percent confidence (Table 33).

Table 33. T-Test for the Difference in Mean CFL Purchases between Rocky Mountain Power and Non-Rocky Mountain Power Customers

Type n Mean SE Lower Limit (90%

Confidence) Upper Limit (90%

Confidence) p-value* Participant Sample 253 13.0 0.7 11.1 15.0

Leakage Sample 70 11.4 0.9 9.4 13.4 Difference -1.6 1.1 -3.6 0.4 0.15 *P-values indicate the degree of confidence to which Cadmus can assert that the given value equals zero. In the t-test shown above, it represents the probability that the two means are equal. Because this value is greater than 10 percent, Cadmus cannot assert with 90 percent confidence that the two values are different.

Findings As shown, CFL purchases did not differ significantly (with 90 percent confidence) between the two groups. Based on the test results, Cadmus chose not to adjust the likely leakage estimated from the Buxton research.

As the sum of each store-specific likely leakage value, program-wide CFL leakage equaled 10.91 percent.

Appliances, HVAC, and Weatherization As the HES Program contains several of measures, this section addresses evaluated gross and net savings estimates for the following measures:

Clothes washers

Dishwashers

Water heaters

Refrigerators

Ceiling fans

Light fixtures

Insulation

Windows

Heat pumps

Central air conditioners

Evaporative coolers



As these measures greatly differ, Cadmus utilized the most effective evaluation techniques for specific measures incented, as shown in Table 34.

Table 34. Gross Savings Evaluation Methodology, by Measure

Measure Methodology Clothes Washers Engineering Review Dishwashers Engineering Review Water Heaters Whole House Model Refrigerators Engineering Review Ceiling Fans Engineering Review Light Fixtures Engineering Review Insulation Whole House Model Windows Whole House Model Heat Pumps Whole House Model Central Air Conditioners Whole House Model Evaporative Coolers Engineering Review

The following sections discussed each methodology and evaluated savings in depth.

Calculation of Gross Savings Calculation of gross savings for these measures involved two steps for each measure group: determination of installation rates, and an engineering review or whole house model. Cadmus enhanced the insulation savings estimates through site visit and billing analyses. These are described in detail below.

Installation Rate For each measure group, the participant telephone surveys asked participants a simple series of questions to determine whether or not they installed incentivized products. For products with multiple units of measure, such as windows, participants could be awarded credit for partially installing incented units. This proved unnecessary as survey results indicated complete installation of each measure surveyed, resulting in 100 percent installation rates. The evaluation granted low savings measure groups not surveyed (such as duct sealing and permanently installed evaporative coolers) the average installation rate of surveyed measures (in this case, 100 percent).

Review Tracking Database Cadmus reviewed PECI’s lighting and HES participant databases to check for duplicate records and ineligible participants. Table 35 shows the outcome of the tracking database review.

Table 35. Tracking Database Review Number of Records Action

Recorded Participation 5,341 2011 Reporting Year 235 Dropped CAC Contractor 41 Dropped

Verified Participation 5,065



The last two weeks of December 2010 fall under Rocky Mountain Power’s 2011 reporting year. Cadmus identified and dropped 235 records that belong in the 2011 reporting year. PECI not only tracks participants who receive central air conditioner (CAC) and heat pump measures, but also the contractors who provided this unit. Cadmus identified and dropped 41 records that corresponded to a CAC contractor.

Neither of these actions impacted reported savings. For CAC contractor measures, PECI correctly set measure quantities to 0 in the tracking database to avoid double counting of savings. For records that fall under the 2011 reporting year, Rocky Mountain Power correctly excluded these from 2010 reported savings.

Cadmus also reviewed both PECI’s tracking of 2009 and 2010 upstream lighting measures. Using both a summary from their tracking database and invoices from the last two weeks of December 2010, Cadmus reconciled reported CFL measure quantities with PECI’s data.

Engineering Review The engineering review used data from the participant phone surveys and secondary data to evaluate gross savings for clothes washers, refrigerators, dishwashers, ceiling fans and light fixtures. As shown in Table 36, realization rates ranged between 16 percent and 187 percent. For a more detailed analysis, please refer to Appendix K.

Table 36. Engineering Review Summary Table

Year Measure Standard

Gross Reported Savings

(kWh/unit)

Gross Evaluated Savings

(kWh/unit) Realization

Rate 2009-10 Clothes Washers Clothes Washer-Tier

One (1.72 - 1.99 MEF) 218 (weighted

average) 303 139%

2009-10 Clothes Washers Clothes Washer-Tier Two (2.0 + MEF)

234 (weighted average)

437 187%

2009-10 Refrigerator ENERGY STAR Refrigerator

98 54 56%

2009-10 Dishwasher ENERGY STAR Dishwasher

29 29 100%

2009-10 Ceiling Fans Ceiling Fans 107 17 16% 2009-10 Fixtures Fixtures 92 64 70%

Whole-House Energy Modeling

Overall Methodology For insulation, heating and cooling measure developed by PECI, Cadmus first examined the following savings values:

PECI Gross Savings Calculated. These PECI-developed per measure kWh savings were based upon either an energy simulation model, Energy Gauge V. 2.8, RTF values, or a 2006 Evaporative and Air Conditioners evaluation and were found in Exhibit A5_Wyoming Savings Summary_2010_122109.xlsx and Exhibit A.5_Wyoming Savings Summary_2008 & 2009_PECI_12312008 NS.xlsx. These calculated savings were usually the basis of the reported savings described below.



PECI Gross Savings Reported. These reported PECI-developed per measure kWh savings were used as the deemed savings for each measure and were found in PC WY HES Extract.xlsx. These reported values did not always match the PECI Gross Savings Calculated values described above. In these instances, Cadmus was not sure how the reported values were derived. Measures with calculated savings but no reported savings had no participating customers for that program year..

In order to verify these savings, Cadmus developed their own independent estimates of realized savings:

Cadmus Gross Savings Realized. For most of the measures, Cadmus used a residential energy simulation model -- Architectural Energy Corporation’s Rem-Rate V12.95 – to estimate realized savings. (RESNET® accredits Rem-Rate for modeling residential homes.) Cadmus first determined the characteristics of a typical home based upon surveys, site visits, and PECI’s measure tracking database. Cadmus then used these characteristics as inputs into Rem-Rate. (See Table 37 and Table 38). The difference in modeled overall energy use between a baseline home (e.g. one with the standard efficiency unit such as a 13 SEER AC unit) and the efficient home (e.g. one with a 15 SEER AC unit) were the measure’s realized gross savings. For a few other measures that could not be modeled through Rem-Rate, Cadmus used other secondary sources to estimate realized savings.

Cadmus Gross Savings Using Energy Gauge. Using the same characteristics of a typical home and baseline and efficient home scenarios, Cadmus checked each Rem-Rate based realized savings value against one calculated using Energy Gauge. If the savings values were substantially different between the two models, algorithm differences were researched to determine why the values differed. Each modeling program is embedded with assumptions, some which are relevant to each measure, and some which are not. Each method behind modeling must be researched to ultimately decide the desired method for this study.

Table 37. Characteristics of Typical Modeled Homes

Parameter Value Weather location Casper, WY Home Type 2 story, single family house Home Conditioned Floor Area 2137sqft Foundation Type Vented Crawlspace Walls 2X4 Exterior walls with R-11 Cavity Insulation Attic Flat trussed attic with R-19 insulation between cord rafters Framed Floor Framed Floor with R-19 insulation between floor joists Windows Double Pane Vinyl windows U-Value 0.48, SHGC 0.58 Infiltration 9.84 ACH @ 50Pa Heating System/Cooling system Varies by model to capture several types of equipment. All types air distribution systems. Duct Leakage 499.4 CFM @25Pa Thermostat Non-Programmable 64.7F Heating 73F Cooling



Table 38. Typical HVAC Systems

Electric Furnace wo/CAC Electric Furnace w/CAC Gas Furnace w/CAC 150 kBTU(44kW) Forced Air Electric Furnace

150 kBTU(44kW) Forced Air Electric Furnace with 13SEER 4.5 ton Central AC

150kBTU Forced Air Gas Furnace with a 13SEER 4.5 ton Central AC

Summary of Results

Cadmus examined the differences between the reported savings, labeled PECI Gross Savings Reported in Table 39, and realized savings, labeled Cadmus Gross Savings Realized. Realization rates between reported gross savings and realized gross savings ranged from 98% to 510%. For a more detailed analysis, please refer to Appendix K.

Table 39. Whole-House Energy Modeling Summary Table


PECI Gross Savings Reported (kWh/unit)

Cadmus Gross Savings Using

Realized (kWh/unit)

Realization Ratio

2009 Electric Water Heater 40+ Gallons (EF 0.93 or higher) 91 110 121% 2010 Electric Water Heater 40+ Gallons (EF 0.93 or higher) 91 110 121% 2009 Windows Windows 1 2 165% 2010 Windows Windows 1 2 200% 2010 Heat Pumps Heat Pump Conversion (8.2+ HSPF) 3,147 11,457 364% 2010 Heat Pumps Heat Pump Upgrade (8.2+ HSPF) 811 1,155 142% 2009 Central Air Conditioner CAC (15 SEER) 86.1 251 292% 2009 Central Air Conditioner CAC TXV and Install 20.3 20 98% 2009 Central Air Conditioner CAC TXV and Sizing 60.3 60 99% 2009 Central Air Conditioner CAC Tune up 13 66 510% 2010 Central Air Conditioner CAC (15 SEER) 86.1 251 296% 2010 Central Air Conditioner CAC TXV and Install 20.3 20 99% 2010 Central Air Conditioner CAC TXV and Sizing 60.3 60.3 100% 2010 Central Air Conditioner CAC Tune up 13 66. 510% 2010 Evaporative Cooler Evaporative Coolers—Whole House 292 1,372 470% 2009 Insulation Insulation: Attic (R-19 +) 0.56 0.63 113% 2009 Insulation Insulation: Floor (R-19 +) 1.0 3.1 312% 2009 Insulation Insulation: Wall (R-11+ or fill cavity) 1.5 3.6 248% 2010 Insulation Insulation: Attic (R-19 +) 0.48 0.53 111% 2010 Insulation Insulation: Floor (R-19 +) 0.4 1.3 301% 2010 Insulation Insulation: Wall (R-11+ or fill cavity) 1.1 2.8 248%

Billing Analysis Billing analysis assessed actual net energy savings associated with insulation measure installations.13 Cadmus determined the savings estimate from a pooled, conditional savings (CSA) regression model, which included the following groups:

13 Billing analysis was only performed for insulation measures. Energy savings achieved through installation of

other measures were not large enough, relative to total energy consumption of households installed, to allow reliable billing analysis.



Insulation (combined attic, wall, and floor insulation for 2009–2010); and

Nonparticipant homes, serving as the comparison group.

However, the small number of insulation participants yielded net savings estimates with an unacceptably high margin of error of 86 percent on average. This was due to the combination of a small sample size and the naturally high degree of variability in billing data. Therefore, Cadmus relied on energy modeling and phone survey results in estimating savings for these measures. Methods and results of the billing analysis can be found in Appendix G.

On-Site Inspections The American Recovery and Reinvestment Act of 2009 and an influx in utility funds spurred a nationwide fear of dishonest insulation contractors, allegedly installing insulation in residences not qualifying for incentives. Additionally, several claims have emerged that “rebate chaser” contractors did not install claimed insulation amounts. The insulation site visits conducted with insulation participants sought to evaluate the quality and quantity of Rocky Mountain Power incented measures. Based on these visits, Cadmus could not conclude claimed and verified square footages differed significantly or installers underestimated or overestimated attic insulation R-Values.

Approach To verify claimed insulation savings, Cadmus completed 65 site visits at homes receiving attic, floor, or wall insulation. Specifically, the site visits sought to:

1. Verify installed insulation square footage matched that claimed in PECI’s tracking database, and ensure the maximum incentive amount did not exceed the claimed incentive.14

2. Confirm customers met HES insulation eligibility requirements, including:

a. The home was constructed before 2008.

b. The home had electric heat or gas heat, or a central air conditioning system or heat pump serving at least 80 percent of its floor area.

c. The home had preexisting wall insulation below R-10, with added wall insulation of R-11 or more.

d. The home had preexisting attic or floor insulation below R-18, with added attic or floor insulation of R-19 or more.

3. Check measure insulation install quality, specifically verifying level installed attic insulation.

To verify attic insulation R-Values, field staff first visually identified the insulation type of each layer of insulation (e.g. loose-fill fiber glass, loose-fill rock wool, loose-fill cellulose, fiber glass batt, perlite, or polystyrene). Field staff then measured the average thickness of each layer and

14 The HES Insulation Incentive Application indicates participants can receive up to $0.35 per square feet of

insulation installed.



calculated the corresponding R-Value based on an assumed R per inch for the given insulation type.15

Cadmus verified 69 insulation measures at 65 sites. Table 40 shows the number of verified measures by insulation type. The small sample sizes for wall and floor insulation limited the analysis to attic insulation. Unless otherwise noted, attic insulation results meet 90 percent confidence and 10 percent precision levels.

Table 40. Sites Verified by Insulation Type

Insulation Type Population Verified Sample* Attic Insulation 168 62 Wall Insulation 13 3 Floor Insulation 9 4 Total 190 69

*Across 65 total sites.

Field staff also collected data on the characteristics of insulation participants’ homes, as summarize in Appendix H.

Attic Insulation Findings

Claimed and Observed Insulation Square Footage

Cadmus calculated attic insulation square footage for each insulation type, and compared it to claimed square footage in PECI’s database. The 61 attic insulation sites averaged 1,308 claimed square feet; the sites’ averaged 1,298 verified square feet, or roughly an 8-square foot (or 0.5 percent) difference.16 Figure 4 shows distributions of differences between reported and verified square footage across sites.

15 Cadmus used R-values per inch assumptions consistent with Rocky Mountain Power’s Home Energy Savings

Insulation Calculator. http://homeenergysavings.net/Downloads/InsulationCalculator.pdf 16 While Cadmus visited 62 sites where participants received attic insulation, the square footage analysis used a

sample size of 61 due to field staff not gaining access to the entire attic at one site, and thus could not verify its square footage of attic insulation.



Figure 4. Verified Attic Insulation Square Feet, Compared to Claimed Footage (n=61)

At 64 percent of sites, observed square footage differed from claimed footage by less than 100 square feet. As shown above, the number of sites where verified square footage exceeded claimed roughly equaled the number of sites where claimed footage exceeded that verified.

Cadmus performed a difference of means t-test to check for a statistically significant difference between reported and verified square footage. Table 41 shows t-test results.

Table 41. Reported and Verified Square Footage Difference of Means T-Test

n Average Claimed

Average Verified

Average Difference

Standard Deviation t stat

p-value

Square Feet of Attic Insulation 61 1305.8 1298.0 7.8 125.5 0.48 0.63 As this test’s p-value did not fall below 0.10, insufficient evidence exists to conclude claimed and verified square footages significantly differed. Therefore, small observed differences could be attributed to random error.

Rocky Mountain Power allowed participants to receive incentives for attic insulation on a square-foot basis. Specifically, participants or contractors could receive incentives up to $0.35 per square foot of attic insulation installed. The insignificant differences between claimed and verified attic insulation square footage indicated, on average, Rocky Mountain Power paid correct incentive amounts.

Attic Insulation Qualification Requirements

To verify if participants met program qualification requirements, Cadmus verified heating fuel, cooling system types, home construction years, old insulation R-values, and added insulation R-values. Table 42 summarizes percentage of eligible and ineligible participants.

2%

16%

7%

18%

13% 13% 13%

18%

0%

2%

4%

6%

8%

10%

12%

14%

16%

18%

20%

By Over 300 By 100 to 299 By 50 to 99 By 1 to 49 No Difference By 1 to 49 By 50 to 99 By 100 to 299

Reported Exceeds Verified (Square Feet) Verified Exceeds Reported (Square Feet)

n = 62



Table 42. Attic Insulation Criteria*

Criteria Evaluated Percent

Precision with 90%

confidence Total In-Eligible 19 32.2% 10.0%

Gas Heating System and CAC/Heat Pump Serves < 80% of Floor Area 1 1.7% 2.8%

Pre-Existing Insulation greater than R-18 14 23.7% 9.1%

Added Insulation Less than R-19 1 1.7% 2.8%

Pre-Existing Insulation greater than R-18 and Added Insulation Less than R-19 3 5.1% 4.7%

Total Eligible 40 67.8% 10.0%

Could Not Verify 3 - -

Total Participants 62 100.0% *Three participants receiving added insulation less than R-19 also had preexisting insulation greater than R-18. **At three sites, field staff could not verify the thickness and type of insulation due to access to entire attics. Though they reviewed contractors’ paperwork at these sites to verify insulation, these sites were excluded from R-Value calculations. Claimed preexisting attic insulation R-value averaged 0.7 less than that verified. For added attic insulation, average claimed R-values exceeded verified R-Values by R 1.5. Table 43 shows average differences between claimed and verified preexisting and added attic insulation R-Values.

Table 43. Average Differences between Claimed and Verified R-Values

R-Value For n

Average Claimed Added

R-Value

Average Verified Added

R-Value Average

Difference Standard Deviation

T Stat

p-value

Pre-Existing Attic Insulation 59 13.0 13.7 0.7 6.1 0.94 0.35 Added Attic Insulation 59 35.5 34.0 -1.5 10.7 1.14 0.26

While claimed and verified R-Values for preexisting and added insulation differed, the difference was not statistically significant at the 10 percent level. In other words, insufficient evidence exists to conclude that participants or installers underestimated or overestimated attic insulation R-Values.

Attic Insulation Quality

As properly installed attic insulation should be level in most places, field staff inspected attic insulation thicknesses in multiple places at each site, reporting both minimum and maximum thicknesses. Figure 5 shows distributions of minimum/maximum thickness differences across the 59 sites.



Figure 5. Range of Added Attic Insulation Thickness (n=59)

On average, maximum attic insulation thickness exceeded minimum thickness by 6.4 inches. At 13 percent of the sites, maximum attic insulation thickness exceeded the minimum thickness by 10 inches or more.

Net Savings Approach Cadmus implemented a NTG methodology addressing the HES Program in 2009 and 2010. Freeridership and spillover comprised NTG’s two components. Freeriders—customers who would have purchased a measure without a program’s influence—reduced savings attributable to Rocky Mountain Power’s programs. Spillover—additional savings obtained by the customer’s decision to invest in additional efficiency measures or activities due to their program participation—increased savings attributable to the program, and improved program cost-effectiveness. The following formula provided final NTG ratios for each program category

Net-to-gross ratio = (1 – Freeridership) + Spillover

The freeridership component drew from a previously developed approach, which ascertained freeridership using patterns or responses of a series of six simple questions. The questions—allowing “yes”, “no” or “don’t know” responses—asked whether participants would have installed the same equipment in the program’s absence, at the same time, at the same amount, and at the same efficiency. Question response patterns were assigned freerider scores, and the confidence and precision estimates were calculated on score distributions.17

17 This approach follows methods outlined in Schiller, Steven et. al. “National Action Plan for Energy Efficiency”.

Model Energy Efficiency Program Impact Evaluation Guide. 2007. www.epa.gov/eeactionplan.



Cadmus calculated participant spillover by estimating savings attributable to additional measures installed and whether respondents credited Rocky Mountain Power with influencing their decisions. Measures counted if eligible for program incentives, but incentives were not requested. NTG ratios then were calculated, accounting for freeridership and spillover.

Appendix I provides a detailed explanation of Cadmus’ NTG methodology, including: a description of how Cadmus categorized Rocky Mountain Power’s HES Program into similar measures; an explanation of survey designs; and descriptions of Cadmus’ freeridership and spillover evaluation methodologies. It also provides: full-text versions of NTG survey questions administered to participants; the freeridership scoring matrix showing all possible combinations of responses to the freeridership survey questions; and the scores Cadmus assigned each combination.

Though this methodology could be used for evaluating NTG for appliances, windows, and insulation, it did not apply for CFLs. As the HES Program incents CFLs at the retailer level, participants do not know they have participated in a program or have purchased an incented CFL. Therefore, calculating freeridership and spillover by surveying participants is not a viable option. Cadmus triangulated results of the participant retailer surveys, the customer willingness to pay analysis, and the secondary data review to determine the CFL NTG estimate.

Summary of Results Table 44 summarizes freeridership, spillover, and NTG percentages for the HES Program. Cadmus divided the measures into two categories for this analysis: appliances and insulation. Appendix I provides a detailed explanation for why measures are separated into categories for NTG analysis.

Table 44. HES NTG Ratio

Program Category Responses (N) FR % Spillover % NTG Ratio

Precision at 90% Confidence

HES Overall Non-Lighting 334 34% 9% 76% ±5.6 Appliances 293 46% 14% 68% ±7.3 Insulation 41 15% 3% 88% ±6.5

Participants purchasing appliances showed NTG ratio of 68 percent, primarily due to the high amount of freeridership. This means that 68 percent of the gross savings for the appliance measures can be attributed to the HES Program. Participants purchasing insulation had a higher NTG ratio (88 percent) because freeridership was much lower. This means that 88 percent of the gross savings for insulation measures can be attributed to the HES Program. Because of the large difference between appliances and insulation, the NTG ratios are applied independently to the corresponding measures. The appliance measures are listed in Table 45. Insulation measures include attic, floor, and wall insulation.

Table 44 also shows the NTG ratio for the HES Program overall (76 percent), which is a savings weighted average of appliances and insulation NTG ratios. This value is applied to HES measures that were not represented in the participant survey, including central air conditioners, evaporative coolers, and heat pumps. However, it should not be used for planning purposes as it does not represent data collected specifically for those measures.



Freeridership Analysis After conducting participant surveys, Cadmus converted resulting responses into a freeridership score for each participant, using the Excel-based matrix approach described in Appendix I’s freeridership methodology section. Each participant’s freerider score derived by translating responses into a matrix value, and then using a rules-based calculation to obtain the final score. This section presents all combinations of responses received for the HES Program, and scores assigned to each combination. Figures that follow show participants’ responses rarely reflected each potential combination, but tended to group around subsets of common patterns. Freeridership scores, confidence intervals, and precision estimates were calculated for each measure category, based on distributions of scores within the matrix.

Table 45. shows freeridership calculation results for insulation and appliance measures. We discuss in-depth freeridership analysis by measure category in the sections following the table.

Table 45. HES Freeridership Results By Measure

Program Category n Freeridership

Score

Precision at 90%

Confidence HES Overall Non-Lighting 334 0.34 ± 0.03 Appliances 293 0.46 ± 0.03 Dishwasher 101 0.42 ± 0.04 Clothes Washer 102 0.45 ± 0.05 Refrigerator 69 0.50 ± 0.06 Electric Water Heater 5 0.65 ± 0.25 Fixture 3 0.50 ± 0.48 Ceiling Fans 1 0.50 N/A Windows 12 0.48 ± 0.15 Insulation 41 0.15 ± 0.06

The seven measures grouped together as appliances had an overall freeridership score of 46 percent, with an absolute precision of 3 percentage points. The average freeridership score for insulation measures was 15 percent, with an absolute precision of 6 percentage points. Combined, the average score for all HES measures was 34 percent, with an absolute precision of 3 percentage points.

Table 46 shows the unique response combinations from the HES appliance measures participant survey, the freeridership score assigned to each combination, and the number of responses for each combination.



Table 46. Frequency of Freeridership Scoring Combinations—HES Appliances

Already planning to

purchase?

Already purchased or

installed?

Installed same m

easure w

ithout incentive?

Installed something

without incentive?

Installed same

efficiency?

Installed same

quantity?

Installed at the same

time?

Would not have

installed measure?

Installed lower

efficiency?

Installed lower

quantity?


time?

Free-ridership Score

Response Frequency

Yes No Yes x Yes Partial Yes x x x x 50.0% 151 Yes Yes x x x x x x x x x 100.0% 28 No No Yes x Yes Partial Yes x x x x 25.0% 27 No Yes x x x x x x x x x 100.0% 15 Yes No Yes x Yes Partial No x x x x 0.0% 11 Yes No No Yes No Partial Yes x x x x 0.0% 8 Yes No Yes x Yes Yes Yes x x x x 50.0% 8 No No Yes x Yes Partial No x x x x 0.0% 5 Yes No Yes x Partial Partial Yes x x x x 25.0% 5 Yes No Yes x Yes Partial Partial x x x x 25.0% 4 Yes No No Yes Partial Partial Yes x x x x 25.0% 3 No No No No x x x Yes x x x 0.0% 3 No No No Yes Yes Partial Yes x x x x 25.0% 2 No No Yes x Yes Partial Partial x x x x 12.5% 2 Yes No Yes x No Partial Yes x x x x 0.0% 2 No No No Yes No Partial No x x x x 0.0% 2 Yes No Yes x No Partial No x x x x 0.0% 2 Yes No No Yes Yes Partial Yes x x x x 50.0% 2 No No No Yes Yes Yes No x x x x 0.0% 1 Yes No Yes x Partial Partial Partial x x x x 12.5% 1 Yes No No No x x x No No Partial Partial 12.5% 1 No No No Yes Partial Partial Yes x x x x 12.5% 1 No No Yes x No Partial Yes x x x x 0.0% 1 Yes No No No x x x No Partial Partial No 0.0% 1 Yes No No No x x x Yes x x x 0.0% 1 Yes No No Yes Yes Partial No x x x x 0.0% 1 No No No Yes No Partial Yes x x x x 0.0% 1 No No Yes x Yes Yes Yes x x x x 25.0% 1 No No Yes x Yes Yes No x x x x 0.0% 1 Yes No No No x x x No Yes Partial Yes 0.0% 1 Yes No No No x x x No Yes Partial No 0.0% 1

Three common patterns appeared in the respondents’ answers to freeridership questions, representing 75 percent (221 out of the 293) of the total appliance participants interviewed:

151 respondents were planning to purchase the measure before hearing about the incentive. They indicated that they would have purchased the same efficiency of measure at the same time without the incentive, but because they had not already purchased the measure when hearing about the incentive, they were considered 50 percent freeriders.

Twenty-seven respondents said they had not already purchased nor were they planning to purchase the measure when they heard about the incentive. However, they were scored



as 25 percent freeriders because they said they would have purchased the same measure at the same time without the incentive, and it would have been just as energy efficient.

Forty-three respondents had already purchased the measure when they heard about the incentive and therefore were considered 100 percent freeriders.

Different patterns emerged from the insulation participants’ answers to the freeridership questions, as shown in Table 47.

Table 47. Frequency of Freeridership Scoring Combinations—HES Insulation

Already planning to

purchase?


installed?

Installed same m

easure w

ithout incentive?

Installed something

without incentive?

Installed same

efficiency?

Installed same

quantity?


time?

Would not have

installed measure?

Installed lower

efficiency?

Installed lower

quantity?


time?

Free-ridership Score

Response Frequency

Yes No Yes x Yes Yes Yes x x x x 50.0% 7 No No Yes x Yes Yes No x x x x 0.0% 6 No No No No x x x Yes x x x 0.0% 5 Yes No Yes x Yes Yes No x x x x 0.0% 3 No No No No x x x No No No No 0.0% 3 No No No Yes Yes Yes No x x x x 0.0% 2 No No Yes x No No No x x x x 0.0% 2 Yes No No No x x x No No No No 0.0% 2 No No No Yes Yes No Yes x x x x 25.0% 1 Yes No No No x x x Yes x x x 0.0% 1 No No No No x x x No Yes No No 0.0% 1 Yes Yes x x x x x x x x x 100.0% 1 Yes No No Yes Yes Yes Yes x x x x 50.0% 1 Yes No Yes x Yes No Yes x x x x 50.0% 1 Yes No No Yes No Yes No x x x x 0.0% 1 No No Yes x Partial Yes Yes x x x x 12.5% 1 Yes No No Yes No No No x x x x 0.0% 1 No No Yes x Partial Partial No x x x x 0.0% 1 Yes No Yes x Partial Yes Yes x x x x 25.0% 1

Most notably, only one respondent who installed insulation indicated they had already purchased the measure when they heard about the incentive and therefore were considered 100 percent freeriders.

Similar to the appliance participants, the most frequent response pattern (seven out of forty-one total respondents) was for respondents who were planning to purchase the measure before hearing about the incentive. They indicated that they would have purchased the same efficiency of measure at the same time without the incentive, but because they had not already purchased the measure when hearing about the incentive, they were considered 50 percent freeriders.



Another way to compare freeridership for appliance and insulation participants is by looking at a distribution of respondents by the freeridership score assigned to each. Figure 6 and Figure 7 show the freeridership score distributions for appliances and insulation participants, respectively.

Figure 6. Distribution of Freeridership Scores – HES Appliances

Figure 7. Distribution of Freeridership Scores – HES Insulation



Approximately 68 percent of respondents installing insulation showed no freeridership, compared to 14 percent of respondents installing an appliance. Conversely, almost 70 percent of respondents installing an appliance defined as either 50 or 100 percent freeriders, compared to only 24 percent of respondents installing insulation.

Respondents were asked to explain in their own words what influence the HES incentive had on their decision to purchase the equipment. Below are a few responses for those scored as 100 percent freeriders, with the measure indicated in parentheses:

“It didn't have any influence whatsoever because I was going to buy one anyway.” (Dishwasher)

“It didn't at all. I was replacing it and it was an extra boost.” (Dishwasher)

“I had to have one either way so the incentive was a bonus.” (Clothes Washer)

“I didn't know about it when I bought it.” (Clothes Washer)

“It didn't have any influence on the washer that I bought. I would have bought with or without the energy savings program.” (Clothes Washer)

“I don't think it has any influence. We just want a good product and I go by name brand.” (Refrigerator)

“None because I didn't know about it until after.” (Windows)

Spillover Analysis This section presents a detailed analysis of additional, energy-efficient measures customers installed after participating in the HES Program. The figures below indicate, while many participants subsequently installed more energy-efficient measures after receiving a incentive from Rocky Mountain Power, only a quarter of additional purchases were reported significantly influenced by participation in the HES Program; therefore, the three-quarters not significantly influenced cannot be considered spillover. Additionally, some participants significantly influenced by the HES Program applied for incentives for additional measures they installed, and could not be included in the spillover analysis.

As detailed in Appendix I’s spillover methodology section, Cadmus used adjusted savings values from the deemed savings analysis to calculate spillover measure savings.

Cadmus calculated the spillover percentage for a program category by dividing the sum of additional spillover savings, reported by participants for a given program category, by the total incentivized gross savings achieved by all respondents in the program category.

Table 48 shows spillover analysis results for all HES appliance and insulation measures.



Table 48. Spillover Savings Analysis

Program Category Spillover Savings

(kWh) Participant Program

Savings (kWh) Spillover % HES Overall Non-Lighting 7,605 80,296 9.5% Appliances 6,792 49,583 13.7% Insulation 812 30,713 2.6%

Though indicating higher potential spillover savings, most residential participants installing additional energy-efficient equipment reported the HES Program did not have much influence on their purchasing decisions. Further, some applied for an incentive for additional measures purchased.

Table 49 summarizes numbers of participants excluded from the spillover analysis due to receiving an incentive.

Table 49. Effects of Program Influence and Incentives on HES Spillover

Program Category

Number of Measures Installed Attributable

to High Program Influence

Number of Measures Installed Not Receiving

Incentive HES Overall Non-Lighting 50 38 Appliances 38 29 Insulation 12 9

Overall, HES Program participants responding to the survey that were highly influenced by the HES Program installed 50 additional measures. Participants received incentives for 12 of these measures, leaving 38 measures qualified for spillover savings. Table 50 displays 39 additional measures HES appliance and window participants installed, qualifying as spillover. Of the 39 measures qualifying as spillover measures, insulation installed outside the HES Program is accounting for the largest proportion of spillover savings (65 percent).

Table 50. HES Appliances Spillover Measures

Spillover Measure Installed Quantity

Per Unit Electric Savings (kWh)

Total Savings (kWh)

Clothes Washers 2 370 740 Refrigerators 4 55 218 Dishwashers 5 39 196 Windows 2 212 425 Fixtures 5 68 341 Ceiling Fans 3 17 52 Electric Water Heater 4 110 440 CFLs 9 25 229 Insulation 5 993 4,963



Spillover Findings NTG analysis results showed predictable trends. Appliance participants showed higher levels of freeridership than insulation participants, consistent with what Cadmus has estimated in previous years for Rocky Mountain Power as well as for similar programs and measures at other utilities. NTG was somewhat lower for the 2009 and 2010 program years than in previous years for other states in Rocky Mountain Power’s service territory. Upcoming NTG evaluations in other states for 2009 and 2010 will provide more insight on whether decreasing NTG ratios are a trend or if Wyoming is an anomaly.

The HES Program evidences a moderate amount of participant spillover. Participant spillover develops slowly, depending on increasing familiarity with energy efficiency, and experience with program-incented measures. Because customers interviewed in 2011 participated in the HES Program during the 2009 and 2010 program years, adequate time had elapsed since program participation to yield purchases that could potentially qualify as HES Program spillover. If Rocky Mountain Power were to interview 2011 HES Program participants about the program’s influence on their additional energy-efficiency purchases, lower spillover estimation levels would likely emerge.

Freeridership is More than a Ratio Response distributions used for calculating average freeridership ratios contain information that can help program managers more effectively manage their programs. In reviewing these distributions, two interesting issues emerged.

First, it appears the HES Program’s appliance components could be even more efficient upon tightening eligibility requirements or if it was marketed differently. This study’s survey asked respondents whether they had installed equipment before hearing about the HES incentive. The 43 answering “yes” were classified as freeriders. Removing the “already installed” responses from analysis significantly lowers freerider ratio, for appliances, from 46 percent to 37 percent, as shown in Table 51.

Table 51. Effect on Freeridership of Removing “Already Installed” Responses

Program Category

With “Already Installed” Without “Already Installed”

Responses Freeridership

Score Responses Freeridership

Score Appliances 293 46% 250 37% Insulation 41 15% 40 13%

The high freeridership levels for appliance measures may relate to a relationship between an appliance’s retail cost and the incentive’s size. A recent study Cadmus conducted for a Pacific Northwest utility in the Pacific Northwest tested the hypothesis that incentive levels affect freeridership. The study graphed the proportion of total measure costs covered by the incentive with the freeridership ratio found in the analysis. As shown in Figure 8, a strong inverse relationship occurred between the proportion of the total measure cost covered by the incentive and the freeridership ratio. The graphs upper left side represents residential appliances, which typically offer small incentives relative to appliance costs. Where incentive amounts do not affect purchasing decisions, high freeridership can be expected. The trend line’s right-hand end



represents nonresidential prescriptive and grocer programs, which evidence low freeridership rates and incentives covering 60 percent of total costs, according to program records.

Figure 8. Proportion of Measure Cost Incented and Freeridership Ratio



Process Evaluation Findings

This section provides the process detailed findings for the HES Program. Findings resulted from the previously detailed extensive data collection activities, including participant surveys, program staff and market actor interviews, participant surveys, and secondary research.

Program Implementation and Delivery

Program Status The HES Program provides cash incentives to residential customers for purchases of energy-efficient products, home improvements, and heating and cooling equipment and services. According to implementation staff, the HES Program basically offers an à la carte energy-efficiency program, allowing customers to install multiple measures to create customized efficiency portfolios. The HES Program operates using the basic premise to “allow customers to pick what they need and apply for an incentive.” Accessible to all customers (even those who are not homeowners), the HES Program provides energy-saving opportunities for the entire customer community.

When the program launched in 2009 it experienced a slower-than-expected initial ramp up due to unanticipated barriers specific to Wyoming. In 2010, implementer staff employed many program design changes to adapt the HES Program to Rocky Mountain Power’s Wyoming customer base, resulting in notably increased participation during the HES Program’s second year. Based on implementation, communication, and marketing changes, implementer staff believe the program is currently: “meeting the needs of the customers who participate.” Ultimately, the HES Program seeks to help customers reduce their energy use and save money on their energy bills, thus reducing Rocky Mountain Power’s growing demand for power in the region.

Delivery Structure and Processes PECI implements the HES Program. For most qualifying program measures, customers receive incentives through a mail-in process. However, because the HES Program’s lighting component uses an upstream mechanism, PECI pays incentives directly to manufacturers of qualifying light bulbs. Local qualified retailers and contractors support the program by: upselling their customers to higher-efficiency equipment measures; installing equipment and service measures; and promoting available incentives. As part of the HES Program, Rocky Mountain Power also offers incentives to contractors for quality installation, sizing, and tune-ups of qualified HVAC measures.

According to implementer staff, PECI primarily uses an allocation system to target lighting retailers. For each retail partner location, PECI staff analyzes the customer base, assigning stores an allocation ranking, determined by the percentage of Rocky Mountain Power customers in that location. Targeted potential participating retailers must have a Rocky Mountain Power customer base of 90 percent or higher. The allocation ranking seeks to minimize leakage of incented bulbs to customers outside Rocky Mountain Power’s service territory.

Per program stakeholders, PECI staff working on Rocky Mountain Power’s HES Programs originally had not been assigned to specific states; rather; implementation staff constantly focused on all five states’ programs. In 2010, PECI began assigning staff to specific service



territories, with state management positions created to streamline the program’s implementation within each individual state. Additionally, PECI created a two-channel structure to better manage relationships with participating retailers and contractors (trade allies) in each state. PECI assigned teams to Channel managers within the retailer or contractor channel. These managers took responsibility for all relationships and activities tied to their respective delivery channels.

The contractor channel further divides, based on types of contractors working with the HES Program. For example, the program specifies two weatherization contractor segments: participating and nonparticipating. While any certified contractor can install qualified measures as long as their work meets Rocky Mountain Power’s quality standards, participating contractors attend informational meetings and undergo program training. Participating contractors can then access program promotional materials and education as to better use the program for promoting their businesses. Nonparticipating contractors do not have to attend program training.

Two types of HVAC contractors work with the program: participating and qualified HVAC contractors. All HVAC contractors undergo program training. Participating HVAC contractors sell qualified products to customers, but do not install the purchased measures. Qualified HVAC contractors, who offer installation services to customers in addition to selling qualified measures, are eligible for program dealer incentives. Qualified HVAC contractors, operating at higher standards than participating HVAC contractors, are trained to install equipment in the most efficient manner.

Initial Implementation Barriers As Rocky Mountain Power designed the Wyoming HES Program to mirror its HES Programs in Idaho, California, and Utah, the program had to overcome some Wyoming-specific barriers at the beginning of its lifecycle.

According to program staff, Wyoming’s unique retail structure presented the largest of these obstacles. Wyoming has a much smaller national retailer presence than in Rocky Mountain Power’s other service territories; so local retailers had to play a larger role in Wyoming’s HES Program. In states with more prevalent national retailers, program staff noted PECI only has to contact one representative at a retail chain’s corporate office, and the corporate representative communicates with all of the chain’s locations within the Rocky Mountain Power service territory in the respective state. In Wyoming, however, only 13 percent of the trade allies surveyed learned of the program through their corporate office (see Figure 14, below).

Additionally, Wyoming businesses tend to favor local partnerships and contacts. According to implementer staff, PECI had difficulty building relationships with retailers and contractors in Wyoming, initially because its outreach staff primarily operated out of Portland and Salt Lake City. Wyoming trade allies, even at national retailers, were hesitant to conduct business with people from outside the state. PECI had to prove to retailers and contractors that the program and the staff would remain and build trust within the community.

Wyoming’s geographically dispersed population represented another barrier specific to the territory. PECI’s field staff must drive at least two to three hours between the state’s main population centers, making in-person meetings with retailers time-consuming and expensive, although, according to program stakeholders, such meetings prove essential in building relationships with local retailers and contractors.



These initial barriers led Rocky Mountain Power and PECI to realize they had to increase field staff to raise their local presence and delivery capacity in Wyoming. Starting in 2010, PECI added locally-based staff to increase program outreach to individual retailers. Implementer staff reported needing more staff on the ground to deliver the message and engage prospective customers in the program. PECI’s field staff now visits retailers every month, recruiting new retailers, and expanding relationships with participating trade allies.

As Wyoming does not maintain a deep national chain presence and due to widely distributed population centers, PECI requires greater time and budget for field staff to visit individual stores to promote the HES Program. Program and implementer staff quickly realized this proved crucial in addressing Wyoming’s unique market barriers and ensuring the program’s success. Due to these efforts, the program has experienced significant increases in retailer and contractor participation.

Energy Independence and Security Act The Energy Independence and Security Act (EISA), an omnibus energy policy law requiring 25 percent greater efficiency for light bulbs, with new standards phased in from 2012 through 2014,18 effectively phases out the 100-, 75-, 60-, and 40-watt incandescent light bulbs currently in the market. EISA standards eventually also will phase out the current lighting savings baseline in the DSM market. Program staff noted that Rocky Mountain Power is working to diversify its lighting portfolio in response to EISA legislation, offering program incentives for all energy-efficient lighting options, including an expanded selection of specialty CFLs and LEDs. Implementer staff believe energy-efficient halogen technologies—rather than CFLs—will represent the lighting baseline in 2012 and beyond.

In-territory lighting survey responses indicated lighting customers reportedly prefer CFLs to other energy-efficient lighting options. When presented with a choice to purchase a more efficient incandescent bulb or a CFL, LED, or halogen bulb, 39 percent of lighting customers chose CFLs. Only 2 percent claimed they would purchase halogen bulbs. Figure 9 illustrates the full distribution of choices lighting customers made regarding energy-efficient lighting technologies. “Something else” responses included: “depends upon the price,” “whichever bulb is more efficient,” and “depends on where I want to use it.”

18 http://www.epa.gov/cfl/



Figure 9. Energy-Efficient Technologies Lighting Customers are Most Likely to Purchase

Rocky Mountain Power WY HES Residential Lighting Survey Question J2.

EISA Awareness Almost all participating lighting retailers (86 percent) knew of the EISA legislation (as drawn from the participating retailer/contractor survey). Three-quarters (75 percent) of participating lighting retailers aware of EISA indicated their stores have already started changing stocking practices to prepare for EISA, including phasing out incandescent inventories and increasing stocks of energy-efficient bulbs such as CFLs and LEDs. Only four participating lighting retailers reported their stores planning to educate customers about EISA, using marketing materials such as in-store displays, brochures, and flyers.

Similarly to participating lighting retailers, almost all nonparticipating lighting retailers (82 percent) expressed awareness of the upcoming EISA changes (as drawn from the nonparticipating retailer/contractor survey). Fifty-seven percent of EISA-aware nonparticipants have already started changing their stocking patterns to prepare for the legislation’s standards, ordering fewer incandescent bulbs and increasing inventories of CFL and LED bulbs. Only three nonparticipating lighting retailers planned to educate customers (through signage) about EISA legislation.

Sixty-four percent of surveyed lighting customers knew of impending EISA changes (as drawn from the in-territory lighting survey). Three of the four participating retailers receiving feedback from customers regarding EISA found customers upset about the upcoming change in lighting availability. Nonparticipating retailers also indicated they most often heard angry feedback from customers regarding EISA. One nonparticipant reported one of his customers “came in and bought enough incandescent bulbs to last the rest of his life.”



Familiarity with Energy-Efficient Lighting Options Of 248 in-territory lighting customers who responded to the familiarity questions, 78 percent recognized the terms “compact fluorescent bulb” or “CFL” before hearing the bulbs’ twisted shape description. Surveyed lighting customers primarily reported being “somewhat familiar” with CFLs (49 percent). Figure 10 illustrates familiarity with CFLs reported by surveyed lighting customers.

Figure 10. Familiarity of CFLs Among Lighting Customers* (with 90% Confidence Intervals)**

Rocky Mountain Power WY HES Residential Light Survey Question C3. * “Don’t know” responses removed from this figure. **Nearly all reported values fell within a ±10 percent interval with 90 percent confidence. To ensure an apparent uncertainty level for this analysis, the report provides confidence intervals (represented by the black line) around summary results, where appropriate. Appendix C provides a more detailed discussion of this methodology

More than half of these lighting customers (52 percent) knew of LED bulbs, though only 15 percent actually purchased LEDs for standard lighting sockets in 2009 and 2010. Eighty-nine percent of lighting customers reported replacing incandescent bulbs in their home with CFLs. Participating lighting retailers recognized customers awareness of CFL bulbs: 45 percent (multiple responses allowed) of participating lighting retailers surveyed believed of all energy-efficient products they sold, customers most commonly knew of and were likely to purchase standard CFLs without requiring additional advertising. However, that awareness may not apply to specialty CFLs: four participating retailers reported customers being least aware of specialty energy-efficient light bulbs’ availability, specifically mentioning dimmable CFLs and flood-shaped energy-efficient bulbs. Of 30 nonparticipating retailers Cadmus surveyed, 17 sold CFLs in their stores; only nine of these stores also sold specialty CFLs.

0%

10%

20%

30%

40%

50%

60%

Very Familiar Somewhat familiar

Not too familiar Not at all familiar

Percent of Responses

n = 248



Mercury Concerns HES Program staff reported observing a prevalent concern among Wyoming customers and retailers regarding CFLs’ mercury content. Program staff reported that if customers expressed discomfort with mercury in CFL bulbs, retailers did not want to increase their CFL inventories. Cadmus’ survey indicated, however, that lighting customers expressed more concerned about CFL lighting quality and performance than mercury content. As shown in Figure 11, when asked why they were “not too satisfied” or “not at all satisfied” with CFLs in their homes, 33 percent stated CFLs were not bright enough. Only three lighting customers explicitly mentioned bulb mercury content as a reason for dissatisfaction.

Figure 11. Reasons Lighting Customers are Dissatisfied with CFLs (with 90% Confidence Intervals)

Rocky Mountain Power WY HES Residential Lighting Survey Question G3.

Cadmus’ in-territory lighting survey also found that lighting customers are not utilizing the proper disposal methods for CFLs. Of the lighting customers who had a CFL burn out in their home within the past 12 months, 77 percent threw the bulb in the trash. Only 18 percent recycled the bulb appropriately. None of the surveyed lighting customers had been to the Rocky Mountain Power CFL disposal webpage to learn about proper CFL disposal.

Sixty-one percent (multiple responses allowed) of lighting customers were not concerned about CFL disposal, however, of those customers that did report having concerns, 16 percent (multiple responses allowed) mentioned mercury content and another five percent (multiple responses allowed) mentioned “dangerous chemicals” in the bulbs. Figure 12 illustrates the distribution of lighting customers’ disposal concerns.

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

Not bright enough

Short lifetime

Slow to warm up

Other Mercury content

High price


n = 49



Figure 12. Concerns with CFL Disposal among Lighting Customers (with 90% Confidence Intervals)

Rocky Mountain Power WY HES Residential Lighting Survey Question K7.

Marketing

Approach Similarly to the program’s design, program marketing materials initially drew upon the Utah HES Program. Implementer staff, quickly realizing different marketing messages might be more effective in Wyoming, developed key messages to resonate within Rocky Mountain Power’s various territories. The tone, language, and colors of marketing materials became Wyoming-focused. Implementer staff estimated the multi-purchase HES Program customer market in all five states increased by 50 percent from 2008 to 2009, and another 30 percent from 2009 to 2010 due to this marketing change.

Rocky Mountain Power and PECI participate in creating and distributing program marketing materials, with Rocky Mountain Power using bill inserts, radio ads, print ads, newspaper ads, and other print media to market the program. PECI provides point-of-purchase displays, aisle violators, incentive applications, brochures, Rocky Mountain Power-branded CFL price tags, and cling-on advertisements (product clings) for the program’s trade allies to promote the program.

During early program implementation, some larger participating lighting retailers asked Rocky Mountain Power to pay for use of end caps or aisle violators for in-store promotion, even when PECI provided the actual displays—essentially equating this to renting space on the retailer floor. However, after increasing outreach staff levels in 2010, retailers became more comfortable with PECI staff and with using HES Program promotional materials. Outreach staff worked to build better relationships with participating retailers, and retail partners no longer charged PECI to display promotional materials for HES-qualifying products.

0% 10% 20% 30% 40% 50% 60% 70%

None

Mercury

Requires special disposal

Dangerous chemicals

Don’t know how

Other

Fire hazard

Against the law to dump CFLs

Percent of Responsesn = 257



Effectiveness According to surveyed appliance customers, retailers provided the most effective program promotion avenue. Over half of participants (54 percent) first heard about the HES Program through retailers. As shown in Figure 13, customers reported bill inserts (13 percent) and print media (7 percent) as other common sources of program awareness. While only 7 percent of appliance participants equated TV or radio ads to program awareness, implementer staff noted a significant increase in program activity following radio ads. Implementer staff commented the radio ads were “almost too successful” because trade allies were unprepared for resulting demand, and “products were swept off the shelves too quickly.” Surveyed insulation customers most frequently reported learning of the program through word of mouth (32 percent) and bill inserts (16 percent).

Figure 13. How Appliance Customers First Heard about the Program

Rocky Mountain Power WY HES Retailer Participant Survey (Appliance, Windows * HVAC) Question M1.

Only 9 percent of lighting customers knew Rocky Mountain Power discounted CFLs through the HES Program. Of customers aware of the lighting discounts, 50 percent heard of the program through bill inserts. Despite this disconnect, 86 percent of participating lighting retailers mentioned receiving point of purchase marketing materials from program staff including: posters, product clings, and aisle violators. Seventy-one percent of participating lighting retailers even found these point-of-purchase materials the most useful marketing materials they received.

Surveys also found participants rarely accessed HES Program information online: only 21 percent of insulation customers, 18 percent of appliance customers, and 8 percent of lighting customers had visited the HES Website.

n=274



Trade Ally and Market Partner Promotion According to program stakeholders, trade allies proved the key to creating program awareness among customers. Implementer staff explicitly mentioned: “The area in which we could meet the needs of more customers would be to build greater awareness for the program, so that more people would participate more frequently. We need to trigger a call to action among customers.” PECI directly worked with retailers and contractors to make sure they know of the program and its incentives, providing them with promotional materials. Retailers and contractors, in turn, promoted the program to customers to increase sales of high-efficiency equipment and products.

The participant retailer/contractor surveys indicated 50 percent of participating retailers and contractors learned of the HES Program through calls or visits from HES field staff. Further, 67 percent of trade allies found HES field staff “very helpful” at addressing their needs. Figure 14 illustrates methods that trade allies reported learning about the program.

Figure 14. How Trade Allies Learned About the HES Program

Rocky Mountain Power WY HES Retailer Participant Survey Question C2.

While almost half surveyed trade allies (43 percent, multiple responses allowed) claimed they brought up Rocky Mountain Power’s incentives when assisting customers, the promotional materials provided to trade allies offered tools they found useful in reinforcing their messages. Almost all trade allies surveyed (93 percent, multiple responses allowed) reported typically informing customers of incentives available for qualifying energy-efficient products through use of promotional materials provided by program staff. Trade allies surveyed found provided product clings (33 percent, multiple responses allowed) and posters (30 percent, multiple responses allowed) the most useful marketing materials. Of seven trade allies providing suggestions for changing marketing materials, six simply wanted more. They specifically requested posters, flyers, brochures, incentive applications, and product clings. Almost one-quarter of trade allies (23 percent, multiple responses allowed) reported relying on program marketing to inform customers of available incentives. Figure 15 depicts the ways trade allies inform customers of available incentives for energy-efficient products.



Figure 15. Trade Allies’ Methods of Informing Customers of Incentives Available for Qualifying Energy-Efficient Products (with 90% Confidence Intervals)

Rocky Mountain Power WY HES Retailer Participant Survey Question E1.

Materials Review Cadmus’ high-level findings from a review of HES Program promotional materials, presented below, indicate Rocky Mountain Power’s has effectively customized and improved marketing since the program’s launch. Additionally, this upfront work proved useful in preparing the HES Program for increased marketing and promotional efforts. The following key findings resulting from the review relate to marketing materials:

Rocky Mountain Power uses a well-constructed HES strategic marketing plan: The 2010 plan includes best practice tactics, providing the appropriate media ranges and retail channels to drive participation.

WattSmart branding allows greater flexibility: The global WattSmart brand provides opportunities for cross-marketing between and within programs, and for greater customer awareness.

HES Program awareness increased in 2010: Rocky Mountain Power’s efforts to adjust the program message in response to unique Wyoming geographic, retail, and customer market characteristics helped increase program acceptance, building on increased natural awareness during the second year of marketing.

HES Program marketing collateral presents a consistent look and feel: Point of purchase, bill inserts, and other collateral consistently include uncluttered and clear designs, bold colors, and large typefaces.

HES Program marketing collateral provides consistent messaging: Marketing content for retailers and end-use customers includes basic calls-to-action and motivating messages, helping all stakeholders choose program measures and easily share information with friends, family, and colleagues.

0% 10% 20% 30% 40% 50% 60%

Did not inform customers of incentives

Mention if customer asks about efficiency

Rebate applications on display

Online/ radio/ print ads

Rely on program marketing

Product clings on qualifying appliances

Posters on the retail floor

Mention when assisting customers

Percent of Responses n = 51



Trade ally support proves critical to program success: From a marketing and customer management perspective, Rocky Mountain Power’s support for retailer channel partners has helped nurture relationships and influence eligible measure purchases.

Local staff presence increases retailer acceptance: Wyoming retail managers prefer conducting business with local contacts, and Rocky Mountain Power’s strategy of increasing locally-based field staff in Wyoming has been critical to marketing success.

Coordination with energy-efficiency organizations can be beneficial to the HES Program’s success: Developing strategic communication partnerships with government, nonprofit, and other efficiency-related organizations and programs may offer additional opportunities for Rocky Mountain Power to reach participants, especially in remote regions.

Table 52 and Table 53 compare elements in the current HES marketing plan to best practice elements in energy-efficiency program marketing. Findings indicate Rocky Mountain Power currently utilizes a significant majority of best practice marketing channels (Table 52) and the program website largely utilizes common efficiency program online marketing best practices.

Table 52. HES Program use of Best Practice Marketing Channels

Best Practice Marketing Channels HES Direct Mail √ Newspaper Ads /articles √ Radio/TV Ads √ Online Advertising √ Website √ Customer Information Sheets √ Contractor Information Sheets √ Telemarketing Bill Inserts √ Brochures √ Newsletters √ Presentations/Meetings √ Events √ Referrals √ Point of Purchase √ Branded Promo Items √ Tests/Demonstrations √ Social Media Outreach * Via Rocky Mountain Power

*Social media (e.g., Twitter, Flickr, YouTube, Facebook) offers channels for utilities to connect with customers. Most utilities leverage one or more social media platform(s) in their communication efforts.



Table 53. HES Program use of Website Best Practices

Website Best Practice Element HES Program highlighted on Rocky Mountain Power home page Yes Number of Clicks from Rocky Mountain Power home page 2 or 3 Description leads with benefits (i.e., What's in it for the participant?) WattSmart Programs and

Incentives or Save Energy Message consistency from Rocky Mountain Power home to subpage Yes Clear call to action Strong and active Many access points Yes Contact capture No Description of each individual program offered Yes Participant eligibility requirements Yes Contractor participation and eligibility requirements Available via phone inquiry Contractor Listing Yes Contractor Search Engine No Online Contractor Application Process No Downloadable Incentive Forms Yes Online Incentive Application Process No Downloadable program information in print format for contractors to share with customers No HES Social Media elements included (e.g. Facebook, Twitter, etc.) No

Quality Assurance PECI conducts on-site QC inspections on 5 percent of all HVAC and weatherization installations, to ensure “service measure” installations have been conducted to HES Program standards.

PECI also performs quality inspections at all participating retail locations. In the past, PECI received several incentive applications for unqualified products because a participating retailer used program marketing materials to promote an incorrect product. To address this, PECI implemented QA protocols, whereby participating retailers became responsible for correctly displaying all provided promotional materials. PECI visits each store to ensure marketing materials are up to date, take pictures of all displayed promotions, and to confirm appropriate marketing materials are on display. PECI also checks correct displays of prices and Rocky Mountain Power’s logo, and verifies products on display are the actual qualified products.

Rocky Mountain Power’s call center handles customer complaints, with call center agents attempting to resolve issues on the first call. If customers have more serious complaints, the call agent contacts program managers at Rocky Mountain Power or PECI. The agent provides all customer complaint correspondence to Rocky Mountain Power’s regulatory group for record. PECI program staff personally call customers back to resolve their issue. Customer complaints regarding participating trade allies are taken very seriously. If several customers complain about a trade ally, PECI informs Rocky Mountain Power, which usually removes the retailer or contractor as a promotional partner. In extreme cases, Rocky Mountain Power may take legal action against the trade ally in question.

A customer may also complain to the public utilities commission. In such cases, Rocky Mountain Power takes a more formal approach. PECI provides all customer correspondence data to Rocky Mountain Power’s regulatory group. Correspondence data includes any e-mails, phone



conversations, meeting dates, and meeting summaries involving any party involved in the complaint. Rocky Mountain Power’s regulatory group then coordinates the customer complaint with the commission until full resolution of the issue.

Customer Response

Satisfaction Participating customers expressed strong satisfaction with the incentives’ timing and amounts (as drawn from the participant telephone surveys). More than half of participating appliance/window customers (51 percent) received incentive payments within four to six weeks of submitting their incentive applications, and 23 percent received payments in less than four weeks. Eighty-eight percent of appliance/window customers reported satisfaction with the amount of time required to receive their incentive check in the mail, and 86 percent expressed satisfaction with the incentive’s amount. Similarly to appliance customers, just under half of insulation customers (48 percent) reported receiving incentive payments within four to six weeks. Seventy-seven percent insulation customers expressed satisfaction with the time required to receive payment from Rocky Mountain Power, and all but one (98 percent) expressed satisfaction with the incentive amount received.

Customers also expressed strong satisfaction with measures they purchased through the HES Program. Ninety-five percent of participating appliance/window customers reported being “very” (72 percent) or “somewhat” (23 percent) satisfied with measures they purchased through the HES Program. Seventy-eight percent of lighting customers were “very” (49 percent) or “somewhat” (29 percent) satisfied with CFLs currently installed in their homes. Seventy-five percent of lighting customers were “very” (50 percent) or “somewhat” (25 percent) satisfied with LED bulbs they purchased in 2009 and 2010. Eight-six percent of insulation customers reported being “very satisfied” with attic or wall insulation they installed in their homes.

One implementer staff expressed: “We are currently meeting expectations of customers, but we are not exceeding expectations. I would like to be doing better than that.” However, surveyed participants generally expressed high customer satisfaction levels. Ninety-five percent of participating participants were “very” (56 percent) or “somewhat” (39 percent) satisfied with their overall experience with the HES Program, as shown in Figure 16.



Figure 16. Customer Satisfaction

Rocky Mountain Power WY HES Retailer Participant Survey (Appliances, Windows & HVAC) Question F7 and Rocky Mountain Power WY HES Participant Telephone Survey (Insulation) Question E8.

Barriers

Perceptions Regarding Energy Efficiency During the management staff and partner interviews, HES Program staff expressed low energy costs, coupled with a lack of awareness regarding energy efficiency in Wyoming, presented a participation barrier for the program. Program staff cited a lack of awareness about the programs, and Wyoming customers’ resistance to the concept of energy efficiency. The state’s very low electric and natural gas rates contributed to customers’ lack of concern about energy efficiency as they normally received low utility bills. Program staff indicated most homes did not have central air conditioning, and natural gas—abundant and very inexpensive in the state—provided the most prominent heating fuel, resulting in low utility costs. Thus, customers were not motivated to insulate their homes for winter or improve air conditioning systems during summer.

Despite these perceptions, 93 percent (multiple responses allowed) of trade allies believed customers understood the energy-related benefits of higher-efficiency products. Almost three-quarters of nonparticipant retailers (71 percent, multiple responses allowed) claimed they sold CFLs as a market existed. Another 35 percent multiple responses allowed) claimed they sold CFLs because their customers were energy conscious.

The majority of participating trade allies (60 percent, multiple responses allowed) and almost half of nonparticipating retailers and contractors (47 percent, multiple responses allowed) believed potential energy cost savings tended to sell high-efficiency products. Forty-four percent (multiple responses allowed) of trade allies and 71 percent (multiple responses allowed) of nonparticipating retailers and contractors reported energy savings and environmental benefits as selling points for energy-efficient products.

56%

39%

4%

2%

Very satisfied

Somewhat satisfied

Not very satisfied

Not at all satisfied

n=331



Surveys of participating customers confirmed opinions of program trade allies. Almost two-thirds of surveyed trade allies (63 percent) reported typical customer as “somewhat interested” when told about the energy-saving potential of ENERGY STAR appliances. Participating insulation customers reported the highest percentage of energy-related drive, with 73 percent (multiple responses allowed) citing they were motivated to install insulation to save energy or reduce energy costs. While 29 percent (multiple responses allowed) of participating appliance customers were motivated to purchase high-efficiency equipment to save energy or reduce energy costs, 38 percent (multiple responses allowed) simply needed new equipment. Various reasons for needing new equipment included: “old equipment wasn’t working,” “old equipment was working poorly,” “never owned the [measure] before,” and “needed all new appliances because of a new home.” Figure 17 illustrates the full distribution of both appliance and insulation customers’ purchasing motivations.

Figure 17. Factors that Motivated Appliance and Insulation Customers to Purchase a Qualifying Measure* (with 90% Confidence Intervals)

* “Don’t know” responses removed from this figure. Rocky Mountain Power WY HES Participant Telephone Survey (Appliance, Windows & HVAC) Question M4 and Rocky Mountain Power WY HES Participant Telephone Survey (Insulation) Question C4.

Surveyed trade allies’ opinions split regarding whether energy efficiency offered a useful tactic to promote their businesses. Slightly over half, of trade allies (53 percent) used the availability of high-efficiency products to attract customers to their business, while 47 percent did not.

Incremental Costs Nonparticipating retailers and contractors reported incremental costs associated with energy-efficient technologies as a participation barrier. While 67 percent of nonparticipating retailers and contractors believed customers were “very aware” of the benefits of purchasing high-efficiency appliances, the majority of nonparticipants (82 percent) surveyed believed customers chose not to purchase high-efficiency appliances due to less expensive alternatives in the market. Eighty-nine percent of nonparticipants reported price as the most significant market barrier

0% 10% 20% 30% 40% 50% 60%

Advertisement

Other

Environmental concerns

Maintain or increase comfort of home

Recommended by friend/ retailer/ contractor

Price

Program incentive

Features/ brand of new equipment

Wanted to save energy/ reduce energy costs

Old product wasn't sufficient or was missing

Percent of Responses n =398



affecting sales of high-efficiency appliances. Only one nonparticipant cited awareness as a market barrier for high-efficiency product sales.

Communication To ensure program success, PECI communicates with program staff and trade allies through channel teams. The retailer channel offers close relationships with store staff at every location, where PECI focuses on exciting stores about the program and disseminating information to as many qualified retailers as possible. The retailer channel also offers field staff to conduct on-the-ground outreach to store staff, and managers to make sure staff understand all program aspects.

PECI’s contractor channel works similarly to the retailer channel. The team meets contractors in the field, training them on how to talk to customers about the program and promote program measures. Within the contractor channel, a team reaches out to contractors to inform them of the program and attempts to recruit new participants. Program staff at Rocky Mountain Power and PECI agree the channel structure has been implemented as a very effective communication tool.

Of nonparticipating retailers and contractors Cadmus surveyed, 37 percent had heard about the HES Program. Of 11 nonparticipants aware of the program, three had been targeted by PECI’s outreach staff through field visits or phone calls. The remaining nonparticipants reported hearing about the program through word of mouth and through Rocky Mountain Power’s marketing efforts (outside of in-store promotional displays); these included bill inserts and other media. When asked about the best ways to contact their store about Rocky Mountain Power’s programs and services, 83 percent of nonparticipating retailers and contractors surveyed reported calls from field staff as the most effective method. Figure 18 shows effective communication methods reported by nonparticipants.

Figure 18. Reported Best Ways to Contact Nonparticipating Retailers and Contractors about Rocky Mountain Power Programs and Services (with 90% Confidence

Intervals)

Rocky Mountain Power WY HES Retailer Nonparticipant Survey Question C5.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Call from HES field staff

E‐mail or fax Mail package of marketing materials

Corporate office


n = 35



Document Review PECI maintains QC protocols for both HES Program paths: the CFL promotion, and the residential incentives. PECI provided the following documents and files for Cadmus’ QC review:

CFL QC Processes Overview;

Rocky Mountain Power HES Program Implementation Manual;

A blank copy of the Attachment A Approval Form;

A copy of the Memorandum of Understanding (MOU) Audit data for agreements active during 2009 through 2010; and

A copy of the Wyoming Inspection Report for insulation and window incentives during the 2009 through 2010 program period.

Cadmus reviewed these QC procedures and documents to determine whether PECI follows its protocols.

CFL Promotion PECI executes a memo of understanding (MOU) with all program partners to define program requirements, including an attachment (designated as “Attachment A”) for each state in which the partner operates. These attachments include a list of products, stores, incentive levels, and active dates. When products or stores change, PECI drafts a new attachment, which supersedes the previous attachment. Prior to executing the MOU, PECI uses a database to check all products and locations comply with tariffed program requirements. Additionally, PECI senior staff conduct secondary reviews of attachments using a checklist to ensure compliance with program requirements, and approving them only if information on the attachment meets all check-listed criteria. PECI retains a hard copy of each Attachment A and its corresponding approved checklist.

Cadmus reviewed a copy of the MOU Audit database for agreements active from 2009 through 2010. PECI provided the data as an Excel workbook, with separate worksheets for product and location data. Of 1,747 products recorded, approximately 5 percent were marked as “Out of Tariff”; of 1,063 locations recorded, approximately 3 percent were marked “Out of territory.” This indicates PECI reviewed the product and location data before executing the MOUs.

Residential Incentives PECI processes incentive applications for the HES Program’s residential incentives path. QC protocols include reviewing the application to determine whether:

The incentive application is not a duplicate;

The customer has not exceeded any program cap or incentive limit; and

The customer received the appropriate incentive amount based on program guidelines.

For all applications with a total incentive amount over $100, data entered are reviewed against the physical application to ensure they have been entered into the system accurately.



In addition, PECI requires internal program management approval for any applications received more than 90 days after the date work was completed or services purchased. If customers request their check mailed to an address other than that of the installation, PECI checks to ensure the check has the correct address. Finally, any incentives targeted for inspection by program management must have an approved inspection on file, or must have been approved by program management for processing without inspection.

PECI must conduct inspections of 5 percent of all submitted applications for weatherization measures and HVAC service work, and must meet this quota for each measure in each state. PECI maintains an inspection database, and coordinates sites each inspector visits per week to maximize travel and logistical efficiencies. Inspectors collect data from installation sites to ensure site details match those reported on the application, recording these data in inspection worksheets later uploaded into the inspection database.

Cadmus reviewed a copy of the Wyoming Inspection Report for insulation and window incentives provided by PECI as an Excel workbook. Inspection dates ranged from August 19, 2009, through December 22, 2010. Of 5,428 records in the database, approximately 700 were marked as “denied,” for reasons such as: no electric heat or central air conditioning; insufficient percentage of square footage cooled by cooling unit; or existing R-Value of insulation in excess of program qualifications. This indicates PECI observes its protocol to inspect installations and make certain they meet program requirements.

Summary and Recommendations Following a slower-than-expected start during the HES Program’s first year, Rocky Mountain Power implemented several changes in 2010 to program operations, delivery structures, and marketing approaches which led to a significant improvement in participation and savings. Conclusions and recommendations have been drawn from process evaluation interviews, surveys, and other analysis conducted. While Cadmus’ process evaluation found several aspects of HES Program operations and delivery have improved, Cadmus believes the program may benefit from additional changes as the HES Program matures and continues adapting to the Wyoming market.

Some of the following conclusions include recommendations, while others indicate the current approach appears to be working well.

Program Design and Implementation Management of the retailer and contractor delivery channels within PECI provides the

structure for communication and program success among program implementers and trade allies. PECI’s revised delivery structure has reduced many of the HES Program’s initial implementation barriers by streamlining program staff responsibilities, building relationships with retailers and contractors, and increasing the total number of trade allies promoting the program to end-use customers.

More Wyoming-based outreach staff are in the field to meet with individual retailers. Field staff visit participating and potential trade allies more frequently than initially anticipated. The strategy is proving helpful in terms of increasing retailer and contractor



participation, especially in a diverse retailer and contractor market like that found in Wyoming.

Energy Independence and Security Act (EISA) Awareness and Concerns Lighting customers prefer CFLs to other energy-efficient lighting options. When

presented with choices to purchase a more efficient incandescent bulb, CFL, LED, or halogen bulb, more than one-third of lighting customers chose CFLs.

Lighting retailers do not plan to educate customers about the EISA legislation. Very few lighting retailers (participating or nonparticipating) reported plans to educate customers about upcoming changes in availability of lighting options resulting from EISA.

o Recommendation: Due to the lack of preparation by retailers, Rocky Mountain Power should consider providing educational materials to customers regarding EISA, in the context of increased availability of utility-supported, high-efficiency lighting options, preparing customers for upcoming changes in lighting availability.

Customers do not know of lighting options provided by specialty CFLs. Of all energy-efficient products retailers sell, customers know the least about specialty CFLs.

o Recommendation: EISA informational materials should highlight the increased variety of discounted lighting options offered by Rocky Mountain Power’s HES Program, including specialty CFLs, LEDs, and halogen options.

Despite program staff perceptions, Wyoming lighting customers’ most prevalent complaint with CFLs is not their mercury content. Customers’ most frequently reported sources of dissatisfaction with CFLs included a lack of brightness and CFLs not living up to advertised lifetimes.

Lighting customers do not know of proper CFL disposal methods. Whether customers feel they do not have access to recycling centers or they simply do not know how to best dispose of CFL bulbs, more than three-quarters of surveyed CFL owners threw a CFL bulb in the trash during the past 12 months.

o Recommendation: Provide recycling centers at all participating retail locations, so customers can simply bring in spent bulbs when purchasing replacements. To raise aware of the program, the look and feel of these centers should be similar to HES in-store promotional materials.

Due to EISA’s phase out of incandescent bulbs, much uncertainty surrounds the potential new lighting savings baseline in the DSM market. The robust variety of EISA-compliant bulbs that program stakeholders report Rocky Mountain Power plans to offer through its lighting portfolio effectively ensure garnering savings from lighting programs.

Marketing and Participation Decisions The creation of territory-focused marketing messages benefits program participation.

The program implementer estimated that moving from a “one-size-fits-all” marketing



campaign to messaging that targets local territories increased multi-purchase HES results by 50 percent from 2008 to 2009, and another 30 percent from 2009 to 2010.

Trade allies serves as the program’s most valuable tool to increase program awareness. Over half of appliance participants learned of the HES Program through a retailer. One staff interviewee said success was “about relationships; not mass marketing.”

o Recommendation: Continue to recruit new trade allies to broaden program awareness throughout the service territory. An increased trade ally network will effectively lead to heightened incentive awareness and further increase program participation.

o Recommendation: Contractor knowledge largely drives participation. While Rocky Mountain Power has done an exceptional job of recruiting and communicating with program stakeholders, developing a formal trade ally program may further increase trade ally awareness and improve existing trade ally perceptions of program value.

o Recommendation: Explore new strategic partners. Website links and co-branding with like-minded partners can increase awareness and earned exposure. Rocky Mountain Power should reach out to government and non-profit organizations focused on energy efficiency and environmental education, exploring ways to broaden their messaging reach.

According to trade allies, product clings and posters serve as the most useful promotional materials provided through the HES Program. Of the seven trade allies providing suggestions for changing the marketing materials, six simply wanted more materials.

o Recommendation: Provide a greater quantity of materials for trade allies’ effective promotional use. Develop a Web-based tool allowing trade allies to place orders for these materials as needed.

Trade allies were unprepared for the increased program activity resulting from radio advertisements. Implementer staff commented radio ads were “almost too successful” because trade allies were unprepared for resulting demand, and “products were swept off the shelves too quickly.”

o Recommendation: To better capitalize on potential participation resulting from marketing investments – radio advertisements in particular – notify retail partners well in advance of planned promotions so that they are prepared with adequate inventory to meet demand.

Customers do not connect upstream lighting products purchased with Rocky Mountain Power’s HES Program incentives. Although most HES Program savings accrue through the lighting component, very few lighting customers know Rocky Mountain Power’s HES Program provides for the CFL discounts.

o Recommendation: Rocky Mountain Power should provide more in-store marketing materials focused on lighting, reinforcing the message that the utility discounts qualifying lighting products at the register.



Rocky Mountain Power has effectively created compelling, broad-reaching marketing materials. Cadmus understands marketing represents a key lever for controlling program participation. The utility’s marketing materials, use of marketing channels, and its online presence largely remain consistent with utility program best practices. However, if Rocky Mountain Power wishes to fill the participation gap left by a slower-than-expected first-year ramp up, it should implement new, additional marketing strategies.

o Recommendation: Continue to leverage “one-to-many” opportunities. “Road Shows” and event exposure can reach rural customers cost-effectively. Invitations to road shows and/or event sponsorships can themselves offer effective marketing opportunities, outlining the program value proposition. Events targeted to trade allies, a highly qualified and motivated audience base, can be particularly effective.

o Recommendation: Track metrics. Metrics can help Rocky Mountain Power assess its return-on-marketing investment and fine-tune marketing resource allocations. In-house marketing evaluations can include: interviews; focus groups; sampling (i.e., surveys prior to and during the campaign); broadcast-generated impressions (i.e., assess the total number of customers registering the brand message through print, radio, outdoor, sponsorship, and TV marketing; customer engagement/CRM Reports; on-line activity (i.e., social media analytics, Web traffic/visits, total impressions, time on site, click path [how a visitor gets to a site], bounce rates, page exit percentages, and other relevant data points); and conversion (when coding marketing material, metrics provide performance of call-to-action offers).

o Recommendation: Per the 2010 outreach plan, leverage on- and off-line social networks. Social network distribution could be provided on-line and in-person. These groups (such as stakeholder trade associations, community networks, Chambers of Commerce, LinkedIn groups, and e-mail networks) could provide low-cost, high-volume information distribution vehicles.

o Recommendation: Per the 2010 outreach plan, promote the program’s URL. Customers do not utilize the HES Website, though on-line marketing can be one of the most cost-effective tools to generate interest and leads in remote geographies. Rocky Mountain Power should emphasize its Website in marketing materials as a key tool for obtaining detailed program information. However, marketing channels should continue to focus on approaches reported as most effective with customers: bill inserts and in-store displays.

Satisfaction and Perceived Barriers Program satisfaction generally runs high. All surveyed customers reported high

satisfaction levels regarding program incentives, purchased measures, and overall program experience.

The need for new equipment most often motivates participating appliance customers to purchase qualified measures. Many participants reported participating in the HES Program as their old equipment was not working or needed to be replaced.



o Recommendation: Offer an additional incentive to customers for early replacement of equipment. Additional savings may be achieved if customers participate sooner.

Benefitting from energy savings motivates customers to purchase high-efficiency products. Participating and nonparticipating retailers and contractors most frequently reported potential energy cost savings as the “selling point” for higher-efficiency products.

o Recommendation: Leverage customer’s interest in saving energy by providing trade allies with materials focusing on potential energy cost savings associated with qualified measures. Information could include estimated annual cost savings as compared to the use of a standard efficiency model using accurate Rocky Mountain Power rates.



Cost-Effectiveness

In assessing cost-effectiveness, Cadmus analyzed program costs and benefits from five different perspectives, using Cadmus’ DSM Portfolio Pro19 model (as used for recent evaluations of Rocky Mountain Power’s residential portfolio). Benefit-to-cost ratios conducted for these tests were based on methods described in the California Standard Practice Manual (SPM) for assessing DSM programs’ cost-effectiveness. Tests utilized included the following:

a. PacifiCorp Total Resource Cost Test (PTRC): This test examined program benefits and costs from Rocky Mountain Power’s and Rocky Mountain Power customers’ perspectives, combined. On the benefit side, it included avoided energy costs, capacity costs, and line losses, plus a 10 percent adder to reflect non-quantified benefits. On the cost side, it included costs incurred by both the utility and participants.

b. Total Resource Cost Test (TRC): This test examined program benefits and costs from Rocky Mountain Power’s and Rocky Mountain Power customers’ perspectives, combined. On the benefit side, it included avoided energy costs, capacity costs, and line losses. On the cost side, it included costs incurred by both the utility and participants.

c. Utility Cost Test (UCT): From Rocky Mountain Power’s perspective, benefits included avoided energy, capacity costs, and line losses. Costs included program administration, implementation, or incentive costs associated with program funding.

d. Ratepayer Impact (RIM): All ratepayers (participants and nonparticipants) may experience rate increases designed to recover lost revenues. This test included all Rocky Mountain Power program costs as well as lost revenues. Benefits included avoided energy costs, capacity costs, and line losses. Most programs do not pass the RIM test due to the adverse impact of lost revenue.

e. Participant Cost Test (PCT): From this perspective, program benefits included bill reductions and incentives received. Costs included a measure’s incremental cost (compared to the baseline measures), plus installation costs incurred by the customer.

Table 54 summarizes the five tests’ components.

Table 54: Benefits and Costs Included in Various Tests

Test Benefits Costs PTRC Present value of avoided energy and capacity costs*

with 10 percent adder for non-quantified benefits Program administrative and marketing cost

TRC Present value of avoided energy and capacity costs* Program administrative and marketing cost UCT Present value of avoided energy and capacity costs* Program administrative, marketing, and

incentive cost RIM Present value of avoided energy and capacity costs* Program administrative, marketing, and

incentive cost + present value of lost revenues PCT Present value of bill savings and incentives received Incremental measure cost and installation cost

19 DSM Portfolio Pro has been independently reviewed by various utilities, their consultants, and a number of

regulatory bodies, including the Iowa Utility Board, the Public Service Commission of New York, the Colorado Public Utilities Commission, and the Nevada Public Utilities Commission.



*Present value of avoided energy and capacity costs includes avoided line losses occurring from reductions in customer electric use.

Table 55 provides selected cost analysis inputs, including: evaluated energy savings for each year, discount rate, line loss, and program costs. Rocky Mountain Power provided all of these values, except energy savings. The discount rate derived from Rocky Mountain Power’s 2008 Integrated Resource Plan. Rocky Mountain Power also provided the values for line loss and program costs.

Table 55. Selected Cost Analysis Inputs*

Input Description 2009 2010 Total Program Net Savings (kWh/year) 2,796,158 4,210,524 7,006,682 Discount Rate 7.40% 7.40% 7.40% Line Loss 9.26% 7.96% NA Inflation Rate 1.90% 1.90% 1.90% Total Program Costs $247,252 $396,108 $643,360

Program Management Costs $225,093 $360,609 585,702 Utility Administrative Costs $22,159 $35,499 57,658

*Savings reflect impacts at generation and have been increased for line losses.

Program benefits included energy savings and their associated avoided costs. The cost-effectiveness analysis used energy savings derived from this study’s evaluated kWh. Analysis used a weighted average measure life of 7.7 years, based on the measures’ lifetimes, and weighted by savings and frequency of installations. All analyses used avoided costs associated with Rocky Mountain Power’s 2008 IRP 46 Percent Load Factor Eastside Residential Whole Home Decrement.20

Cadmus analyzed cost-effectiveness for two scenarios. The first assumed zero percent freeridership and spillover (NTG equaling 100 percent). The second incorporated evaluated freeridership and spillover.

Table 56, below, presents program cost-effectiveness analysis results with NTG equaling 100 percent for all program measures for the evaluation period (2009–2010), though not accounting for non-energy benefits (except those represented by the 10 percent conservation adder included in the PTRC). For this scenario, cost-effectiveness analysis results indicated the program was cost-effective from all perspectives except the RIM (a 1.0 or greater benefit-cost ratio is considered cost-effective).

20 IRP decrements are detailed in Appendix G of PacifiCorp’s 2008 Integrated Resource Plan, Vol. II Appendices:

http://www.pacificorp.com/content/dam/pacificorp/doc/Environment/Environmental_Concerns/Integrated_Resource_Planning_6.pdf



Table 56. Program Cost-Effectiveness Summary for 2009–2010 (NTG = 100 percent)


Levelized

Costs Benefits Net


Total Resource + Conservation Adder (PTRC) $0.057 $2,437,188 $3,538,533 $1,101,345 1.45 Total Resource No Adder (TRC) $0.057 $2,437,188 $3,216,848 $779,660 1.32 Utility (UCT) $0.026 $1,112,945 $3,216,848 $2,103,903 2.89 Ratepayer Impact (RIM) $0.108 $4,623,053 $3,216,848 ($1,406,205) 0.70 Participant (PCT) $0.042 $1,819,961 $4,006,986 $2,187,025 2.20

Table 57 presents program cost-effectiveness analysis results including evaluated NTG for all program measures for the evaluation period (2009–2010), though not accounting for non-energy benefits (except those represented by the 10 percent conservation adder included in the PTRC). For this scenario, cost-effectiveness analysis results indicated the program was cost-effective from all perspectives except the RIM (a 1.0 or greater benefit-cost ratio is considered cost-effective).

Table 57. Program Cost-Effectiveness Summary for 2009–2010 (Evaluated NTG)


Levelized

Costs Benefits Net


Total Resource + Conservation Adder (PTRC) $0.055 $2,131,526 $3,122,975 $991,449 1.47 Total Resource No Adder (TRC) $0.055 $2,131,526 $2,839,068 $707,542 1.33 Utility (UCT) $0.029 $1,112,945 $2,839,068 $1,726,123 2.55 Ratepayer Impact (RIM) $0.110 $4,278,165 $2,839,068 ($1,439,097) 0.66 Participant (PCT) $0.042 $1,819,961 $4,006,986 $2,187,025 2.20

Table 58 presents program cost-effectiveness analysis results including evaluated NTG for all program measures for the 2009 evaluation period, though not accounting for non-energy benefits (except those represented by the 10 percent conservation adder included in the PTRC). For this scenario, cost-effectiveness analysis results indicated the program was cost-effective from all perspectives except the RIM (a 1.0 or greater benefit-cost ratio is considered cost-effective).

Table 58. Program Cost-Effectiveness Summary for 2009 (Evaluated NTG)


Levelized

Costs Benefits Net


Total Resource + Conservation Adder (PTRC) $0.051 $799,725 $1,132,188 $332,463 1.42 Total Resource No Adder (TRC) $0.051 $799,725 $1,029,262 $229,536 1.29 Utility (UCT) $0.027 $425,066 $1,029,262 $604,195 2.42 Ratepayer Impact (RIM) $0.107 $1,685,598 $1,029,262 ($656,336) 0.61 Participant (PCT) $0.038 $653,916 $1,552,531 $898,614 2.37

Table 59 presents program cost-effectiveness analysis results including evaluated NTG for all program measures for the 2010 evaluation period, though not accounting for non-energy benefits



(except those represented by the 10 percent conservation adder included in the PTRC). For this scenario, cost-effectiveness analysis results indicated the program was cost-effective from all perspectives except the RIM (a 1.0 or greater benefit-cost ratio is considered cost-effective).

Table 59. Program Cost-Effectiveness Summary for 2010 (Evaluated NTG)


Levelized

Costs Benefits Net


Total Resource + Conservation Adder (PTRC) $0.058 $1,430,354 $2,138,105 $707,751 1.49 Total Resource No Adder (TRC) $0.058 $1,430,354 $1,943,732 $513,378 1.36 Utility (UCT) $0.030 $738,782 $1,943,732 $1,204,950 2.63 Ratepayer Impact (RIM) $0.112 $2,784,417 $1,943,732 ($840,685) 0.70 Participant (PCT) $0.045 $1,252,332 $2,636,085 $1,383,753 2.10



Appendices

Appendix A – Survey and Data Collection Instruments

Appendix B – Precision Calculations

Appendix C – HOU Methodology

Appendix D – Lighting NTG (Retailer Surveys)

Appendix E – Lighting NTG (Secondary Review)

Appendix F – Lighting NTG (WTP)

Appendix G – Billing Analysis

Appendix H – Insulation Site Visits

Appendix I – NTG Evaluation Methodology

Appendix J – Marketing Materials

Appendix K – Engineering Review and Whole House Modeling

Please find the appendices to this report attached in a separate file.

Rocky Mountain Power Wyoming Evaluation 2009-2010 Report October 14, 2011

The Cadmus Group, Inc. / Energy Services Appendix B1

Appendix B. Precision Calculations

To determine the level of uncertainty for results, Cadmus considered the effect of sampling error on all estimates presented in the report. Sampling error refers to the uncertainty that is introduced by the use of sampled data to infer characteristics of the overall population. These data include survey results, meter data, and those from secondary sources. Cadmus used sampled data to estimate parameters for per-unit savings calculations (such as installation rates) or in the consumption of specific equipment types (such as in billing analysis).

Sampling error is reflected in confidence intervals about the estimates. Unless otherwise noted, Cadmus estimated intervals at 90 percent confidence; meaning that we can be 90 percent confident that the true population value lies within the given interval. Cadmus calculated confidence intervals for means, proportion, regression estimates, and any calculated values that used sample estimates as an input. Cadmus calculated all confidence intervals using standard formulae to estimate uncertainty for proportions and means. For mean values, we used the following formula:

1.645 ∗

Where s2 is equal to the sample variance and 1.645 is the z-score for a 90 percent confidence interval.

In some cases, uncertainty of the estimates came from several sources. For example, in cases of summed estimates, such as those for total program savings, the root of the sum of the squared standard errors was calculated to estimate the confidence interval1:

1.645 ∗

In some cases, Cadmus multiplied estimates. For instance, net savings calculations involve combining gross estimates with an in-service rate and/or NTG ratio estimated from participant surveys. For these results, Cadmus calculated combined standard errors for the final estimates. In cases where the relationship was multiplicative, Cadmus used the following formula2:

∗ ∗ 1.645 ∗

1 This approach to aggregation errors follow the methods outlined in Appendix D from Schiller, Steven et. al.

“National Action Plan for Energy Efficiency”. Model Energy Efficiency Program Impact Evaluation Guide. 2007. www.epa.gov/eeactionplan.

2 Derived from Goodman, Leo, "The Variance of the Product of K Random Variables," Journal of the American Statistical Association. 1962.


The Cadmus Group, Inc. / Energy Services Appendix B2

In some cases a ratio of two estimates was needed. An example of this is in the estimation of the spillover ratio, expressed as the ratio of spillover savings to program savings. For this calculation, Cadmus used the following formula3:

/ 1.645 ∗

To ensure transparency of the error aggregation process, Cadmus reported precision for both individual and combined estimates where relevant.

3 Stuart, A. and Ord, J. Kendall’s Advanced Theory of Statistics (6th Edition). Edward Arnold. 1998.


The Cadmus Group, Inc. / Energy Services Appendix C1

Appendix C. Hours of Use Methodology

Cadmus estimated CFL hours of use (HOU) using a multistate modeling approach, built on light logger data collected from four states: Missouri, Michigan, Ohio, and Maryland. Cadmus chose these data instead of those from the most recent California evaluation for a number of reasons:

1. These states are more comparable to Wyoming in terms of latitude (a factor in seasonal variation in daylight hours).

2. These states all have relatively new CFL programs (unlike California, where programs have been in place for a number of years).

3. These states have a more comparable distribution of urban versus rural population (as shown in Figure C1).

Figure C1. Urban vs. Rural Comparison between States

Source: 2000 US Census

Metering Protocol Following whole-house lighting audits, Cadmus installed up to five loggers in each participant home. Metering periods varied by utility, ranging from three months to one year. For homes where we identified five or fewer CFL fixture groups, Cadmus field staff installed light loggers on every CFL fixture. For homes with more than five CFL fixture groups, field staff randomly selected which five fixtures to meter. This method relied on a systematic sampling that involved installing a logger on every nth CFL fixture (the nth number also based on the number of total possible CFL fixtures and is randomly generated).

During the logger removal process, field staff collected additional data for evaluating data quality and for determining if loggers had failed, tampered with, or removed. Moreover, prior to

94%

77%65%

6%

23%35%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

California HOU Sample Average Wyoming

Rural

Urban



removing each logger, field staff noted whether the logger was correctly installed and where its sensor was oriented.

Model Specification To estimate HOU, Cadmus determined the total “on” time for each individual light logger per day, using the following guidelines:

If a light logger did not record any light for an entire day, the day’s HOU was set to zero.

If a light logger registered a light turned on at 8:30 p.m. on Monday, and turned off at 1:30 a.m. on Tuesday morning, 3.5 hours were added to Monday’s HOU and 1.5 hours to Tuesday’s HOU.

Cadmus then modeled both weekday and weekend daily HOU as a function of room type, the presence of children in the home, and CFL saturations in the home. This was done using two analysis of covariance (ANCOVA) models, one for each day type.

ANCOVA models are regression models where a continuous variable is modeled as a function of single continuous explanatory variable (in this case, CFL saturation) and a set of binary variables. This way, an ANCOVA model simply serves as an analysis of variance (ANOVA) model with a continuous explanatory variable added. Cadmus chose this specification because of its simplicity, making it suitable to a wide variety of contexts. Though the model may lack the specificity of other methods, its estimates are not nearly as sensitive to small differences in explanatory variables, compared to more complex methods. Therefore, these models can produce consistent estimates of the average daily HOU for a given region using its specific distribution of bulbs by room and household type, and by the existing CFL saturation.

Cadmus specified the final models as cross-sectional, ANCOVA regressions for day-type1 (j), and bulb (i), as:

,

Where:

CFL Saturation = the proportion of bulbs in the home that are CFLS;

Kids = a dummy variable2 equal to one if the household has children under 18 living in the home and zero otherwise;

Kitchen = a dummy variable equal to one if the bulb is in the kitchen and zero otherwise;

Basement = a dummy variable equal to one if the bulb is in the basement and zero otherwise;

1 The two day-types for this analysis were weekend and weekday. We defined weekend as Saturday and Sunday, as

well as the following federal recognized holidays: Christmas, Thanksgiving, Labor Day, Memorial Day, New Year’s Day, Fourth of July, Presidents’ Day, and Veterans’ Day.

2 Dummy variables are binary which take only values of either zero or one. Coefficients for these variables can be interpreted as the difference in mean values between the two mutually exclusive groups.



Outdoor = a dummy variable equal to one if the bulb is in the outdoor and zero otherwise;

Bedroom = a dummy variable equal to one if the bulb is in the bedroom and zero otherwise;

Bathroom = a dummy variable equal to one if the bulb is in the bathroom and zero otherwise; and

Other = a dummy variable equals to one if the bulb is in a low-use room (such as a utility room, laundry room, or closet) and zero otherwise.

Cadmus tested the potential influences of other demographic and regional variables in model specification, such as latitude, income, education, home characteristics.

However, these variables were not included because their estimated coefficients did not differ significantly from zero or produced signs inconsistent with expectations.

Final Estimates and Extrapolation As shown in Table C1, not all estimated coefficients of the two models differed significantly from zero for both day types, most likely due to differences in schedules between days. Nevertheless, we included the same independent variables in each model for better cross comparability.

Table C1. HOU Model ANCOVA Estimates

Coefficient Weekday Weekend Parameter Estimate p-value3 Parameter Estimate p-value

Intercept4 2.58 <.0001 2.90 <.0001 CFL Saturation -1.05 0.0359 -0.32 0.6657

Kids 0.80 <.0001 0.51 0.1135 Kitchen 1.18 0.0001 0.35 0.4049 Basement -0.25 0.5489 -1.50 0.0134 Outdoor 2.80 <.0001 -1.46 0.1347 Bedroom -1.10 <.0001 -2.02 <.0001 Bathroom -0.98 0.0019 -1.54 0.0025 Other -1.30 0.0071 -2.16 0.0008

Cadmus used these model parameters to predict average daily use for HES by taking the sum of the product of each coefficient shown in Table C1 and its corresponding average independent variable. Table C2 shows the independent variables used for HES. Except for CFL saturation, Cadmus estimated independent variables using 2009–2010 participant survey data. Due to a lack

3 P-values indicate the degree of confidence to which analysis asserts the given coefficient equals zero. In other

words, it is the probability that the effect of a given variable on HOU is random. Therefore, a lower p-value indicates a higher degree of confidence in the estimated effect.

4 The models’ intercept can be interpreted as the average HOU in the main living space (defined as the dining room, hallways, living rooms, and office/den areas) when existing CFL saturations are zero and no children live in the home.



of detailed CFL saturation data for Rocky Mountain Power’s Wyoming service area, Cadmus used secondary data to estimate CFL saturations by room.5

Table C2. Weekday HOU Estimation Input Values

Variable Value CFL Saturation 18% Kids 33% Kitchen 13% Basement 5% Outdoor 3% Bedroom 26% Bathroom 14% Other 8%

Using these values, the following equation calculates a 2.36 average weekday HOU:

2.58 1.05 ∗ 0.18 0.8 ∗ 0.33 1.18 ∗ 0.13 0.25 ∗ 0.05 2.8

∗ 0.03 1.1 ∗ 0.26 0.98 ∗ 0.14 1.3 ∗ 0.08 2.36 Using the same method, Cadmus calculated weekend HOU using parameter estimates from the weekend model. The weighted average of these two values then provides the average annual HOU:

Table C3. HOU by Day Type

Day HOU Weight Weekday 2.36 69.3% Weekend 2.02 30.7% Overall 2.25

Precision calculations for model estimates accounted for sampling errors in model estimates and sample inputs, which largely arose from participant surveys. Precision of individual HOU estimates can be impacted by the precision of logger data model estimates and the accuracy of model inputs used for extrapolation. Cadmus estimated the final relative precision for CFL HOU in the HES program to be ±4.3 percent with 90 percent confidence.

5 Cadmus used an average CFL saturation for service areas with relatively new programs, taken from: Albee, K., et.

al. (2011). “One Analysis to Rule Them All and In the Darkness Give Them CFLs.” In proceedings of the 2011 IEPEC Conference. Boston, MA: International Energy Program Evaluation Conference.


The Cadmus Group, Inc. / Energy Services Appendix E1

Appendix E. Lighting NTG Secondary Data Review

California The 2006-2008 Upstream Lighting Program (implemented by PG&E, SCE, and SDG&E) provided manufacturer and distributor buy-downs or retailer instant discounts for eligible lighting products, which were then sold through participating retailers. This program was evaluated by Kema and Cadmus in 2010.

Cadmus derived the final recommended NTG ratio estimates from supplier interviews and revealed-preference intercept surveys. Cadmus based the supply-side, self-reported NTG method on information collected during in-depth interviews and surveys with manufacturers, retail buyers, and retail store managers. Cadmus analyzed the results to determine the NTG ratio by channel for basic CFLs, specialty CFLs, and energy-efficient fixtures. The final NTG recommendation for PG&E is 0.49, for SCE is 0.64, and for SDG&E is 0.48.

Colorado The Colorado Home Lighting Program was designed to provide energy savings for Xcel Energy’s demand-side management residential program portfolio. The program was first implemented in 2006. We based NTG ratios for this program on four different data collection sources: (1) self-report, end-use customer telephone surveys; (2) supply-side interviews; (3) a multistate regression model using on-site audit results; and (4) benchmarking of other utilities around the country.

Cadmus incorporated questions in the end-use customer telephone survey to determine levels of freeridership and spillover. Cadmus also established NTG calculations for the retail channel, based on our retail store manager interviews. Cadmus based the third method of calculating the NTG ratio—the multistate regression model—on data from 16 different geographic regions in the U.S., and incorporated data from telephone surveys of over 9,300 households and on-site saturation surveys from approximately 1,400 households. Finally, Cadmus gathered NTG values from secondary data on other lighting programs across the country. Cadmus and Nexus recommended an overall NTG ratio of 1.0 based on the range of values Cadmus established through the four calculation methods.

Illinois Through the Lighting and Appliance Program, Ameren Illinois encourages its customers to purchase high-efficiency lighting products, such as CFLs and ENERGY STAR-rated dehumidifiers, ceiling fans, and room air conditioners. For the lighting portion of the program, Ameren provides upstream buy-downs to CFL manufacturers, and markets the program through participating retail stores and an online store that sells discounted CFLs. Several types of lights are discounted through the program, with an average incentive of $1.04 for each standard CFL and $1.86 for each specialty bulb. This program was launched in August 2008, and Cadmus evaluated the Program Year 2 (2009-2010) operation in 2011.



Maine The intent of the Efficiency Maine Residential Lighting Program was to transform the lighting market towards energy efficiency, rather than to achieve any specific level of energy savings or sales volume. This program was implemented from 2003 to 2006, and was evaluated by Nexus in 2007.

Nexus determined spillover and freeridership using the results of three different telephone surveys they conducted with Maine residents, including: (1) surveys with participants who purchased a lighting product through the coupon program after November 2005; (2) surveys with participants who purchased a bulb through the coupon program prior to November 2005; and (3) surveys with the general population of customers. They determined freeridership using respondents’ awareness of efficient lighting products prior to their purchases through the program, their intent to purchase the product (either at the same time or within three months of the program purchase), and their willingness to pay the average retail price for the products they purchased. The NTG ratio that Nexus recommended in their final analysis is 0.94.

Massachusetts The Massachusetts ENERGY STAR Lighting Program incents their residential customers to use ENERGY STAR-qualified lighting. The program was implemented in 2009 and 2010, and was evaluated by Nexus in 2010.

A panel of experts determined which of the NTG ratios Nexus developed for the program was most accurate. The methodologies Nexus used to calculate the NTG ratios that the panel assessed included: (1) willingness-to-pay assessments; (2) supplier self-reports; (3) active purchaser revealed preferences; (4) a multistate regression model; and (5) a conjoint/pricing elasticity analysis (for specialty bulbs only). The NTG value Nexus recommended in the final analysis is 0.47.

Missouri The Ameren Missouri Lighting and Appliance Program is a market transformation program intended to deliver energy savings through higher sales of residential, energy-efficient ENERGY STAR products, including CFLs. ENERGY STAR CFLs and ENERGY STAR lighting fixtures are discounted through this program, with an average incentive of $1.09 per bulb and $15 per fixture. This program was launched in 2009, and Cadmus evaluated the Program Year 2 (2009-2010) operation in 2011.

Cadmus determined the NTG ratio for this program using a multistate fitted model to predict per-household CFL purchases with the program. For the model, Cadmus used actual bulb purchases per household supported by the program during January through June of 2010. In order to predict purchases made through the program, Cadmus included the assumption in the model that the program had not supported any bulbs in the first six months of 2010 (the without program scenario). The recommended NTG value we used in this analysis is 0.96.

New Mexico The Southwestern Public Service Company Home Lighting and Recycling Program provides two ways for customers to purchase energy saving CFLs: (1) through mail order, and (2) through



instant rebates at retail stores. SPS worked with retailers and manufacturers to buy down the price of the bulbs to roughly $1.00 each. This program was implemented and evaluated by Xcel Energy in 2009.

Xcel Energy used information they collected through surveys of program participants to develop estimates of freeridership. During these surveys, they questioned customers about their knowledge of energy efficiency, their reasons for participating, and the measure implementation decisions they would have made in absence of the program. The final NTG value that Xcel Energy recommended in their analysis is 0.81.

Pennsylvania The PPL Electric Compact Fluorescent Lighting Campaign provides incentives to CFL manufacturers, thereby reducing the retail price of ENERGY STAR CFL bulbs. Cadmus’ NTG analysis addressed the period of December 1, 2010 through February 28, 2011.

Cadmus based our NTG analysis on findings from participant and non-participant telephone surveys. Our analysis incorporated all respondents who purchased one or more CFLs in the three months prior to the survey, regardless of whether or not they were aware of the CFL Campaign. The freeridership estimates calculated from the customer surveys indicate that the NTG ratios range from 0.72 to 0.93. Cadmus chose a value of 0.85, estimated from the higher end of the range. Cadmus based this estimation on the assumption that it is unlikely that all recent CFL purchasers who were unaware of the CFL Campaign before they participated in the customer survey would have purchased the same quantity of CFLs without the program incentive.

Utah and Washington The Rocky Mountain Power residential lighting programs within the 2006-2008 Utah Home Energy Savings Program and the 2006-2008 Washington Home Energy Savings Program offer upstream incentives for manufacturers to reduce retail prices on CFL bulbs. Both programs were implemented from 2006 to 2008, and were evaluated by Cadmus in 2010.

Cadmus determined freeridership and spillover results through participant and non-participants phone surveys. Additionally, Cadmus used CFL retailer interviews to calculate CFL leakage, and conducted a secondary data analysis to determine per unit savings based on deemed reported savings, DEER, and RTF. Finally, Cadmus prepared a third data analysis in order to compare NTG values across programs. The final NTG values recommended for the Utah Home Energy Savings Program are 0.84 for PY2006, 0.822 for PY2007, and 0.868 for PY2008. The final NTG values recommended for the Washington Home Energy Savings Program are 0.919 for PY2006, 0.894 for PY2007, and 0.807 for PY2008.

Wisconsin The Wisconsin Focus on Energy ENERGY STAR Lighting Program is a statewide program that was launched in 2001. The program provides $2 instant and mail-in rebates for the purchase of CFLs, $15 instant and mail-in rebates for the purchase of fixtures, and $20 instant rebates for the purchase of a torchiere. In 2010, PA Consulting Group and NMR Group, Inc. established the most recent NTG ratio for the program.



’s NTG analysis included three steps: (1) analysis of the retailer-provided 2008 CFL sales data and a review of secondary research sales data and NTG values, (2) analysis of the 2008 CFL reward database, and (3) calculation of NTG estimates.

In 2010, PA Consulting/NMR Group used a multistate modeling effort to establish a program NTG value of 0.62. In a 2010 memo, these evaluators stated that the multistate modeling method is preferable, with advantages that include: the ability to isolate program effects on CFL use and purchases, the use of a large sample size of households and a diversity of states, and the inclusion of non-program factors that influence CFL use. The final NTG values they used in their analysis are 0.75 for PY2007, 0.67 for PY2008, and 0.62 for PY2010.


The Cadmus Group, Inc. / Energy Services Appendix F1

Appendix F: Lighting NTG – Customer Willingness to Pay

Cadmus conducted 254 in-territory lighting phone surveys in August 2011 through a random digit dial (RDD) process. In this survey, we asked respondents a battery of questions to determine their willingness to pay for CFLs in absence of HES program mark-downs, as well as a battery of freeridership and spillover questions. After determining how many CFLs participants purchased in 2009 and 2010, we asked participants to indicate whether they would: 1) generally purchase more CFLs, fewer CFLs, or the same number of CFLs at various un-incentivized hypothetical price levels, and 2) the quantity of CFLs they would hypothetically purchase at various un-incented prices. Specifically, Cadmus determined that the average price of an un-incented standard twister CFL is $2.20,1 and then asked participants how many lamps they would purchase at twice the average un-incented price ($4.40), five times the average un-incented price ($11), and half the average un-incented price ($1.10).

We assumed that the demand for CFLs is inversely related to price, which is true for most normal economic goods, and that therefore participants would purchase more CFLs at lower prices. In order to estimate participants’ willingness to pay for un-incentivized lamps, we constructed a standard Marshallian demand schedule2 (representing a mini-market for CFLs) which relates hypothetical prices with quantities. Figure F1 illustrates the program lamp demand function based on responses from the in-territory lighting telephone survey. Prices are shown on the Y-axis, and the X-axis represents the quantity of lamps that would be purchased at each price.

1 To estimate the average price for an un-incentivized CFL, we first reviewed CFL pricing data by participating

retailers as provided by Rocky Mountain Power. Next, we found each store’s average un-incented per-lamp price. Lastly, we calculated a weighted average of store prices and each stores’ distribution of program lamps.

2 Demand schedules are traditionally presented with price on the Y-axis and quantity on the X-axis.



Figure F1. Demand Schedule for Hypothetical Lamps

After plotting the data points shown in Figure F1, Cadmus specified an exponential function to relate these quantities with hypothetical prices; which is represented by the following equation:

Equation F1:

To estimate the number of lamps that would be purchased at the average program price per lamp (net lamps), and the number of lamps that would be purchased without the program incentive (freeridership), Cadmus solved Equation F1 for x; the quantity of lamps. We determined that 2,846 CFLs would be purchased at the average incented price of $1.19, and that 2,232 lamps would be purchased at the average un-incented price of $2.20. These modeled quantities are shown in Figure F2.

Figure F2. Modeled CFL Quantities for NTG Estimation

$4.40

$11.00

$1.10

$0.00

$2.00

$4.00

$6.00

$8.00

$10.00

$12.00

0 500 1000 1500 2000 2500

Hyp

othe

tical

Pric

e

Quantity of CFLs

$0.00

$2.00

$4.00

$6.00

$8.00

$10.00

$12.00

0 500 1,000 1,500 2,000 2,500 3,000 3,500

Pric

e

Quantity of CFLs= Modeled Q CFLs

Quantity at avg. non-incented price

Quantity at avg. incented price



To estimate NTG of HES program lamps, we used the following equation:

2: _ _

_

Where:

Qcflavg_incented = 2,846; the quantity of CFLs that would be purchased at the average price of incented lamps ($1.19).

Qcflavg_unincented = 2,232; the quantity of CFLs that would be purchased at the average price of un-incented lamps ($2.20).

The NTG estimate based on responses to the survey is 0.43.

Statistical Significance and Uncertainty A RDD phone survey is designed to avoid bias through the randomness of the selection process. Some bias may enter the estimate because certain types of people are more likely to be home or agree to participate in the survey. Such bias can be addressed through post-weighting responses to more closely reflect the known demographic characteristics of the population.

Every sample, however, is subject to a type of random error that reflects the particular group of people who participated in the study. For instance, members of our sample reported that they would purchase a combined total of 1,235 CFLs at twice the current price. If we had sampled a different group of people we would expect, by the random circumstance of who was in the sample, that the total could be somewhat larger or somewhat smaller. Using classical sampling theory, we estimated the likely boundaries within which that error lies. We constructed a 90 percent confidence interval for this random error around the sum of CFLs and LEDs that reportedly would be purchased at each hypothetical price level.

Table F3 presents the sampling error for the sum of purchased bulbs at each price for CFLs.

Table F3. 90% Confidence Interval and Summary Statistics – CFLs (n=143)

Price Sum of CFLs purchased

Standard Deviation

Standard Error

Relative Precision +/-

90% Confidence Interval

Twice the avg. price 1,235 9.0 107.8 14.6% 1,058 – 1,412 Five times the avg. price 478 5.7 67.7 23.3% 367 - 589 Half the avg. price 2,297 19.6 234.3 16.8% 1,912 – 2,682


The Cadmus Group, Inc. / Energy Services Appendix G1

Appendix G. Billing Analysis Final Models

Billing analysis assessed actual net energy savings associated with insulation measure installations.1 Cadmus determined the savings estimate from a pooled, conditional savings (CSA) regression model, which included the following groups:

Insulation (combined attic, wall, and floor insulation for 2009–2010); and

Nonparticipant homes, serving as the comparison group.

However, the small number of insulation participants yielded net savings estimates with an unacceptably high margin of error of 54 percent on average. Therefore, Cadmus relied on energy modeling and phone survey results in estimating savings for these measures.

Program Data and Billing Analysis Methodology The following sources used to create the final database for billing analysis:

Program data collected and provided by PECI (including account numbers, measure types, installation dates, square footage of insulation installed, and expected savings for the entire participant population).

Control group data. The control group sample used in the billing analysis, selected by Rocky Mountain Power, drew a random sample of 5,000 nonparticipating customers in the Wyoming territory, having complete billing data from January 2007 through June 2011. Control energy use matched to quartiles of participant pre-participation energy. To ensure adequate coverage of the nonparticipating population, Cadmus included four times more nonparticipants than participants.

Billing data, provided by Rocky Mountain Power, including all 178 Wyoming HES insulation measure customer accounts and the random sample of 5,000 Wyoming nonparticipating customers. Data included: meter-read dates, and kWh consumption from January 2007 through June 2011 for participants and nonparticipants. The final sample used in the billing analysis savings model consisted of 108 participants and 432 control customers.

Wyoming weather data, including daily average temperatures from January 2007 to July 2011 for five weather stations associated with HES participants.

Cadmus first matched program data with billing data. This separated billing data into groups of participants and nonparticipants. Daily heating and cooling degree days matched to respective monthly read date periods in the billing data for use in weather-adjusted savings modeling (using zip code to weather station mapping).

1 Billing analysis was only performed for insulation measures. Energy savings achieved through installation of other

measures were not large enough, relative to total energy consumption of households installed, to allow reliable billing analysis.



The pre-period for the billing analysis was defined as 2008, before all measure installations occurred. The post-period was defined as the period from July 2010 through June 2011.2

Data Screening Table G1 shows participant and nonparticipant screening criteria used in the billing analysis. To ensure the final model used complete pre- and post-participation billing data, Cadmus selected accounts with a minimum of 300 days in the pre- and post-periods (i.e., before the earliest 2009 installation, and after the latest available installation in 2010). Additionally, Cadmus removed outlier accounts with less than 1,200 kWh per year3 or more than 40,000 kWh per year. As approximately one in four HES participants also had electric heating as their primary heating fuel, and some expected savings proved more substantial, the analysis used a less restrictive 50 percent change screen. This screen ensures nonparticipants matched better with participants. The analysis also removed instances of more than 50 percent account changing usage from the pre- and post-period.

Table G1. Screen for Inclusion in Billing Analysis

Screen Nonparticipant

Attrition (n)

Participant Attrition

(n)

Remaining Nonparticipant

(n)

Remaining Participant

(n) Original HES insulation measure participant database and random sample of 5,000 nonparticipants

5,000 178

Nonparticipants not in participant zip codes 2,257 - 2,741 178 Less than 300 days in pre- or post-period 6 55 2,735 123 Less than 1,200 kWh in pre- or post-period 11 - 2,724 123 More than 40,000 kWh in pre- or post-period 13 2 2,711 121 Changed consumption by more than 50% from pre to post*

268 11 2,443 110

Expected savings over 70% of pre-consumption** - 2 2,443 108 Nonparticipant matched quartile sample selection 2,011 - 432 108 Final Sample 432 108 *Expected engineering savings estimates were generally fairly low, compared to pre-usage for participants with gas space heating. However, a substantial participant group had electric space heating and higher expected savings. Thus, a 50% change screen placed on the billing data allowed more substantial savings for insulation installations in homes heated with electricity. **If expected engineering estimates of savings exceeded 70% of pre-consumption, either a mismatch occurred between the participant measure installation data and the billing data account or address, or the participant had a vacancy during the pre-period. As either one of these would have led to unreasonable savings estimates for that customer, they were dropped.

Billing Analysis Results After screening and matching account attrition, the final analysis group consisted of 108 participants and 432 nonparticipants.

The following final CSA regression model specification estimated savings from insulation measures:

2 For participants installing after July 2010 the post-period included only months after the measure installation.

Some participants installing late in 2010 had less than 10 months of post-period and were removed from the analysis.

3 The minimum participant usage was 1,200 kWh. This screen removed some very low usage nonparticipants.



itittititit PARTPOSTPOSTCDDHDDADC 4321

Where for customer (i) and month (t):

ADCit = Average daily kWh consumption

HDDit = Average daily heating degree-days (base 65)

CDDit = average daily cooling degree-days (base 65)

POSTt= Indicator variable of 1 in the post-period for participants and nonparticipants, 0 otherwise

PARTPOSTit= Indicator variable of 1 in the post-period for participants, 0 otherwise

The β4 key coefficient determined average insulation savings. This value averaged daily insulation savings per program participant, after accounting for nonparticipant trends. Including individual customer intercepts (i) as part of a fixed effects model specification ensured no participants or nonparticipants had an undue influence over the final savings estimate, resulting in a more robust model.

The above-pooled model combined nonparticipants (the baseline) and participants for the wall, floor, and attic insulation component of the HES Program.

Table G2 presents billing analysis results for wall, floor, and attic insulation in Wyoming, with a billing analysis savings estimate of 606 kWh. The billing analysis had average expected savings of 705 kWh, translating to an 86 percent realization rate for insulation measures. With an average participant pre-usage of 12,327 kWh, savings represented a 5 percent reduction in energy usage from insulation measures. Table G2 also includes savings for insulation measures in smaller-usage homes and higher-usage homes.

Table G2. HES Attic, Floor, and Wall Insulation Realization Rate

Home Type kWh Savings Realization Rate (90% Confidence

bounds) Model savings (low to medium usage homes, under 10,593 kWh)

409 58% (27%–89%)

Model savings (medium to high usage homes over 10,593 kWh)

835 118% (36%–201%)

Model Savings (average quartile) 622 88% (40%–136%) Model Savings (overall) 606 86% (39%–133%)



Billing Analysis Regression Models Table G3. Regression Model for Wyoming

Source

Analysis of Variance

DF Sum of

Squares Mean

Square F Value Pr > F Model 544 18,212,614 33,479 216.11 <.0001 Error 12,400 1,920,946 154.91498 Corrected Total 12,944 20,133,560

Root MSE 12.44648 R-Square 0.9046 Dependent Mean 33.27548 Adj. R-Square 0.9004 Coefficient of Variation 37.40437

Source

Parameter Estimates

DF Parameter Estimates

Standard Error t value Prob. t

Intercept* 540 20.7182 2.5603 8.09 <0.001 Post 1 -0.1521 0.2446 -0.62 0.5341 PartPost 1 -1.6597 0.5481 -3.03 0.0025 AvgHdd 1 0.4873 0.0092 52.79 <.0001 AvgCdd 1 1.9143 0.0667 28.68 <.0001 * We ran all of the models with a fixed effects specification, which is a separate intercept for each customer. Due to the large amount of output from showing the model coefficients for each of the intercepts, we only present the average of all the separate intercepts in the output.

Table G4. Regression Model for Wyoming – Small Homes (Quartiles 1 and 2)

Source


DF Sum of

Squares Mean

Square F Value Pr > F Model 274 3,020,918 11,025 325.27 <.0001 Error 6,196 210,018 33.89578 Corrected Total 6,470 3,230,936


Source

Parameter Estimates



Intercept 270 15.1220 1.2062 12.54 <0.001 Post 1 0.3745 0.1619 2.31 0.0207 PartPost 1 -1.1205 0.3624 -3.09 0.002 AvgHdd 1 0.1885 0.0062 30.62 <.0001 AvgCdd 1 1.4312 0.0439 32.57 <.0001



Table G5. Regression Model for Wyoming – Large Homes (Quartiles 3 and 4)

Source


DF Sum of

Squares Mean

Square F Value Pr > F Model 274 15,401,221 56,209 232.11 <.0001 Error 6,200 1,501,403 242.16171 Corrected Total 6,474 16,902,624


Source

Parameter Estimates



Intercept 270 26.5818 3.2227 8.25 <0.001 Post 1 -0.6647 0.4324 -1.54 0.1243 PartPost 1 -2.2873 0.9698 -2.36 0.0184 AvgHdd 1 0.7748 0.0162 47.84 <.0001 AvgCdd 1 2.3306 0.1186 19.65 <.0001


The Cadmus Group, Inc. / Energy Services Appendix H1

Appendix H. Insulation Site Visit Verification: Home Characteristics of WY

HES Insulation Participants

General Home Characteristics

Figure H1. Distribution of Home Age

Figure H2. Distribution of Home Size (square feet)

6%5%

2% 2%

14%

17%16%

31%

3%5%

0%

5%

10%

15%

20%

25%

30%

35%

Before 1920

1920s 1930s 1940s 1950s 1960s 1970s 1980s 1990s 2000s

Percent of Sites

Decade Constructed

n = 64Mean = 1967Std Dev = 22 yearsMin = 1911Max = 2008

8%

31%

22%

17%

8% 8%6%

0%

5%

10%

15%

20%

25%

30%

35%

Less than 1,000

1,000 to 1,499

1,500 to 1,999

2,000 to 2,499

2,500 to 2,999

3,000 to 3,499

3,500 or greater

Percent of Sites

Home Size (Square Feet)

n = 65Mean = 1,935 sq ftStd Dev = 832Min = 400Max = 4,200



Figure H3. Basement Type

Figure H4. Number of Occupants

21.9%

59.4%

18.8%

Crawlspace

Full finished basement

Unifinished basement

n = 64

28.1%

34.4%

10.9%

17.2%

4.7%3.1% 1.6%

1 person

2 people

3 people

4 people

5 people

6 people

7 people



Heating and Cooling

Figure H5. Primary Heating System Types

Figure H6. Primary Cooling System Types

76.2%

3.2%

9.5%

6.3%

3.2%1.6%

Gas Furnace

Gas Stove

Electric Baseboard

Electric Furnace

Ground Source Heat Pump

Electric Other

n = 63

71.9%

18.8%

6.3%

1.6% 1.6%

Central AC

No Cooling System/Other

Room AC

Fans

Evaporative Cooler

n = 64



Figure H7. Winter Heating Temperature Set Points (with 90 percent confidence intervals)

Figure H8. Summer Cooling Temperature Set Points (with 90 percent confidence intervals)

69.8

65.5

67.4

61.0

62.0

63.0

64.0

65.0

66.0

67.0

68.0

69.0

70.0

71.0

Home (n = 62)

Away (n = 52)

Work (n = 53)

Temperature (°F)

73.2

75.4 75.2

70.0

71.0

72.0

73.0

74.0

75.0

76.0

77.0

78.0

Home (n = 49)

Away (n = 30)

Work (n = 32)

Temperature (°F)


The Cadmus Group, Inc. / Energy Services Appendix I1

Appendix I. NTG Evaluation Methodology

Net-to-gross (NTG) estimates serve as a critical part of demand-side management (DSM) program impact evaluations as they allow utilities to determine the portion of gross energy savings influenced by and attributable to their DSM programs, free from the result of other influences. Freeridership and spillover comprise NTG’s two components. Freeriders are customers who would have purchased the measure without any program influence. Spillover is the amount of additional savings obtained by customers investing in additional energy-efficient measures or activities due to their program participation. Various methods can be used to estimate program freeridership and spillover. Our baseline evaluation approach uses self-reports through participant surveys.

Program Categorization Prior to designing the NTG surveys, Cadmus worked with Rocky Mountain Power to conduct a thorough review of their DSM programs, determining the following:

Each program’s unique characteristics. Since each DSM program and measure operates differently, we had to determine a clear understanding of them. This helped inform the survey design and question wording to assure nuances were acknowledged and accounted for.

The appropriate interviewee. This step was critical as we had to be confident the survey questions reached the right decision maker. For example, a review of an Energy Star Homes program may indicate the decision maker is the home builder, not the customer purchasing the home. Thus, our survey questions would be worded to apply to home builders, not homeowners.

Resulting from the program review, Cadmus aggregated the HES program measures into three distinct categories:

Appliances (ceiling fans, clothes washers, dishwashers, electric water heaters, fixtures, refrigerators, and windows)

Insulation

CFLs

In creating the program categories, we balanced each measure’s unique characteristics that require the NTG influence to be measured differently, with retaining a sufficiently large participant population to obtain a statistically significant and reliable sample.

The methodology described in this appendix was not used to evaluate NTG for CFLs. Because the HES program incents CFLs at the retailer level, participants are often unaware that they have participated in a program or have purchased an incented CFL. Calculating freeridership and spillover by surveying participants of upstream measures is therefore not a viable option.

Survey Design Direct questions (such as, “Would you have installed measure X without the program incentive?”) tend to result in exaggerated “yes” responses. Participants surveyed likely provide



answers they believe surveyors seek; so this question becomes the equivalent of asking: “Would you have done the right thing on your own?” An effective solution to avoid such bias involves asking the question several different ways to check for consistent responses.

Cadmus designed survey questions to determine why customers installed a given measure and the program’s influence over those decisions. The survey goal was to establish what the decision maker might have done in the program’s absence. Five core freeridership questions addressed that answer:

Would the participant have installed the measure without the program?

Had the participant already ordered or installed the measure before learning about the program?

Would the participant have installed the measure to the same efficiency level without the program incentive?

Would the participant have installed the same quantity of measures without the program?

In the absence of the program, when would the respondent have installed the measures?

The spillover survey sought to answer three primary questions:

Since participating in the program being evaluated, has the participant installed additional energy-efficient equipment or services that are incented through a utility program?

How influential was the evaluated program in the participant’s decision to install additional energy-efficient equipment in their home?

Did the customer receive an incentive for the additional measure installed?

Freeridership Survey Questions Eleven questions were included in the residential survey’s freeridership portion, covering the five core freeridership questions listed above. Several skip patterns were included in the survey design, allowing the interviewers to confirm answers previously provided by respondents by asking the same question in a different format.

1. When you first heard about the incentive from Rocky Mountain Power, had you already been planning to purchase the measure?

2. Ok. Had you already purchased or installed the new measure before you learned about the Home Energy Savings program?

3. [Ask if question 2 is No or Don’t Know] Would you have installed the same measure without the incentive from the Home Energy Savings program?

4. [Ask if question 3 is No or Don’t Know] Help me understand, would you have installed something without the Home Energy Savings program incentive?

5. [Ask if question 3 or question 4 is Yes] Let me make sure I understand. When you say you would have installed the measure, would you have installed the same one that was just as energy efficient?



6. [Ask if question 3 or question 4 is Yes AND measure quantity > 1] And would you have installed the same quantity?

7. [Ask if question 3 or question 4 is Yes] And would you have installed the measure at the same time?

8. [Ask if question 3 or question 4 is No] To confirm, when you say you would not have installed the same measure, do you mean you would not have installed the measure at all?

9. [Ask if question 8 is No or Don’t Know] Again, help me understand. Would you have installed the same type of measure but it would not have been as energy-efficient?

10. [Ask if question 8 is No or Don’t Know AND measure quantity > 1] Would it have been the same measures but fewer of them?

11. [Ask if question 8 is No or Don’t Know]And, would you have installed the same measure at the same time?

Spillover Survey Questions As noted, the spillover questions sought to determine whether program participants had installed any other energy-saving measures since participating in the program. Savings participants received from additional measures would be considered spillover savings if the program significantly influenced their decisions to purchase additional measures and if they did not receive additional incentives for those measures. For residential participants, we specifically asked whether they had installed the following types of measures:

1. Clothes Washers

2. Refrigerators

3. Dishwashers

4. Windows

5. Fixtures

6. Heat Pumps

7. Ceiling Fans

8. Electric Water Heater

9. CFL’s

10. Insulation

If the participant installed one or more of these measures, they were asked additional questions about what year they purchased the measure, if they received an incentive for the measure, and how influential (on a scale of 0 to 10, with 0 being not at all influential and 10 being very influential) the HES program was on their purchasing decision.

Cadmus combined the freeridership and spillover questions in the same survey, asking them simultaneously through telephone interviews of randomly selected program participants. Prior to



beginning the live participant phone calls, Cadmus worked with the survey company to pretest the survey, ensuring all appropriate prompts and skip patterns were followed. Cadmus also monitored the initial phone calls to verify: (1) the survey respondents understood the questions; and (2) adjustments were not required.

Freeridership Methodology Cadmus developed a transparent, straightforward matrix approach to assign a score to participants, based on their objective responses to targeted survey questions. Question response patterns were assigned freeridership scores, and the confidence and precision estimates were calculated on the distribution of these scores. This specific approach is cited in the NAPEE Handbook on DSM Evaluation, 2007 edition, page 5-1.

The response patterns and scoring weights remain explicit; they can be discussed, changed and results shown in real time. Our approach provided other important features, including:

Derivation of a partial freeridership score, based on the likelihood of a respondent taking similar actions in the incentive’s absence.

Use of a rules-based approach for consistency among multiple respondents.

Use of open-ended questions to ensure quantitative scores matched respondents’ more detailed explanations regarding program attribution.

The ability to change weightings in a “what if” exercise, testing the response set’s stability.

The Cadmus method offers a key advantage by introducing the concept of partial freeridership. Experience has taught us that program participants do not fall neatly into freerider and not-freerider categories. For example, partial freeridership scores were assigned to participants with plans to install the measure; though, the program exerted some influence over their decision, other market characteristics beyond the program also proved influential. In addition, with partial freeridership, we can utilize “Don’t Know” and “Refused” responses by classifying them as partial credit, rather than removing the entire respondent from the analysis.

Freeridership was assessed at three levels. First, each participant survey response was converted into freeridership matrix terminology. Each participant’s combination of responses was then assigned a score from the matrix. Finally, all participants were aggregated into an average freeridership score for the entire program category.

Convert Responses to Matrix Terminology We independently evaluated each survey question’s response to assess participants’ freeridership level for each question. Each survey response option was converted into a value of “yes” (100 percent freerider), “no” (0 percent freerider), or “partial” (50 percent freerider).

Table I1 lists eleven survey questions, their corresponding response options, and the value which we converted them to (in parentheses). “Don’t Know” and “Refused” responses were converted to “Partial” for all but the first three questions. For those questions, we determined if a participant was unsure whether they had already purchased or were planning to purchase the measure before learning about the incentive, they were unlikely to be a freerider.



Table I1. Assignments of HES Survey Response Options into Matrix Terminology

Already planning to

purchase?


installed?

Installed same

measure w

ithout incentive?

Installed something

without incentive?

Installed same

efficiency?

Installed same

quantity?


time?

Would not have

installed measure?

Installed lower

efficiency?

Installed lower

quantity?


time?

Yes (Yes)

Yes (Yes) Yes (Yes)

Yes, I would have

installed something.

(Yes)

Yes (Yes)

Yes (Yes)

Within the

same year (Yes)

Yes (Yes)

Yes (Yes)

Yes (Yes)

Within the

same year (Yes)

No (No)

No (No) No (No)

No, I would not have installed anything.

(No)

No (No) No (No)

Within one to

two years (No)

No (No) No (No) No (No)

Within one to

two years (No)

Don't Know (No)

Don't Know (No)

Don't Know (No)

Don't Know (Partial)

Don't Know

(Partial)

Don't Know

(Partial)

Within three to

five years (No)

Don't Know

(Partial)

Don't Know

(Partial)

Don't Know

(Partial)

Within three to

five years (No)

Refused (No)

Refused (No)

Refused (No)

Refused (Partial)

Refused (Partial)

Refused (Partial)

In more than five

years (No)

Refused (Partial)

Refused (Partial)

Refused (Partial)

In more than five

years (No)

Don't Know

(Partial)

Don't Know

(Partial)

Refused (Partial)

Refused (Partial)

Participant Freeridership Scoring After converting survey responses into matrix terminology, we created a freeridership matrix, so the combination of each participant’s responses to the questions could be assigned a freeridership score. To create the matrix, we determined every combination of possible responses to the survey questions, and then assigned a freeridership score of 0 to 100 percent to each combination. Using this matrix, every participant combination of responses was assigned a score of 0 to 100 percent.



Program Category Freeridership Scoring After assigning a freeridership score to every survey respondent, Cadmus calculated a savings weighted average freerider score for the program category. Respondent freerider scores are individually weighted by the estimated savings of equipment installed. Savings-weighted freeridership and NTG scores serve a recent standard practice of the California Public Utilities Commission.

∑ ∑

The Cadmus Freeridership Scoring Model Cadmus has developed an Excel-based model to assist with freeridership calculation and improve consistency and quality of results. Our model translates raw survey responses into matrix terminology, and then assigns each participant’s response pattern a score from the matrix. Program participants in the sample can be then aggregated by program category to calculate the average freerider score.

The model incorporates the follow inputs described in this methodology:

Raw survey responses for each participant, along with the program category for their incented measure, and energy savings from that measure, if applicable.

Figures converting the raw survey responses into matrix terminology for each program category.

Custom freeridership scoring matrices for each unique survey type.

The model uses a simple interface, allowing users to quickly reproduce a scoring analysis for any program category. It displays each participant’s combination of responses and corresponding freeridership score, and then produces a summary table, providing the average score and precision estimates for the program category. The model uses the sample size and a two-tailed test target at the 90 percent confidence interval to determine the average score’s precision.

Table I2 shows a summary table example for the HES appliances program category. The figure shows the final freeridership score in the lower right corner. The example program category averaged freeridership of 46 percent, meaning that 46 percent of the energy savings were derived from freeriders and should be removed from gross program savings. Based on a 281 response sample size, the program’s absolute precision was 2.8 percentage points.

Table I2. Freerider Scoring Model Output

Population (P) 10000 SE of Mean (SEMean) 0.0171 ed Relative Precision 6.14%

Total Responses (n) 281 Relative Precision 6.14% efficient of Variation 0.6254

Responses Removed 0 Absolute Precision 0.028 Upper Bound Score 0.49

variance of mean (varMean) 0.0823 Finitie Pop. Correction (pCorr) 1 eighted Mean Score 0.46

standard deviation (sd) 0.2868 Adjusted SE (adjSE) 0.02 Lower Bound Score 0.43



Spillover Methodology Spillover refers to additional savings generated by program participants due to their program participation, but not captured by program records. Spillover occurs when participants choose to purchase energy-efficient measures or adopt energy-efficient practices because of a program, but they choose not to participate or are otherwise unable to participate in the program. As these customers are not participants, they do not typically appear in program records of the savings generated by spillover impacts.

Examples of spillover include:

Program participants adopting additional measures without an incentive.

Consumers acting on the programs’ influence resulting from changes in available energy-using equipment in the marketplace.

Changes brought about by more efficient practices employed by architects and engineers, ultimately forcing consumer behavior into desired patterns.

Changes in nonparticipants behaviors resulting from direct marketing or changes in stocking practices.

The energy-efficiency programs’ spillover effect serves as an additional impact, which can be added to the program’s valid results, in contrast to the freeriders’ impacts (which reduce net savings attributable to the program).

For the HES program, Cadmus measured spillover by asking a sample of participants purchasing and receiving an incentive for a particular measure if, due to the program, they installed another efficient measure or undertook other energy-efficiency activity. Respondents were asked to rate, on a scale of 0 through 10, the relative influence of the HES program and incentive on their decision to pursue additional savings.

Participant Spillover Analysis For calculating spillover savings, we used a top-down approach. We started the analysis with a subset only containing survey respondents who indicated they installed additional energy-savings measures after participating in the HES program. From this subset, we removed participants who indicated the program had little influence on their decision to purchase additional measures, only keeping participants rating the influence as 8, 9, or 10. We also removed participants indicating they applied for HES incentives covering additional measures they installed. Although energy savings resulted from the measures these participants installed, they could not be attributed to the original program for which the participant received an incentive.

For remaining participants with legitimate spillover savings, we estimated energy savings from additional measures installed. Savings values calculated by Cadmus engineers were matched to the additional measures installed by survey participants.

The spillover percentage per program category was calculated by dividing the sum of the additional spillover savings reported by respondents for a given program category by total incentivized gross savings achieved by all respondents in the program category:



% ∑ ∑


The Cadmus Group, Inc. / Energy Services Appendix J1

Appendix J. Marketing Materials Review

Interactive Best Practices

Leverage first impressions Include a simple, attention-grabbing, and relevant offer

Keep offer highlights above the fold

Offer clear calls to actions

Communicate value Always ask “what’s in it for my reader?”

Make offer attractive and easy to access

Target to site visitor as much as possible

Keep it simple Design clear and intuitive navigation

Don’t make your visitor hunt for the program/offer

Offer simple forms

Request the minimum contact information for lead capture

Focus on “conversion” to maximize results Make the “submit” or conversion button prominent

Offer more information and assistance in exchange for contact information

Become customer-centric; offer not only information, but also support

Build trust Communicate your privacy policy clearly

Make sure visitors know where any contact information will (and won’t) be used

Offer educational value; residents and businesses appreciate more information multiple energy efficiency programs

Test, measure, fine tune, repeat Website designers serious about leveraging their online presence constantly test multiple landing page variables in image, copy, look and feel, offer, and lead marketing. Suggested metrics are included for review at the end of this report.


The Cadmus Group, Inc. / Energy Services Appendix K1

Appendix K. Engineering Review and Whole House Modeling

Engineering Review The engineering review used data from the participant phone surveys and secondary data to evaluate gross savings for clothes washers, refrigerators, dishwashers, ceiling fans and light fixtures. As shown in Table K1, realization rates ranged between 16 percent and 187 percent.

Table K1. Engineering Review Summary Table


Gross Reported Savings

(kWh/unit)

Gross Evaluated Savings

(kWh/unit) Realization

Rate 2009-10 Clothes Washers Clothes Washer-Tier One (1.72 - 1.99

MEF) 218 (weighted

average) 303 139%

2009-10 Clothes Washers Clothes Washer-Tier Two (2.0 + MEF) 234 (weighted average)

437 187%

2009-10 Refrigerator ENERGY STAR Refrigerator 98 54 56%

2009-10 Dishwasher ENERGY STAR Dishwasher 29 29 100%

2009-10 Ceiling Fans Ceiling Fans 107 17 16%

2009-10 Fixtures Fixtures 92 64 70%

Clothes Washers Cadmus based clothes washer deemed savings values for 2009 and 2010 on the Bonneville Power Administration Planning, Tracking and Reporting System (PTR).1 .

A PTR value for one of eight different configurations of modified energy factor (MEF) level and dryer fuel was applied to each clothes washer measure. Cadmus calculated savings based on a metering study in 2009,2 which metered more than 100 clothes washers in California homes for three weeks. The largest in situ metering study on residential clothes washers and dryers conducted in the last decade, this study indicated higher consumption and savings values than those often estimated. Dryers evidence the majority of energy consumption and savings, as high-efficiency washing machines remove more moisture from clothes, allowing shorter drying times.

Cadmus determined the annual electricity savings by multiplying the metering study’s kWh/cycle values by 326 cycles/year, the average number of cycles obtained though the phone survey of clothes washer incentive participants.

1 http://www.ptr.nwcouncil.org 2 “Do the Savings Come Out in the Wash? A Large Scale Study of In-Situ Residential Laundry Systems”, The

Cadmus Group, Inc, 2010. http://eec.ucdavis.edu/ACEEE/2010/data/papers/2223.pdf



The survey indicated 29 percent of Rocky Mountain Power customers receiving clothes washers incentive used electric domestic water heaters (DWH), and the 2006 Rocky Mountain Power Energy Decisions Survey indicated 95 percent of Wyoming single-family homes used electric dryers. Cadmus developed savings values for each tier by taking weighted averages based on these parameters and an estimate from PECI analysis for distribution of sales based on MEF.

Table K2 compares the assumptions used for deemed and adjusted savings values. As shown, Cadmus measured lower energy savings for machines with modified energy factor (MEF) ratings of at least 2.2 than for those with ratings between 2.0 and 2.19. Factors contributing to this difference may include machine sizes and user settings.

Table K2. Clothes Washer Calculations, 2009–2010

Input Reported Value Evaluated Value Cycles per year Unknown 326

Water Heater Fuel Electric N/A 29% Gas N/A 71%

Dryer Fuel Electric N/A 95% Gas N/A 5%

MEF 2.0+ Distribution of Sales MEF 2.0–2.19

N/A 19%

MEF 2.2 + 81%

Gross Unit Savings (kWh/year)

Electric DHW & Electric Dryer

MEF 1.72–1.99 295 309 MEF 2.0–2.19

320 586

MEF 2.2 + 508

Gas DHW & Electric Dryer

MEF 1.72–1.99 188 326 MEF 2.0–2.19

211 498

MEF 2.2 + 420

Electric DHW & Gas Dryer

MEF 1.72–1.99 128 -26 MEF 2.0–2.19

170 101

MEF 2.2 + 91

Gas DHW & Gas Dryer

MEF 1.72–1.99 37.0 -7 MEF 2.0–2.19

44.0 16

MEF 2.2 + 3 The following determined adjusted unit savings:

kWh savings/cycle x cycles/year = annual kWh savings

As shown in Table K3, both tiers showed higher adjusted savings values than deemed savings.



Table K3. Clothes Washer Savings, 2009-10

Modified Energy Factor

Configuration Gross Unit Savings (kWh/year) Reported Evaluated Difference

1.72-1.99

Electric DHW & Electric Dryer 295 - -

Electric DHW & Gas Dryer 128

Gas DHW & Electric Dryer 188

Gas DHW & Gas Dryer 37

Weighted average 218 303 +85

2+ Electric DHW & Electric Dryer 320 - -

Electric DHW & Gas Dryer 170

Gas DHW & Electric Dryer 211

Gas DHW & Gas Dryer 44

Weighted average 234 437 +203

Refrigerators PTR provided deemed savings value for refrigerators, a value since updated in the PTR, based on a database of available models in August 2008. The updated value was used in the Northwest Power and Conservation Council’s Sixth Power Plan (6PP) and includes a weighted average based on the estimated market share of each model type.

For 2009 and 2010, Cadmus used the weighted average site savings value from 6PP to estimate gross per unit energy savings, as shown in Table K4.

Table K4. Refrigerator Savings, 2009-10

Gross Unit Savings (kWh/year) Reported Evaluated Difference

2009–2010 97.5 54.5 -43.0

Dishwashers Cadmus based dishwashers’ deemed savings value on data from the Northwest Power and Conservation Council’s Regional Technical Forum (RTF). An RTF value for one of eight different configurations of energy factor (EF) level and water heater fuel was applied to each dishwasher measure. In review of the RTF analysis, Cadmus found that it is based on an assumption of 215 cycles/year, the same value reported in the participant survey. Cadmus does not suggest any difference in gross savings, as shown in Table K5.



Table K5. Dishwasher Savings

Energy Factor Water Heater Fuel Gross Unit Savings (kWh/year)

Reported Evaluated Difference 0.65-0.67 Electric 40.9 40.9 0 0.68-0.71 Electric 55.7 55.7 0 0.72-0.8 Electric 73.4 73.4 0 0.81-1.11 Electric 104.0 104.0 0 0.65-0.67 Gas 9.3 9.3 0 0.68-0.71 Gas 13.0 13.0 0 0.72-0.8 Gas 17.6 17.6 0 0.81-1.11 Gas 24.2 24.2 0

Ceiling Fans The HES Program offered ENERGY STAR ceiling fans in both 2009 and 2010 program years. Ceiling fan 2009–2010 reported saving values derived from the sum of motor savings from the ENERGY STAR savings calculator and lighting savings based on the PTR. The PTR based CFL savings on the average room type and, multiplied by three, the assumed number of bulbs per ceiling fan.

Cadmus used the same motor savings value as the ENERGY STAR calculator, and calculated lighting savings based a methodology similar to CFL lamps. The following calculation provides the ceiling fan savings methodology, with Table K6 detailing the input assumptions:

ΔkWh = (MotorkWh) + (((ΔWatts) /1000) * ISR * (HOU * 365) * WHF * Number of Bulbs)

ΔWatts = Wbase - Weff

Where:

MotorkWh = Motor savings per ceiling fixture (kWh)

Weff = Wattage of efficient ENERGY STAR CFL

Wbase = Wattage of baseline fixture

HOU = Hours of use per day

ISR = In Service Rate or percentage of incented units installed

WHF = Waste Heat Factor for energy to account for HVAC interaction affects (heating and cooling)

365 = Constant (days per year)

1000 = Constant (conversion watts to kilowatts)

Number of Bulbs = Number of bulbs per fixture



Table K6. Ceiling Fan Input Assumptions

Ceiling Fan Input Variable Input Source MotorkWh 6 ENERGY STAR Calculator* Weff 20.28 Median ceiling fan lamp wattage based on ENERGY STAR Qualified

Product List** Wbase 75 Comparable incandescent wattage based on Cadmus’ CFL lamp analysis HOU 2.42 Cadmus’ hours of use model and PacifiCorp’s HES Residential Survey ISR 1 Assume all fixtures were installed WHF 0.91 Based on Cadmus’ CFL lamp analysis Number of Bulbs 0.26 Model data; average number of bulbs based on 2009–2010 participant

product data * ENERGY STAR Ceiling Fan Calculator http://www.energystar.gov/index.cfm?fuseaction=find_a_product.showProductGroup&pgw_code=CF ** ENERGY STAR Qualified Product List (ENERGY STAR Ceiling Fans with Light Kits Product List, August 15, 2011) Ceiling fan HOUs derived from room location assumptions, with common room locations identified where ceiling fans would be installed. This differed from the CFL HOU analysis, which included all room locations in determining overall daily hours of use. Ceiling fan rooms included main living spaces, kitchens, and bedrooms.

Rocky Mountain Power’s savings analysis documentation assumed all ceiling fans included a three-bulb lighting fixture. Rocky Mountain Power’s 2009–2010 participant data did not include the number of bulbs: only model and brand. The two program years paid out only 31 ceiling fan incentives. Cadmus verified the number of bulbs per fixture by reported model numbers, using through Web-searches and ENERGY STAR product lists. As shown in Table K7, of the 31 ceiling fans sold, only five products had lighting fixtures attached.

Table K7. Ceiling Fan Lighting Kits

Ceiling Fan Number of Unique Products Total Number of Products Sold

Total Number of Lamps

No Light Kit 11 24 0 Light Kit 3 5 8 Model Not Found 1 2 0 Total 15 31 8

* Based on overwhelming results that only 20 percent had light kits; zero was assumed for the model not known.

Research determined fixtures averaged 0.26 bulbs. Table K8 shows reported and evaluated ceiling fan per unit savings.

Table K8. Ceiling Fan Per Unit Savings

Ceiling Fan Measure Unit Reported Evaluated Motor per Unit Savings (kWh) 6 6 CFL per Bulb Savings (kWh) 33.8 43.8 CFL per Fan Savings (kWh) 101.4 11.3 Total Ceiling Fan Savings (kWh) 107.4 17.3

The largest per unit savings variance resulted from the assumed number of bulbs per fixture. Rocky Mountain Power’s HES Program allowed ENERGY STAR ceiling fans with and without



light fixtures, but the savings analysis assumed installation only of ceiling fans with light fixtures.

Table K9 shows HES Program evaluated savings of 537 kWh for the 31 ENERGY STAR ceiling fans receiving incentives.

Table K9. Evaluated and Reported Ceiling Fan Savings for 2009–2010

Measure Unit

HES Program Year Participants

Number of Units

Evaluated Gross Savings (kWh)

Reported Gross Savings (kWh)

Ceiling Fan HES 2009 8 12 208 1,289 HES 2010 12 19 329 2,041

Total 20 31 537 3,329

ENERGY STAR Fixtures In both 2009 and 2010 program years, HES offered ENERGY STAR fixtures, with 2009–2010 reported saving values for ENERGY STAR fixtures based on the PTR. Using the PTR, Regional Technical Forum, and 6th Power Planning assumptions, the evaluation assumed total fixture savings of two bulbs per fixture.

Cadmus calculated lighting savings based on a methodology similar to CFL lamp analysis. Using the ENERGY STAR fixtures calculation and input assumptions, shown in Table K10, the following equation provided savings:

ΔkWh = (ΔWatts) /1000) * ISR * (HOU * 365) * WHF * Number of Bulbs

ΔWatts = Wbase - Weff

Where:

Weff = Wattage of efficient ENERGY STAR CFL

Wbase = Wattage of baseline fixture

HOU = Hours of use per day

ISR = In Service Rate or percentage of incented units installed

WHF = Waste Heat Factor for energy to account for HVAC interaction affects (heating and cooling)

365 = Constant (days per year)

1000 = Constant (conversion watts to kilowatts)

Number of Bulbs = Number of bulbs per fixture



Table K10. ENERGY STAR Fixture Input Assumptions

Ceiling Fan Input Variable Input Source

Weff 18 Median fixture lamp wattage based on ENERGY STAR Qualified Product List* Wbase 75 Comparable incandescent wattage based on Cadmus’ CFL lamp analysis HOU 2.25 Cadmus’ hours of use model and PacifiCorp’s HES Residential Survey ISR 1 Assume all fixtures were installed WHF 0.91 Based on Cadmus’ CFL lamp analysis Number of Bulbs 1.61 Model data; average number of bulbs based on 2009–2010 participant product data

*ENERGY STAR Qualified Product List (ENERGY STAR Residential Light Fixtures Product List; August 15, 2011). Cadmus based ENERGY STAR fixture HOUs on CFL HOU analysis, with all room locations included in determining overall daily HOUs. As described in the CFL analysis, Cadmus used an HOUs model and data collected from the HES residential survey, detailing lighting information by room type. The HES participant data did not include wattages; therefore the efficient CFL wattage (Weff) is based on the ENERGY STAR fixture product list where the medium wattage is 18 watts per lamp for each fixture.

Rocky Mountain Power’s 2009–2010 participant data did not include the number of bulbs, only model and brand. The HES Program incented 76 ENERGY STAR fixtures over two years. Cadmus verified the number of bulbs per fixture by reported model numbers using Web-searches and ENERGY STAR product lists, for an average 1.61 number of bulbs per fixture.

Table K11 shows reported and evaluated ENERGY STAR fixture per unit savings

Table K11. ENERGY STAR Fixture Per Unit Savings

ENERGY STAR Fixture Measure Unit Reported Evaluated

Number of Bulbs per Fixtures 2 1.61 Per Bulb Savings (kWh) 46.0 42.5 Total Fixture Savings (kWh) 92.0 68.1

The large variance in the number of bulbs per fixture direct impacted total fixture savings. The HES Program reported evaluated savings for ENERGY STAR fixtures of 5,179 kWh for 76 products incented, as shown in Table K12.

Table K12. Evaluated and Reported ENERGY STAR Fixture Savings for 2009–2010

Measure Unit

HES Program Year Participants

Number of Units

Evaluated Gross Savings (kWh)

Reported Gross Savings (kWh)

ENERGY STAR Fixtures

HES 2009 18 43 2,930 3,956 HES 2010 21 33 2,249 3,036

Total 39 76 5,179 6,992



Whole House Energy Modeling

Overall Methodology For insulation, heating and cooling measure developed by PECI, Cadmus first examined the following savings values:

PECI Gross Savings Calculated. These PECI-developed per measure kWh savings were based upon either an energy simulation model, Energy Gauge V. 2.8, RTF values, or a 2006 Evaporative and Air Conditioners evaluation and were found in Exhibit A5_Wyoming Savings Summary_2010_122109.xlsx and Exhibit A.5_Wyoming Savings Summary_2008 & 2009_PECI_12312008 NS.xlsx. These calculated savings were usually the basis of the reported savings described below.

PECI Gross Savings Reported. These reported PECI-developed per measure kWh savings were used as the deemed savings for each measure and were found in PC WY HES Extract.xlsx. These reported values did not always match the PECI Gross Savings Calculated values described above. In these instances, Cadmus was not sure how the reported values were derived. Sometimes these values were not reported which meant that no one participated in that program that year.

In order to verify these savings, Cadmus developed their own independent estimates of realized savings:

Cadmus Gross Savings Realized. For most of the measures, Cadmus used a residential energy simulation model -- Architectural Energy Corporation’s Rem-Rate V12.95 – to estimate realized savings. (RESNET® accredits Rem-Rate for modeling residential homes.) Cadmus first determined the characteristics of a typical home based upon surveys, site visits, and PECI’s measure tracking database. Cadmus then used these characteristics as inputs into Rem-Rate. (See Table K13 and Table K14). The difference in modeled overall energy use between a baseline home (example. one with the standard efficiency unit such as a 13 SEER AC unit) and the efficient home (example. one with a 15 SEER AC unit) were the measure’s realized gross savings. For a few other measures that could not be modeled through Rem-Rate, Cadmus used other secondary sources to estimate realized savings.

Cadmus Gross Savings Using Energy Gauge. Using the same characteristics of a typical home and baseline and efficient home scenarios, Cadmus checked each Rem-Rate based realized savings value against one calculated using Energy Gauge. If the savings values were substantially different between the two models, algorithm differences were researched to determine why the values differed.



Table K13. Characteristics of Typical Modeled Homes

Parameter Value Weather location Casper, WY Home Type 2 story, single family house Home Conditioned Floor Area 2137sqft Foundation Type Vented Crawlspace Walls 2X4 Exterior walls with R-11 Cavity Insulation Attic Flat trussed attic with R-19 insulation between cord rafters Framed Floor Framed Floor with R-19 insulation between floor joists Windows Double Pane Vinyl windows U-Value 0.48, SHGC 0.58 Infiltration 9.84 ACH @ 50Pa Heating System/Cooling system Varies by model to capture several types of equipment. All types air distribution systems. Duct Leakage 499.4 CFM @25Pa Thermostat Non-Programmable 64.7F Heating 73F Cooling

Table K14. Typical HVAC Systems

Electric Furnace wo/CAC Electric Furnace w/CAC Gas Furnace w/CAC 150 kBTU(44kW) Forced Air Electric Furnace

150 kBTU(44kW) Forced Air Electric Furnace with 13SEER 4.5 ton Central AC

150kBTU Forced Air Gas Furnace with a 13SEER 4.5 ton Central AC

Summary of Results Cadmus examined the differences between the reported savings, labeled PECI Gross Savings Reported in Table K15, and realized savings, labeled Cadmus Gross Savings Realized. Realization rates between reported gross savings and realized gross savings ranged from 98 percent to 510 percent.



Table K15. Summary Table


PECI Gross Savings Reported (kWh/unit)

Cadmus Gross

Savings Using

Realized (kWh/unit)

Realization Ratio

2009 Electric Water Heater 40+ Gallons (EF 0.93 or higher) 91 110 121% 2010 Electric Water Heater 40+ Gallons (EF 0.93 or higher) 91 110 121% 2009 Windows Windows 1 2 165% 2010 Windows Windows 1 2 200% 2010 Heat Pumps Heat Pump Conversion (8.2+ HSPF) 3,147 11,457 364% 2010 Heat Pumps Heat Pump Upgrade (8.2+ HSPF) 811 1,155 142% 2009 Central Air Conditioner CAC (15 SEER) 86.1 251 292% 2009 Central Air Conditioner CAC TXV and Install 20.3 20 98% 2009 Central Air Conditioner CAC TXV and Sizing 60.3 60 99% 2009 Central Air Conditioner CAC Tune up 13 66 510% 2010 Central Air Conditioner CAC (15 SEER) 86.1 251 296% 2010 Central Air Conditioner CAC TXV and Install 20.3 20 99% 2010 Central Air Conditioner CAC TXV and Sizing 60.3 60 100% 2010 Central Air Conditioner CAC Tune up 13 66 510% 2010 Evaporative Cooler Evaporative Coolers—Whole House 292 1,372 470% 2009 Insulation Insulation: Attic (R-19 +) 0.6 0.6 113% 2009 Insulation Insulation: Floor (R-19 +) 1.0 3.1 312% 2009 Insulation Insulation: Wall (R-11+ or fill cavity) 1.5 3.6 248% 2010 Insulation Insulation: Attic (R-19 +) 0.5 0.5 111% 2010 Insulation Insulation: Floor (R-19 +) 0.4 1.3 301% 2010 Insulation Insulation: Wall (R-11+ or fill cavity) 1.1 2.8 248%

Detailed Results by Measure Type - Central Air Conditioning Table K16 below shows the different savings values both observed and calculated across all the central air condition measures. In addition, each value in the table has a reference that indicates the source of the savings value.

The following was observed when examining the results for the 15+SEER/ 12.5+EER and TXV measure:

A reported savings value of 86.1 kWh savings, based upon a Quantec (now Cadmus)3 report, was used in both 2009 and 2010. The same reported value was used in 2010 even though PECI calculated a new Energy Gauge-based value of 317 kWh savings in 2010. The Energy Gauge model calculated savings going from a baseline 12 SEER, 4.5-ton central air conditioner to a 15 SEER, 4.5-ton central air conditioner.

The realized savings for 2009 and 2010 using Rem-Rate, 251 kWh, were verified by the result of using the Energy Gauge model, 267 kWh. Both models assumed a 4.5-ton central air conditioner as a 13 SEER baseline unit upgraded to a 15 SEER central

3 Quantec LLC. Evaporative Cooling and Central Air Conditioning Incentive Program. 2006.



air conditioner. Cadmus choose a 13 SEER baseline unit because the HES Program sought to get customers to upgrade to high-efficiency air conditioners rated higher than the federal standard of 13 SEER. Consequently, only incremental savings from 13 SEER to the installed unit could be reported as savings.

The following was observed when examining the results for the TXV & Proper Installation and TXV & Proper Sizing measures:

Both of these measures relied upon the same Quantec study for the reported savings.

Cadmus verified air conditioner proper sizing and quality installation savings as reported, determining that the Quantec study provided the best data available for those measures.

The following was observed when examining the results for the Tune-Up measure, which involved checking and adjusting refrigerant charges and assuring proper evaporator air flow:

A Cadmus study4 researching seven air conditioning studies concluded savings for refrigerant charges and evaporator airflow tune-ups at 4.5 and 3.0 percent, respectively. These percentages were applied to the energy consumption estimated via the Rem-rate model of a 13 SEER air conditioner to yield the realized estimate of 66.33 kWh.

For both program years, PECI reported savings of 13 kWh per air conditioner tune-up yet calculated savings were observed to be closer to 30 kWh. Variances between calculated and reported PECI savings for both years are unclear since there was no information available on the source of the 13 kWh.

4 The Cadmus Group. Energy Savings Impact of Improving the Installation of Residential Air Conditioners. 2005.



Table K16. Central Air Conditioning Annual Savings Summary

Year Measure Original

Name

PECI Gross Savings

Calculated (kWh/ unit)

PECI Gross Savings

Reported (kWh/ unit)

Cadmus Gross

Savings Using

Energy Gauge

(kWh/ unit)

Cadmus Gross

Savings Realized

(kWh/ unit)

Realization PECI

Reported and Cadmus

Realized Per Unit

2009 CAC: 15+SEER/ 12.5+EER and TXV

86.1A 86.1 267 251C 292% AC

2010 CAC: 15+SEER/ 12.5+EER and TXV

317B 86.1 267 251C 292% AC

20092010

CAC: TXV & Proper Installation 20.3A 20.3 – 20.3A 100% AC

20092010

CAC: TXV & Proper Sizing 60.3A 60.3 – 60.3A 100% AC

2009 CAC Tune-up 29.7A 13 – 66.33E 510% AC2010 CAC Tune-up 30A 13 – 66.33E 510% ACA Quantec LLC. Evaporative Cooling and Central Air Conditioning Incentive Program. 2006 B Energy Gauge Model (12SEER Baseline CAC) C Rem-Rate Model (13SEER Baseline CAC) D Rem-Rate Model E The Cadmus Group. Energy Savings Impact of Improving the Installation of Residential Air Conditioners. 2005

Detailed Results by Measure Type – Heat Pumps Table K17 below shows the different savings values both observed and calculated across all the heat pump measures. In addition, each value in the table has a reference that indicates the source of the savings value. The following was observed when examining the results for the heat pump measures:

Reported system conversion savings of 3,147 kWh were based upon data gathered from the Energy Trust of Oregon. Other data showing savings of 9,394 kWh, calculated in 2010, were developed using Energy Gauge but never adapted as reported savings.

Reported upgrade savings of 811 kWh were based upon data gathered from the Energy Trust of Oregon. Other data showing savings of 1,870 kWh, calculated in 2010, were developed using Energy Gauge but never adapted as reported savings.

Realized system conversion savings assumed a baseline heating and cooling system of a 150-kBTU (44kW) electric furnace and a 4.5-ton 13 SEER Central Air Conditioner, converted to a 57.5-kBTU, 15 SEER, 8.5-HSPF heat pump. Energy savings resulting from this conversion, when modeled in Energy Gauge, provided unusual negative energy savings of -6,134 kWh. Given an electric furnace has a heating Coefficient Of Performance (COP) of 1.0, and a heat pump has a COP of approximately 2.5, heat pumps should use significantly less energy. Research into Energy Gauge’s calculations indicated the heat pump ran 2.3 times longer throughout



the heating season than the electric furnace. According to Energy Gauge’s developers, this caused the duct loss load to increase to the point that it eliminated energy savings from the heat pump’s increased COP. Increasing the modeled heat pump’s size would have allowed for reduced operating hours, but would not provide an accurate model as it would incorrectly estimate the supplemental heat provided by the model. An increased heat pump size to meet the load of the home was what was modeled by PECI for 2010 calculated savings. In that model a 108kBTU heat pump was used; residential heat pump are typically no larger than 60kBTU’s and any additional heat required to meet the load of the home is added by a electric resistance coil. By modeling an oversized heat pump Energy Gauge neglects to account for any electric resistance heat. In this case, Energy Gauge results were discarded as inaccurate and not applicable to the HES Program’s context. This was done because the amount of duct leakage would be proportional to how much air flows through the duct system, not how long the duct system operated; so Energy Gauge’s assumption would not be valid. As a further check, the Rem-Rate based estimate of savings, 11, 457 kWh, represented a reduction in consumption of 34 percent. The Department of Energy5 claims 30 to 40 percent savings with heat pump systems over electric resistance heat. For the 2009 program year no participants were tracked for this measure so no savings were reported for that year.

Realized upgrade savings assumed a homeowner purchased a high-efficiency, 15 SEER, 8.5-HSPF air-source heat pumps instead of a 13 SEER, 7.7-HSPF heat pumps. As discussed, high duct losses modeled by Energy Gauge resulted in inflated savings, 1,662 kWh, over the Rem-Rate model estimates, 1,155 kWh. For the 2009 program year no participants were tracked for this measure so no savings were reported for that year

5 Department of Energy. Heat Pumps. 2011.

http://www.energysavers.gov/your_home/space_heating_cooling/index.cfm/mytopic=12610



Table K17. Heat Pump Annual Savings Summary


Name

PECI Gross Savings


PECI Gross Savings

Reported (kWh/ unit)

Cadmus Gross

Savings Using

Energy Gauge

(kWh/ unit)

Cadmus Gross

Savings Realized

(kWh/ unit)

Realization PECI

Reported and

Cadmus Realized Per Unit

2009 Heat Pump System Conversion

3,147A – -6,134 11,457C – ASHP

2010 Heat Pump System Conversion

9,394B 3147 -6,134 11,457C 364% ASHP

2009 Heat Pump Upgrade 811A – 1,662 1,155C – ASHP2010 Heat Pump Upgrade 1,870B 811 1,662 1,155C 142% ASHPA Energy Trust of Oregon B Energy Gauge Model (108 kBTU Efficient Heat Pump) C Rem-Rate Model (57.5 kBTU Efficient Heat Pump)

Detailed Results by Measure Type – Evaporative Coolers Table K18 below shows the different savings values both observed and calculated across all the evaporative cooler measures. In addition, each value in the table has a reference that indicates the source of the savings value. The following was observed when examining the results for the evaporative cooler measures:

Reported whole house savings, 291.9 kWh, were sourced from a 2006 Program Evaluation of Cool Cash for Nexant in Utah by Quantec. The Utah savings were adjusted to Wyoming savings by multiplying by the ratio of cooling degree days. Other 2010 PECI calculated savings data for portable and whole house evaporative coolers were developed using Energy Gauge but never adapted as reported savings.

No one participated in this program in 2009. In 2010, 4 homeowners participated in the whole house program; no one participated in the portable program.

Realized savings for evaporative coolers were estimated based on 2010 metering study, performed by Cadmus for Xcel Energy on evaporative coolers in Denver, Colorado. Cadmus normalized the results of this metering study from Denver to Casper using TMY3 weather data. Denver and Casper have very similar climates: both cities share similar elevations, cooling degree days, and wet bulb temperatures—important factors for evaporative cooler operation. The metering study was used because, given their operation methods, evaporative coolers can be difficult to model using traditional approaches. They require occupants to open windows in their homes, move very large volumes of air through the structure, and have a cooling capacity based on wet bulb ambient temperatures.



Realized whole house savings estimate of 1,372 kWh is very close to the Energy Gauge-based 2010 PECI estimate of 1,302 kWh that was calculated but never adapted as the reported savings. This observation supports our use of the 2010 Denver metering study to estimate savings.

Table K18. Evaporative Cooler Annual Savings Summary


Name

PECI Gross Savings


PECI Gross

Savings Reported

(kWh/ unit)

Cadmus Gross

Savings Using

Energy Gauge

(kWh/ unit)

Cadmus Gross

Savings Realized

(kWh/ unit)

Realization PECI

Reported and


2009 Evaporative coolers 291.9A – – 1,205C – Evap

Cooler

2010 Evaporative Coolers, Portable

203B – – 642C – Evap

Cooler

2010 Evaporative Coolers, Whole House 1,302B 291.9 – 1,372C 470%

Evap Cooler

A Quantec LLC. Evaporative Cooling and Central Air Conditioning Incentive Program. 2006 B Energy Gauge Model C The Cadmus Group. Evaporative Cooling Rebate Evaluation. 2010

Detailed Results by Measure Type – Insulation Table K20 below shows the different savings values both observed and calculated across all the insulation measures. In addition, each value in the table has a reference that indicates the source of the savings value. Savings for each of attic, wall and floor insulation measures are broken out by heating and cooling system type. Details of each of those systems are documented in Table 43. The following was observed when examining the results for the insulation measures:

All realized insulation savings were modeled in Both Rem-Rate and Energy Gauge. Baseline and efficient insulation levels were taken as weighted averages from the PECI tracking database, then verified for accuracy against site visits and survey data. For insulation upgrade programs, baseline insulation levels prove very important in calculating program energy savings. Rem-Rate and Energy Gauge treat uninsulated building constructions differently. For example, using the ASHARE method of parallel path heat transfer to construct an uninsulated 2 x 4 wall yields a total, uninsulated R-Value of 4.37. Energy Gauge assumes an uninsulated construction R-Value of 3.77, while Rem-Rate assumed an uninsulated wall construction of R-3.70. During modeling, these low R-Value uninsulated wall constructions could lead to inflated savings. All baseline and efficient measure insulation levels were accurately calculated using parallel path heat transfer and applied to the models. Table K19 details the overall construction U-Values and R-Values used in modeling.



Table K19. Insulation Baselines and Efficient Values

Modeled Inputs

Floor Measure Baseline

Floor Measure Efficient

Wall Measure Baseline

Wall Measure Efficient

Attic Measure Baseline

Attic Measure Efficient

U-Value 0.376 0.042 0.226 0.069 0.118 0.022 R-Value 2.7 24.0 4.4 14.5 8.5 46.0

PECI used two separate sources to determine calculated insulation savings for the two program years. In 2009, calculated insulation savings were taken as deemed values from the RTF. In 2010, calculated savings were sourced from a set of Energy Gauge models. However, savings reported for each measure did not match savings calculations reviewed. Reported savings for each measure ranged from 67 to 24 percent of calculated savings. PECI did not document differences between reported and calculated savings. This review concentrates on differences between verified savings and reported savings.

The following was observed when examining the results for attic insulation measure:

Calculated savings for attic insulation measures were sourced from Energy Gauge models. The models were run at three baseline insulation levels R-3, R-11, R-18 and at three heating and cooling system configurations. All post insulation levels were R-49. Reported savings for attic insulation were much lower than savings calculated with the Energy Gauge Model. The source of these savings is not documented however they do follow a similar trend to the modeled saving with respect to heat and cooling equipment.

Attic insulation realized savings when modeled in Rem-Rate yield similar savings to reported saving, with the exception of gas furnace systems with CAC. Casper has a relatively cool, dry climate, with a high daily temperature range according to the ACCA6. These factors generally create issues when modeling cooling savings, as home occupants generally open windows in the evening to cool their homes (when outdoor temperatures allow). Cadmus determines saving for attic insulation with gas furnaces and CAC equipment cannot be claimed without further monitoring and verification.

Differences between savings calculated by Cadmus operating Energy Gauge and Rem-Rate are significant. These differences were explored questioning the developers of both Rem-Rate and Energy Gauge. The exact difference why the savings vary by such a large margin could not be simply explained however the calculation method each program uses to generate savings can be. Energy Gauge

6 ACCA. Manual J. 2006



uses as hourly simulation modeling technique based on the DOE 2 simulation engine. Rem-Rate is an annual simulation based on proprietary calculations. These differences play a significant role in how each software determines savings.

The following was observed when examining the results for floor insulation measures:

Calculated savings for floor insulation measures were sourced from Energy Gauge models. Baseline models document no existing floor insulation and an efficient measure condition of R-30 modeled at three heating and cooling equipment combinations. Reported saving were again much lower than calculated saving. The source of these savings was again not documented however they do follow a similar trend to modeled savings with respect to heating and cooling equipment.

Floor insulation savings were also modeled using Rem-Rate and Energy Gauge, and savings for gas furnaces with CAC equipment again appeared negative—not an unexpected result, given cool ground temperatures behind homes benefitted the cooling load. Insulating from such constant cool sources requires greater mechanical cooling. This measure showed a net positive saving for homes with gas heat, as furnace fan savings outweighed cooling load increases. Verified savings for this measure were significantly higher than PECI-reported savings, but slightly lower than PECI-calculated savings.

Differences between savings calculated by Cadmus operating Energy Gauge and Rem-Rate are not as significant as attic insulation. Since the same inputs were used for both models the difference is likely due to the calculation approach used by the differing software’s.

The following was observed when examining the results for wall insulation measures:

Calculated savings for wall insulation measures were sourced from Energy Gauge models. Baseline models document no existing wall insulation and an efficient measure condition of R-13 modeled at three heating and cooling equipment combinations. Reported saving were again much lower than calculated saving. The source of these savings was again not documented however they do follow a similar trend to modeled savings with respect to heating and cooling equipment.

Realized wall insulation savings were also modeled in both Rem-Rate and Energy Gauge. The two models agree very closely for this measure. Realized saving are however significantly lower than PECI calculated savings. This difference can be attributed to the adjusted un-insulated wall R-value done by Cadmus. Each program underestimates the R-value of a wall construction with no insulation unless this is corrected when modeling. This explains why calculated savings are higher than realized saving; a low uninsulated baseline R-value will inflate savings.



Table K20. Insulation Annual Savings Summary


Name

PECI Gross Savings


PECI Gross

Savings Reported

(kWh/ unit)

Cadmus Gross

Savings Using

Energy Gauge

(kWh/ unit)

Cadmus Gross

Savings Realized

(kWh/ unit)

Realization PECI

Reported and


20092010

Insul Attic: Electric Heat wo/CAC 2.9A 1.95 5.3 2.37B 122% Attic Area sq ft

20092010

Insul Attic: Electric Heat w/CAC 3.0A 1.97 5.6 2.33B 118% Attic Area sq ft

20092010

Insul Attic: Gas Heat w/CAC 0.15A 0.02 0.39 (0.00)B 0% Attic Area sq ft

20092010

Insul Floors: Electric Heat wo/CAC 7.6A 1.87 7.9 5.89B 315% Floor Area sq ft

20092010

Insul Floors: Electric Heat w/CAC 7.6A 1.89 7.6 5.76B 305% Floor Area sq ft

20092010

Insul Floors: Gas Heat w/CAC 0.0A 0.02 -0.09 0.01B 50% Floor Area sq ft

20092010

Insul Walls: Electric Heat wo/CAC 12.0A 2.97 7.5 7.25B 244% Wall Area sq ft

20092010

Insul Walls: Electric Heat w/CAC 12.3A 3.00 7.8 7.28B 243% Wall Area sq ft

20092010

Insul Walls: Gas Heat w/CAC 0.31A 0.02 0.41 0.18B 900% Wall Area sq ft

A Energy Gauge Model B Rem-Rate Model

Detailed Results by Measure Type – Windows Table K21 below shows the different savings values both observed and calculated across window measures. In addition, each value in the table has a reference that indicates the source of the savings value. The following was observed when examining the results for window measures:

Calculated window saving on a per square foot basis were sourced from an Energy Gauge model. Modeled savings achieved by upgrading from a 0.48 U-Value and 0.58 SHGC to a 0.30 U-Value and 0.45 SHGC windows. The HES Program required new participating windows to meet a NFRC rating of 0.35 U-value and 0.33 SHGC. Reported savings for this measure differ from calculated savings. These differences were not documented.

The HES program requirement was modeled in Rem-Rate in comparison to a baseline window of 0.48 U-value and 0.58 SHGC. When modeled, the lower efficient



window U-Value and higher SHGC resulted in lower saving when compared to calculated savings. Though Wyoming is in IECC Zones 6 and 7, IECC 2009 had no recommended minimum SHGC for windows in these heating-dominated climate zones. Saving gained in the summer by a low SHGC window would be lost in the winter due to these windows blocking solar heat from the sun, explaining why realized savings in regard to electric heating equipment is lower than calculated savings.

Realized saving for models with air conditioning equipment are higher than without air conditioning due to the requirement of 0.33 SGCH.

The differences between realized savings modeled with Energy Gauge and Rem-Rate can be partially explained by each model treating shading on the windows of the home differently and partially by the calculation method of each model.

Table K21. Window Annual Savings Summary


Name

PECI Gross Savings


PECI Gross

Savings Reported

(kWh/ unit)

Cadmus Gross

Savings Using

Energy Gauge

(kWh/ unit)

Cadmus Gross

Savings Realized

(kWh/ unit)

Realization PECI

Reported and


20092010

Windows: Electric Heat w/o CAC 4.2A 2.32 2.2 1.07 46%

Window Area

20092010

Windows: Electric Heat w/CAC

5.3A 2.34 1.0 2.63 112% Window Area

20092010

Windows: Gas Heat w/CAC

1.04A 0.02 1.6 1.71 8,500% Window Area

A Energy Gauge Model (Efficient Window U-value 0.30; SHGC 0.45) B Rem-Rate Model (Efficient Window U-value 0.35; SHGC 0.33)

Detailed Results by Measure Type – Duct Sealing Table K22 below shows the different savings values both observed and calculated across duct sealing measures. In addition, each value in the table has a reference that indicates the source of the savings value. The following was observed when examining the results for duct sealing measures:

No savings were reported for duct sealing measures since there were no participants in duct sealing.

PECI sourced duct sealing calculated savings for the 2009 program year from RTF values, which values were weighted for heating system types. For 2010 calculated savings, an Energy Gauge model was used to model a 433 CFM @ 25Pa reduction in leakage with R-6 insulation added to the duct system.



A 433 CFM @25Pa reduction in duct leakage was assumed from an initial 499.4 CFM @ 25Pa leakage in the PECI Energy Gauge model. A 499.4 CFM @ 25Pa for the 2,200 sq-ft home would result in a 23 CFM @25 Pa leakage rate per 100 sq ft of floor area. New homes built to the IECC 2009 standard were required to meet requirements of 12 CFM @25Pa per 100 sq ft; so a baseline leakage twice that rate would be very reasonable for Wyoming. A 433 CFM @25 Pa leakage reduction would result in an 86 percent reduction in duct leakage. The PECI documents’ 2010 savings summary projected a 50 percent reduction in duct leakage, which would be much more reasonable for the HES Program. This would indicate ducts were sealed to meet the new standard IECC 2009.

Duct savings modeled in both Rem-Rate and Energy Gauge provided very comparable savings. The major conflict between the models appeared in electric fan power savings for duct sealing, modeled with a gas furnace only in Energy Gauge. The much lower electric savings resulted from the method Energy Gauge uses to calculate the fan runtime, based on modeled system’s airflow. Cadmus disagrees with Rem-Rates modeled savings since the model is showing an unreasonable reduction of fan power usage.

The lower savings resulting when comparing Rem-Rate savings to PECI resulted from a higher CFM @ 25Pa for an efficient duct condition, along with other modeling differences. A 250 CFM @ 25Pa rate was modeled to represent the 50 percent reduction in duct leakage.

Table K22. Duct Sealing Annual Savings Summary


Name

PECI Gross Savings


PECI Gross

Savings Reported

(kWh/ unit)

Cadmus Gross

Savings Using

Energy Gauge

(kWh/ unit)

Cadmus Gross

Savings Realized

(kWh/ unit)

Realization PECI

Reported and


2009 Duct Sealing: Electric Heat

2,151.2A – 5,565 5,766B – Household

2010 Duct Sealing: Electric Heat w/o CAC

9,584C – 5,565 5,766B – Household

2010 Duct Sealing: Electric Heat w/CAC

10,035C – 6,125 6,048B – Household

2009 Duct Sealing: Gas Heat 18.1A – 35 35C – Household

2010 Duct Sealing: Gas Heat w/CAC 451C – 728 740B – Household

A Regional Technical Forum (RTF) B Rem-Rate Model C Energy Gauge Model



Detailed Results by Measure Type – Water Heaters Table 52 below shows the different savings values both observed and calculated across the water heater measure. In addition, each value in the table has a reference that indicates the source of the savings value. The following was observed when examining the results of the water heater measure:

Reported savings for 2009 and 2010, 90.7 kWh were based upon upgrading from a 0.90 energy factor (EF) 40-gallon water heater to a 0.93 EF 40-gallon water heater. PECI calculated savings for 2010 of 125 kWh, assuming upgrading from a 0.90 EF to 0.94 EF unit, but never adapted these as reported savings. The HES Program required purchasing units at or above a 0.93 EF and 40 gallons.

Realized savings of 110 kWh were estimated based on the following inputs to the Rem-rate model -- replacing a baseline 0.90 EF 50 gallon unit with a 0.93 EF 50 gallon electric water heater. Cadmus developed these inputs by examining PECI’s tracking database. Energy Gauge provided similar estimates of savings, 119 kWh, using these same inputs.

Table K52. Water Heater Annual Savings Summary


Name

PECI Gross Savings


PECI Gross

Savings Reported

(kWh/ unit)

Cadmus Gross

Savings Using

Energy Gauge

(kWh/ unit)

Cadmus Gross

Savings Realized

(kWh/ unit)

Realization PECI

Reported and


2009 Electric Water Heaters 90.7 90.7 119 110 121% water heater 2010 Electric Water Heaters 125 90.7 119 110 121% water heater

A Regional Technical Forum (RTF) B 6th Power Plan C Rem-Rate Model

final report rocky mountain power wyoming residential home energy

Documents