ex ante measure cost study research plan - … · related to particular types of measures that have...

Ex Ante Measure Cost Study Research Plan

Final

Itron, Inc. 1111 Broadway, Suite 1800

Oakland, CA 94607

(510) 844-2800

January 31, 2012

Itron, Inc. i Table of Contents

Table of Contents

1 Introduction ........................................................................................................ 1-1

1.1 Regulatory Use of Ex Ante Measure Costs ................................................. 1-1 1.2 Scope and Objectives of MCS Update ........................................................ 1-2 1.3 Key Analytic Challenges in MCS Studies .................................................... 1-4

1.3.1 Basic Analytic Challenges .......................................................................................... 1-4 1.3.2 Special Analytic Challenges ....................................................................................... 1-5 1.3.3 Policy-Related Analytic Challenges ............................................................................ 1-6

2 Scope of Work .................................................................................................... 2-1

2.1 Task 1: Needs Assessment and Related Methodology Research .............. 2-2 2.1.1 Task 1.1 Needs Assessment ...................................................................................... 2-2 2.1.2 Task 1.2 Needs-related Methodology Research ........................................................ 2-3 2.1.3 Task 1 Deliverables .................................................................................................... 2-4

2.2 Task 2: Identify Qualified Subcontractors ................................................. 2-5 2.2.1 Task 2 Deliverables .................................................................................................... 2-5

2.3 Task 3: Custom and New Construction Measure Cost Research ............... 2-5 2.3.1 Identify Key Cost Estimation Issues and Options ...................................................... 2-6 2.3.2 Develop and Test Methods and Data Sources .......................................................... 2-7 2.3.3 Develop Guidelines, Tools, and Recommendations .................................................. 2-7 2.3.4 Task 3 Deliverables .................................................................................................... 2-8

2.4 Task 4: Define Measure and Technology List and Analysis Scope ........... 2-8 2.4.1 Task 4.1 Define Measure and Technology List and Specifications ........................... 2-9 2.4.2 Task 4.2 Define Prioritization Criteria and Process .................................................... 2-9 2.4.3 Task 4.3 Define Cost Data Segmentation and Cost Cases ..................................... 2-10 2.4.4 Task 4 Deliverables .................................................................................................. 2-12

2.5 Task 5: Develop Data Collection Strategies ............................................. 2-13 2.5.1 Data Collection Options Using External Resources ................................................. 2-13 2.5.2 Data Collection Options Using Internal Resources .................................................. 2-14 2.5.3 Task 5 Deliverables .................................................................................................. 2-15

2.6 Task 6: Implement Data Collection Strategies ......................................... 2-16 2.6.1 Task 6 Deliverables .................................................................................................. 2-17

2.7 Task 7: Conduct Data Analysis ............................................................... 2-17 2.7.1 Task 7.1 Data Cleaning ............................................................................................ 2-17 2.7.2 Task 7.2 Cost Modeling ............................................................................................ 2-18 2.7.3 Task 7.3 Cost Forecasting ....................................................................................... 2-19 2.7.4 Task 7.4 Uncertainty Analysis and Benchmarking ................................................... 2-19 2.7.5 Task 7.5 Database of Recommended Values .......................................................... 2-19 2.7.6 Task 7 Deliverables .................................................................................................. 2-19

2.8 Task 8: Reporting ..................................................................................... 2-20 2.8.1 Task 8 Deliverables .................................................................................................. 2-20

2.9 Task 9: Integration of Final Ex Ante Measure Costs into DEER .............. 2-21 2.9.1 Task 9 Deliverables .................................................................................................. 2-21

2.10 Task 10: Project Management ............................................................... 2-21 2.10.1 Task 10 Deliverables .............................................................................................. 2-21

3 Project Schedule and Stakeholder Engagement ............................................. 3-1


Itron, Inc. ii Table of Contents

3.1 Project Schedule and Milestones ................................................................ 3-1 3.2 Stakeholder Engagement and Early Feedback ........................................... 3-2

4 Task Budgets ...................................................................................................... 4-1

List of Tables

Table 3-1: Project Timeline by Task ....................................................................... 3-1

Table 3-2: Schedule of Early Feedback Deliverables ............................................. 3-2

Table 4-1: Initial Budget Allocation by Task ........................................................... 4-1

Table 4-2: Anticipated Staged Budget Authorizations ............................................ 4-3

List of Figures

Figure 2-1: Flowchart of Overall Research Process and Tasks for 2010-2012 MCS ................................................................................................................. 2-1

Itron, Inc. 1-1 Introduction

1 Introduction

As part of the portfolio of 2010-2012 EM&V activities, the Database for Energy Efficient Resources (DEER) and the ex ante values contained therein are being updated with the best and latest information available. In the case of measure costs, these updates (and the associated research and data development) will be conducted through both a stand-alone Measure Cost Study (MCS) and, where efficient, necessary, and appropriate, through cost-related data collection and research conducted as part of other EM&V studies (e.g., impact evaluations, process evaluations, and market studies). The studies will provide the California Public Utilities Commission (CPUC) and the Investor Owned Utilities (IOUs) with improved estimates of measure costs to support fulfillment of CPUC policy requirements.

This work is under the direction of the Energy Division (ED) with input from ED’s DEER and energy efficiency cost-effectiveness teams. For measures contained in the DEER database, incremental measure costs will be developed to be incorporated into, and directly aligned with, the ex ante energy savings estimates within the DEER database. For non-DEER measures in the IOU program portfolios, measure costs will also be estimated and analyzed to align with and support application in ex post program and portfolio cost-effectiveness analyses.

Itron, Inc. serves as the Prime Contractor managing this study, lead by Mr. Mike Ting. The ED Project Manager for this study is Mr. Jeorge Tagnipes. The following is Mr. Ting’s contact information:

Firm Lead Contact Info

Itron, Inc 1111 Broadway, Suite 1800 Oakland, CA 94607

Mike Ting, Principal Energy Consultant

Phone: (510)844-2883 Fax: (510)844-2900 Email: [email protected]

1.1 Regulatory Use of Ex Ante Measure Costs

For each program cycle, the California IOUs are required under CPUC policy to file ex ante estimates of the expected per-unit costs and savings for each measure included in proposed



resource acquisition programs.1 These values are then used by the CPUC to assess the overall cost-effectiveness of the proposed portfolios and by the IOUs to track program accomplishments against annual savings goals over the course of the program cycle.2,3 Additionally, ex ante savings values have been used in the determination of the level of rewards the IOUs can receive for successful energy efficiency efforts.4

For deemed measures contained in the CPUC-managed Database for Energy Efficient Resources (DEER), the IOUs are required to use the corresponding ex ante values from DEER. For deemed measures not contained in DEER, the IOUs must submit workpapers that estimate and justify proposed ex ante values for each non-DEER measure. These workpapers must be then reviewed and approved by ED before the corresponding ex ante values can be applied to assess cost-effectiveness or to track and report accomplishments.

1.2 Scope and Objectives of MCS Update

The primary objective of this MCS update is to develop ex ante measure cost estimates for measures supported by IOU programs in the current program cycle and likely to be supported by IOU programs in the next program cycle. The scope of this measure cost update therefore includes all deemed measures contained in the DEER database, as well as non-DEER deemed measures and custom measures supported by “calculated” incentive programs. This scope is distinctly different from those in previous measure cost studies, which have been strictly limited to DEER measures.

Importantly, because the scope of this study includes non-DEER measures, this MCS update will focus not only on developing ex ante measure cost values but also on developing the methods,

1 Ex ante values are energy efficiency impact and cost estimates developed prior to or during implementation of

programs and prior to installation of the measures by end users. Ex post values are estimates of savings that are informed by observation and measurement of savings or factors related to savings that occur after energy efficiency measures have been implemented.

2 For a detailed overview and discussion of current CPUC policy related to the use of ex ante estimates, see Decision Approving 2010 to 2012 Energy Efficiency Portfolios and Budgets, D.09-09-047 (September 24, 2009) and Third Decision Addressing Petition for Modification of Decision 09-09-047, D.11-07-030 (July 22, 2011).

3 Currently, the two primary cost-effectiveness tests used to inform energy efficiency policy making (including goal setting, portfolio cost-effectiveness requirements, and the risk-reward incentive mechanism) are the total resource cost test (TRC) and the program administrator test (PAC). The TRC test requires program costs, incentive costs, and participant costs. Program costs and incentive costs are the two principal cost elements for the PAC test.

4 See Section VIII (Performance-Based Risk and Reward Incentive Mechanism) of the Energy Efficiency Policy Manual, version 4.0 available at: http://www.cpuc.ca.gov/NR/rdonlyres/FCE88E10-C186-479F-BFFF-CB722750B1AA/0/CPUCEnergyEfficiencyPolicyManual.doc. Note, however, that the CPUC is currently soliciting comments from parties on the design of future risk/reward mechanisms, including the use of ex ante values, in Rulemaking 09-01-019.



tools, and data sources to support standardized estimation of ex ante measure costs for non-DEER measures going forward.

Costs vs. Prices, Measures vs. Technologies, Measure Costs vs. Incremental Measure Costs

Note that the term “cost” will be used in this study, as it was in previous measure cost studies due to common usage of this term in cost-benefit applications. Technically, however, the study deliverables are estimates of the prices paid by customers for energy-efficient products and services. This distinction is important because, in the economics and business literature, the term “cost” usually refers to production costs and/or opportunity costs borne by businesses when producing a particular good or service.5

Similarly, the term “measure” will be used throughout this study. In general, “measures” refers to interventions that increase the efficiency performance of an energy-consuming system. In practice, energy efficiency measures are often the replacement of one technology for another. Robust incremental cost accounting thus requires developing prices and cost streams for both high-efficiency technologies and their in-situ or standard-efficiency counterparts. In this respect, the primary focus of this research is estimating prices and cost streams for both high-efficiency technologies and their in-situ or standard-efficiency counterparts, from which incremental measure costs can then be estimated under a variety of different implementation contexts (e.g. early replacement, replace-on-burnout, etc).

Finally, the term “measure costs” will be used generically throughout this study. Strictly speaking, however, what determines the cost-effectiveness of measures and programs to society (as represented in the TRC test) and what program administrators typically use to set incentive levels is the incremental cost of energy efficiency measures, i.e. the additional cost associated with adopting a high-efficiency technology compared to a standard- or average-efficiency technology. Depending on the type of measure, the incremental measure cost may in fact be the full measure cost (e.g. for add-on measures like pipe insulation), the difference in costs between a base technology and replacement technology at two different points in time (e.g., early replacement), or it may be the difference between the full measure cost and the cost of the standard-efficiency counterpart (e.g. for replace-on-burnout measures like SEER 15 central air conditioners). In this study, we will be developing both full measure costs and incremental measure costs and will use the umbrella term “measure costs” to generically refer to both.

5 See, for example, the definition of “cost” offered at: http://en.wikipedia.org/wiki/Cost.



1.3 Key Analytic Challenges in MCS Studies

To date, six measure cost studies have been conducted in California since the early 1990s (XENERGY, 1992; XENERGY, 1994; XENERGY, 1996; XENERGY, 2001; Navigant, 2005; Navigant, 2009). While the current MCS update has the advantage of building directly upon the lessons learned and knowledge gained from the previous six studies, the basic analytic challenges to developing robust ex ante estimates of incremental measure costs remain largely the same. Additionally, there are a host of new analytic challenges for the current MCS update related to particular types of measures that have not been in the scope of previous MCS efforts (e.g. custom industrial measures). In this section, we present and describe these basic analytic challenges as a backdrop to the scope of work proposed in Section 2. We then present and describe the additional analytic challenges associated with measures that have not been included in previous MCS efforts. Finally, we discuss key analytic challenges that stem from the regulatory application of ex ante measure costs in California.

1.3.1 Basic Analytic Challenges

In all measure cost studies, the core objective is to estimate the average incremental costs associated with a high-efficiency technology or measure. For add-on measures such as attic insulation and occupancy sensors for lighting, the incremental costs are typically equivalent to the total installed cost of the measure. However, for early replacement (ER) and replace-on-burnout (ROB) measures such as central air conditioners or water heaters, this requires, at a minimum, a “matched pair” analysis where price and labor cost data must be collected and developed for both a particular high-efficiency technology and its standard-efficiency counterpart. In this sense, developing data sets of matched pairs to support incremental cost estimates for ER and ROB measures requires roughly twice the number of price observations compared to that required for retrofit measures and introduces the additional analytic challenge of determining (with minimal bias) the appropriate baseline technology from which to estimate the incremental costs of the high-efficiency version.6

Regardless of whether a measure is retrofit or ROB, isolating price differences that are attributable strictly to differences in energy efficiency – rather than differences in product features that do not impact energy consumption – is another basic analytic challenge in estimating incremental measure costs. Indeed, the six previous MCS studies in California have demonstrated that price differences related to non-energy related feature sets have a significant influence on total retail prices paid by customers, due to breadth and scope of measures included

6 Note that for some types of measures (e.g. high-definition televisions), the correct “matched pair” may not

necessarily be the standard-efficiency unit with the exact same specifications and feature set in cases where there is evidence that consumers are purchasing program-qualified units with markedly different specifications and features (e.g. larger screen sizes). The issues associated with determining the appropriate “cost case” from which to estimate incremental costs for a given measure are discussed further in Section 2.4.3.



in the DEER database, which in turn requires a variety of different approaches to adequately isolate the price differences due strictly to energy efficiency performance.

The two basic analytic challenges described above are associated with accurately estimating incremental differences in the retail price, or first cost, of high-efficiency technologies compared to their standard-efficiency counterparts. However, the TRC test and indeed most investment decision-making models used in the business world (e.g. internal rate of return) often use lifecycle costs, not first costs, to assess the cost-effectiveness of investments in energy efficiency. Reliably estimating lifecycle costs for energy efficiency measures requires estimating a host of other cost parameters that have largely not been within the scope of previous MCS efforts and are relatively understudied compared to first costs. Examples of lifecycle cost parameters that will be evaluated for inclusion in this MCS effort include the average remaining useful life (RUL) of in-situ equipment targeted for early retirement, recurring costs associated with operation and maintenance, salvage values, and disposal costs.

Finally, reconciling the natural tension between timeliness, accuracy, and breadth – i.e., balancing the desire to produce highly precise and up-to-date incremental cost estimates for all measures with the realities of resource and calendar constraints – is another basic analytic challenge in all MCS studies. Addressing this challenge requires prioritization of data collection and analytic activities. While this natural tension exists in most research studies across many fields, this basic challenge is particularly relevant in California MCS studies due to the breadth and scope of the measures included in the DEER database and offered through utility programs by the California IOUs.

1.3.2 Special Analytic Challenges

In addition to the basic analytic challenges described above, there are also a host of additional analytic challenges that are specific to estimating incremental measures costs for particular types of energy efficiency measures. Below we describe the special analytic challenges associated specifically with new construction measures and non-residential custom measures.

For energy efficiency measures in the new construction segment, the principal analytic challenge when estimating incremental measure costs is accounting for the interaction between building design choices and equipment selection and sizing (and therefore cost), particularly for HVAC. For example, residential new construction designs that include tight building shells, high levels of insulation, and/or passive solar design can sometimes allow HVAC systems to be downsized and in turn reduce the incremental cost associated with choosing high-efficiency HVAC equipment.7 Because of these interactions, estimating incremental costs for new construction

7 In commercial new construction, the design choices that can affect HVAC system sizing (and therefore

incremental costs) are typically related to window/day-lighting systems and non-human internal loads (e.g. lighting and office equipment).



measures individually is often ambiguous and introduces added uncertainty. Additionally, multiple different new construction designs (and resulting equipment choices) can lead to similar or identical energy performance at the end-use or whole-building level, further confounding efforts to estimate incremental costs for new construction measures on an individual basis. To avoid this source of uncertainty and more accurately reflect the diversity of new construction designs and approaches, it may prove appropriate to develop “whole-building” incremental costs for new construction designs that exceed Title 24 energy performance benchmarks by a given amount (e.g. 5%, 15%, or 30%). However, for such “whole-building” incremental costs to be meaningful and defensible in a regulatory context, one must necessarily assess (and develop cost estimates for) the full range of new construction designs that produce the same level of energy performance, which presents a host of unique and difficult data collection and analysis issues.

In the large commercial and industrial (C&I) customer segment, while some energy efficiency measures are applicable to systems found across nearly all large C&I customers (e.g. high-efficiency lighting, high-efficiency motors, variable speed drives), many large C&I energy efficiency measures are highly customized to the particular customer involved, which in turn makes it difficult, if not impossible, to reasonably estimate average measure-level costs. In this sense, the primary analytic challenge associated with large C&I energy efficiency projects is to first develop methodologies, tools, and data collection guidelines (rather than pre-determined ex ante values) that could in turn provide standardized, comparable, and meaningful incremental cost estimates for such custom measures for use in the CPUC’s ex-ante activities.

1.3.3 Policy-Related Analytic Challenges

In addition to the analytic challenges described above, an additional analytic challenge faced by this MCS study is related to the policy and regulatory use of ex ante measure parameters in the CPUC’s regulatory processes. Specifically, the primary regulatory application of ex ante measure cost values is to assess the cost-effectiveness of proposed IOU program portfolios, i.e. proposed for the next program cycle, to track progress towards annual savings and cost-effectiveness goals over the course of the program cycle, and, in the case of custom measures, to be a part of Energy Division review prior to project approval if requested. Currently, IOU programs operate on a 3-year cycle. When developing updates to ex ante measure costs estimates in the current MCS update, therefore, the focus must be on producing ex ante parameter estimates that are applicable for measures and programs that will be implemented over the next program cycle (the CPUC is concurrently considering whether to extend the current cycle from 2012 to 2013 or 2014).

This forecasting dimension of developing robust ex ante measure parameters is particularly significant in the case of incremental measure costs. Indeed, certain technology markets have been demonstrated to be quite dynamic in recent years with respect to product prices. For example, the Residential Market Share Tracking (RMST) studies conducted by Itron annually



between 1999 and 2007 have clearly shown that CFL product prices have declined significantly, and the same has been observed in other technology markets such as LEDs. The analytic challenge for this MCS study, therefore, is to identify technology markets that are experiencing product prices changes, the magnitude of these changes over various periods of time, and develop methods and data by which to defensibly forecast changes in product prices expected to occur over the 2013-2014 time period.

Itron, Inc. 2-1 Scope of Work

2 Scope of Work

Having described the regulatory use of ex ante incremental measure costs in California and the analytic challenges associated with developing them, this section presents the overall research plan for the 2010-2012 MCS and describes the specific tasks that will be conducted and the interim and final deliverables that will be produced. This overall process is summarized in Figure 2-1 below.

Figure 2-1: Flowchart of Overall Research Process and Tasks for 2010-2012 MCS

Due to the breadth and scope of the energy efficiency measures contained in the IOU program portfolios (and the diversity and complexity of the associated analytic challenges), it is

Develop data collection strategies

Implement data collection strategies

Raw cost data set

Data cleaning, cost modeling & forecasting

Integrate final results into DEER

Final report and documentation

Develop/test approaches and data sources for estimating ex ante costs

Recommended methods, tools, and data collection processes

DEER and non‐DEER Deemed Custom and NC

Define measure/technology list and scope

Identify qualified subcontractors

Identify key ex ante cost estimation issues and needs

Needs assessment and related methodology research



unreasonable for the study team to attempt to specify a priori the exact data sources and methods that will be used for each in-scope measure at this time. Rather, this research plan is designed to produce a series of interim deliverables that allow detailed, measure-specific data collection and analysis plans to be developed, reviewed, approved, and executed as the data sources and analytic options for each in-scope measure are identified and assessed. In this respect, this research plan lays out the overall process by which measure-specific data collection and analysis plans will be developed and implemented.

In the subsections that follow, we present and describe the scope of the tasks shown in Figure 2-1 in more detail, along with the interim and final deliverables associated with each.

2.1 Task 1: Needs Assessment and Related Methodology Research

In Task 1, Itron will conduct a needs assessment related to ex ante measure costs and their role in CPUC regulatory processes and IOU program planning, both currently and going forward. Based on the outcomes of the needs assessment, Itron will then conduct research to develop and test methodological approaches to address those needs. These activities are described in more detail in the subsections that follow.

The objective of this task is to accurately frame the expectations for the MCS beyond what is stated explicitly in currently policy and ensure, to the extent possible with the resources and time available, that the scope, methods, and outputs of the MCS align with the highest-priority needs of the CPUC and its stakeholders.

2.1.1 Task 1.1 Needs Assessment

In the first phase of Task 1, Itron will conduct a needs assessment among staff from Energy Division, the IOUs, and other stakeholders. The needs assessment will be designed to precisely identify the regulatory and program planning needs of the CPUC and its stakeholders related to ex ante measure costs, both currently and going forward, and to help frame the corresponding expectations for the MCS to ensure that its outputs are aligned, to the extent possible, with the highest priority needs.

To conduct the needs assessment, Itron will develop a written questionnaire that will allow participants to identify, describe, and rank their respective needs related to ex ante measure costs. The questionnaire will be designed to solicit perspectives related to questions including, but not limited to, the following:

The need for ex ante measure cost values to support backwards-looking applications like cost-effectiveness versus forward-looking applications like program planning, filings, reporting?



The need for ex ante measure cost needs at the tracking level (e.g. SPT) versus the planning level (e.g. DEER or even greater levels of measure aggregation)?

The need for ex ante measure costs for deemed measures versus custom measures?

The need for ex ante measure cost values to support NPV and lifecycle costing?

The need for ex ante measure cost values to support refinement of the inputs to and requirements of the TRC and PAC tests?

The need for ex ante measure costs that include a portion of program costs (e.g. for direct install measures)?

The questionnaire will be sent to key staff from Energy Division, the IOUs, and other stakeholders. Itron will then convene either an in-person or telephone meeting with the same group to discuss stakeholder needs, using the results of the questionnaires to organize and guide the group discussion. Areas of clear consensus, as well as areas of diversity, will be documented in the form of a written memorandum summarizing the results of the needs assessment.

2.1.2 Task 1.2 Needs-related Methodology Research

Following the needs assessment described above, Itron will then conduct research to develop and test methodological approaches to address clearly identified needs, with a primary focus on those needs considered to be highest priority by Energy Division.

An example of the type of research likely to be conducted in this task is related to the (assumed) need for costing methodologies that allow for high degrees of flexibility and/or multiple levels of granularity. For purposes of this discussion, flexibility refers to the ability to produce defensible cost estimates for a variety of different measure specifications without the need for additional, measure-specific data collection (e.g. across ranges of capacities, SEER levels, or wattage levels). Granularity refers to the ability to produce cost estimates for both a wide variety of very specific measures and more aggregate, prototypical measures using the same basic analytic framework and data sources.

In previous MCS efforts, the overall methodological approach has been to collect data and produce costs for each pre-specified, individual measure in the scope of the study (e.g. in DEER). However, methodological approaches that exhibit flexibility could be desirable and needed in environments where measure specifications are anticipated to change regularly or otherwise be in flux over time. Similarly, methods that are capable of producing cost estimates across multiple levels of measure granularity could be needed where internal consistency between highly granular cost estimates (e.g. for tracking) and more aggregate, prototypical cost estimates (e.g. for planning) is highly valued and desired.



Examples of potential methodological approaches that could address needs related to flexibility and granularity include, but are not limited to the following:

Taking a sample-based approach, where one would collect data and develop costs for a random sample of individual measures in a measure group, or

Taking an extrapolation approach, where one would collect data and develop costs for a set of prototypical measures (as defined by their relative frequency/importance in the portfolio) and then extrapolate those costs to the rest of the measures in a measure group based on one or more relevant technology parameters (e.g. capacity, SEER rating, or wattage).

Examples of other needs-related methodological research areas include:

Methods and approaches to forecast equipment prices over the short- and medium-term,8

Methods and approaches to determine where and how measure-specific program costs should be included in ex ante measure cost values, and

Lifecycle costing frameworks suitable for energy efficiency measures and their corresponding application in cost-effectiveness tests.

Note that methodological research related specifically to how best to estimate ex ante measure costs for custom and new construction measures will be done separately in Task 3.

Depending on the outcomes of the needs assessment, Itron will develop and test methodological approaches to address the highest-priority needs. Findings from this research and how those findings will be applied in the MCS update will be summarized in the form of a technical memorandum.

2.1.3 Task 1 Deliverables

The specific deliverables that will be produced under Task 1 are listed below:

Written questionnaire that will allow staff from Energy Division, the IOUs, and other stakeholders to identify, describe, and rank their respective needs related to ex ante measure costs (draft and final)

Memorandum summarizing the findings from needs assessment (draft and final)

8 See, for example, the equipment price forecasting methodology recently proposed by the US Department of

Energy for use in its appliance standards rulemaking proceedings. Available at: http://www1.eere.energy.gov/buildings/appliance_standards/supplemental_info_equipment_price_forecasting.html.



Memorandum summarizing the findings and recommendations from needs-related methodological research (draft and final)

Note that each of these deliverables will be produced first in draft form and circulated to the project oversight team for review. The final deliverables will integrate and address comments raised by the project oversight team as deemed appropriate by the ED Project Manager.

2.2 Task 2: Identify Qualified Subcontractors

Given the breadth and scope of the measures in the IOU program portfolios, neither Itron nor KEMA have all of the required technology-specific expertise to cost-effectively develop robust incremental cost estimates for all measures in the scope of the MCS. As such, the primary objective in Task 2 will be to develop a Request for Qualifications (RFQ) that is strategically designed to identify qualified subcontractors that can readily provide the raw measure cost data required for this study and/or have demonstrated experience in related measure cost data development and analysis. Note that Task 2 will be conducted in parallel to Task 1.9



A complete RFQ designed to identify high-leverage subcontractors that can readily provide the raw measure cost data required for this study and/or have experience in related measure cost data development and analysis (draft and final)

A list of solicitation candidates and distribution outlets that will be used to circulate the final RFQ to potential subcontractors (draft and final)


2.3 Task 3: Custom and New Construction Measure Cost Research

As discussed earlier in Section 1.3.2, there are a host of unique analytic issues associated with estimating incremental costs for new construction measures and non-residential custom measures. Because neither of these categories of measures has been in the scope of previous

9 This task is currently underway. The RFQ was first released on July 8, 2011 but did not result in a sufficient

number of responses to address the needs of the study. The RFQ was subsequently revised and re-released on September 19, 2011 and submissions are due October 12, 2011. The final RFQ document is available at: http://eega.cpuc.ca.gov/Docs/REQUEST%20FOR%20QUALIFICATIONS%20-%20Final_9.19.11.docx.



MCS efforts, the methods and approaches for developing robust ex ante measure cost estimates are not as well-developed as those used for deemed measures.10 Nonetheless, custom and new construction measures play an important role in the overall IOU program portfolio and are forecasted by the IOUs to account for roughly 30% of total claimed electric energy and peak demand savings and 70% of total claimed gas savings from IOU programs in the current cycle.

To address this knowledge gap, Task 3 will focus not only on developing ex ante measure cost values for custom and new construction measures but also on developing the methods, tools, and data to support standardized estimation of ex ante measure costs for such “calculated” incentive measures going forward. Specifically, Task 3 will take the form of three sub-studies – one each for residential new construction (RNC), commercial new construction (CNC), and non-residential custom measures. Each of these sub-studies will be comprised of three main steps, each of which is described below.

2.3.1 Identify Key Cost Estimation Issues and Options

The first step for each of the Task 3 sub-studies will be to identify and assess the key measure cost estimation issues and options related to estimating incremental measure costs for new construction and custom energy efficiency measures. The objective of this assessment is to develop a comprehensive characterization of the analytic, data availability, and data collection issues and identify possible methodological approaches and data sources that could be used to produce defensible ex ante measure cost estimates for use in the CPUC’s ex ante activities.

This assessment will build upon the current state of knowledge among IOU and third party implementers, third party program evaluators, the IOU work paper analyses and reviews produced by the CPUC’s Data Management and Quality Control (DMQC) contractors, and other ED consultants. The assessment will take into account the quantity and quality of measure cost data collected in previous program cycles, the recently-established Calculation Tool Archive (CTA) and Custom Measure and Project Archive (CMPA), and opportunities to integrate augmented and/or improved cost data collection into current program reporting and evaluation efforts.11 Finally, the assessment will identify and describe in detail the specific analytic approaches that offer the best potential to produce defensible ex ante measure cost estimates that are appropriate for use in the CPUC’s ex ante activities.

The final assessment will take the form of a technical memorandum that will be presented to ED and the project oversight team for review.

10 See Heschong Mahone Group (2009) for a review and discussion of the history (or lack thereof) of efforts to

develop costing methodologies for custom energy efficiency measures and new construction projects. Available at: http://www.calmac.org/publications/HMG_IMC_White_Paper_v3_Final.pdf.

11 The IOUs were recently ordered to establish the CTA and CMPA in D.11-07-030 and are required to update both archives on a bi-weekly basis through the course of the 2010-2012 program cycle.



2.3.2 Develop and Test Methods and Data Sources

The second step for each of the Task 3 sub-studies will be to develop and test methods and data sources needed to produce ex ante measure costs for custom and new construction measures in the context of the CPUC’s ex ante activities. These methods and tools will be designed to directly leverage and align with the findings from Task 1.

Among the approaches that will likely be developed and tested for custom efficiency measures is an extrapolation approach based on developing average $/kWh saved from actual records or prototypical project costs that can then be extrapolated to the participant population.

For new construction, one particular approach that will likely be tested is to build upon and expand the analysis approach used for commercial new construction in the California Energy Efficiency Potential Study (Itron, 2006).12 Specifically, the approach would develop normalized “whole-building” incremental costs (e.g. $/ft2) for a wide range of measure packages that deliver similar whole-building performance improvements beyond Title 24 (e.g. 10%, 20%, and 30%) based on a set of prototypical buildings (perhaps defined by DEER building prototypes). This potential approach would expand upon that used in the 2006 Potential Study in terms of the number and content of measure packages analyzed, the number of prototypical buildings considered, and/or the types of costs included in the overall incremental cost analysis (e.g. design and other “soft” costs not included in previous efforts).

The methods and data sources that offer the greatest capacity to produce defensible ex ante measure cost estimates for custom and new construction measures will then be summarized and presented to ED, and the project oversight team for review. Methods and data sources that are determined to be appropriate for use in the CPUC’s ex ante activities will then be fully documented in a technical memorandum, and the corresponding ex ante cost values will be delivered to ED in a format specified by the ED PM.

2.3.3 Develop Guidelines, Tools, and Recommendations

The final step for each of the Task 3 sub-studies will be to develop data collection guidelines and recommendations for the calculation and reporting of ex ante measure costs for custom and new construction measures going forward. These guidelines and recommendations will identify specific opportunities to integrate augmented and/or improved cost data collection into current program reporting requirements and evaluation efforts.

To the extent that the research conducted in Task 3 yields methods and data sources that are determined to be appropriate for use in the CPUC’s ex ante activities, this final step will also

12 The methods and data sources used in the commercial new construction analysis are documented in Appendix P.

Available at: http://www.calmac.org/publications/PGE_PotentialStudy_AppEtoS_Vol2_05242006.pdf



include development of analytic tools to support the use and further refinement of such methods by the CPUC, the IOUs, and other stakeholders going forward. These analytic tools will be developed in a platform approved by ED and the project oversight team and will be accompanied by complete written documentation and guidelines for use.



A technical memorandum that provides a comprehensive characterization of the analytic, data availability, and data collection issues associated with estimating custom and new construction measure costs and identifies possible methodological approaches and data sources that could be used to produce defensible ex ante measure cost estimates for use in the CPUC’s ex ante activities (draft and final)

A technical memorandum that documents the specific methods and data sources approved by ED and the project oversight team to produce defensible ex ante measure cost estimates for custom and new construction measures (draft and final)

A technical memorandum that provides data collection guidelines and recommendations for the calculation and reporting of ex ante measure costs for custom and new construction measures going forward (draft and final)

Analytic tools to support the use and further refinement of recommended methods by the CPUC, the IOUs, and other stakeholders going forward in a platform specified by ED, including complete written documentation and guidelines for use (draft and final)


2.4 Task 4: Define Measure and Technology List and Analysis Scope

In Task 3, the primary objective will be to precisely define the analytic scope of the 2010-2012 MCS in terms of the in-scope deemed measures (and associated measure specifications), the segmentation that will be used to define technology markets, the cost cases from which to estimate incremental costs for each measure, and the prioritization criteria and process that will be used to guide the allocation of financial and analytic resources over the course of the study. Each of these subtasks is described in more detail below, along with the anticipated deliverables from Task 3 activities.



2.4.1 Task 4.1 Define Measure and Technology List and Specifications

For DEER measures, one of the basic requirements of the MCS will be to produce ex ante incremental cost estimates that are internally consistent (in terms of analytic boundaries) with the corresponding ex ante energy and peak savings, effective useful life, and net-to-gross values being developed in parallel by the DEER team. As such, developing the in-scope deemed measure list and corresponding measure definitions (e.g. technology-specific capacity ranges and efficiency levels) for the 2010-2012 MCS will require close coordination with key members of the DEER management team and the DEER technology groups.

We will also coordinate closely with the Energy Division team to identify non-DEER deemed measures that are high-value candidates to be included in the scope of the 2010-2012 MCS. For all such measures identified, we will then work with ED to develop precise measure definitions to be used in MCS data collection and analysis activities. This activity will be designed to explicitly leverage the information contained in the IOU’s ex ante work papers as well as ED’s reviews and analyses of those work papers.

Finally, it should be noted that we will also coordinate with the Portfolio Management and Audit Assessment (PMAA) study, Work Order 18, which includes a study element focused on analysis of program and portfolio costs. The Portfolio Management team is coordinating with the CPUC’s Utility Audit, Finance and Compliance Branch (UAFCB), which is conducting its own financial audit activities, specifically those assessing and quantifying the IOU’s program-related administrative costs. The MCS team will work through the PMAA team (also managed by Itron) to coordinate data collection needs and results with the UAFCB, as useful and appropriate. The primary area of possible coordination with the PMAA and UAFCB would be with respect to costs for efficiency measures for which the programs costs are central to the measure costs themselves. This might be the case, for example, for program-related services or programs that include installation services (e.g., direct install programs).

2.4.2 Task 4.2 Define Prioritization Criteria and Process

Given the breadth and scope of deemed measures in the IOU portfolios and their differing overall importance in total portfolio-level expenditures and impacts, it is critical to develop criteria and a process by which financial and analytic resources can be effectively allocated to ensure that incremental cost estimates for the highest priority measures can be developed in an appropriately robust and timely way.

At the highest level, it is clear that the deemed measures designated as High Impact Measures (HIMs) by ED – i.e. those that account for 1% or more expected total portfolio energy or demand savings – are higher priority deemed measures in the context of the MCS update. These deemed HIMs include both DEER HIMs and non-DEER HIMs. For the 2010-2012 program cycle, 43 of the 111 measure groups represented in the IOU portfolios have been designated as HIMs by ED.



Beyond the HIM designation, other criteria that will be considered in the prioritization process include, but are not limited to:

The measure’s expected contributions to total portfolio kWh, kW, and therm savings

The measure’s expected role in next cycle’s portfolio, taking into account emerging/growth markets and interactions with future building codes and appliance standards

The vintage, sample size, and overall quality of the data that supports the current measure cost value in DEER (where applicable)

Longitudinal data and other quantitative evidence that support expectations for technology prices going forward (e.g. changing or stable)

We will also leverage the updated priority rankings developed by the DEER technology groups, the IOU’s comments on those priority rankings, and the findings and lessons learned from the previous MCS studies conducted in California. Additionally, we will explicitly consider the time value of updated ex ante measure cost estimates in the IOU’s next cycle program planning process and the CPUC’s related regulatory processes.

2.4.3 Task 4.3 Define Cost Data Segmentation and Cost Cases

Apart from developing precise definitions for all in-scope deemed measures, the other key activities required to precisely define the analytic scope of the MCS update are to define the dimensions that will be used to segment the measure cost data and define the baseline cost cases that will be used to then calculate incremental measure costs. These definitions are critical to developing appropriate and properly scoped data collection and analysis plans.

Once the cost data segmentation and cost cases are defined by the MCS team, we will work with the DEER team to ensure that all of the associated DEER database fields are fully delineated, defined, and consistent so that all of the corresponding cost data collection and analysis activities for DEER measures can be designed to feed directly into the DEER database and its related requirements.

Cost data segmentation

In previous MCS studies conducted in California, the cost data was segmented across the dimensions described below. These segmentation dimensions will be used as the starting point for defining the segmentation variables that will be used in the 2010-2012 MCS.

Distribution channel refers to the product distribution channel upon which the cost estimate is based. The principal distribution channels for energy efficiency measures are “retail” (for most



residential measures), “contractor” (for many commercial measures), and “wholesale” (for upstream and bulk purchased measures).

Purchase volume refers to whether the cost estimate is associated with high or low volume purchases. The discounts associated with high volume purchases can have a significant impact on final prices for end users and are therefore an important element to consider explicitly. “High volume” is defined as the quantity of items that must be purchased to obtain a substantial discount from list prices (e.g. for fluorescent lamps, this would be on the order of 1,000 or more).

Vintage refers to whether the cost estimate applies to new, retrofit, or replace-on-burnout (ROB) contexts. In general, the new and ROB vintages are associated with incremental costs because a customer’s analysis of efficiency alternatives is typically made when an equipment purchase must be made anyway. The decision in these cases is one of comparing a high-efficiency option with a standard or lower efficiency option (defined as the base case). Conversely, many other efficiency decisions are made on an add-on or early replacement retrofit basis. In these cases, costs for efficient measures may be evaluated on a full cost (add-on) or deferred cost (early replacement) basis.13

Program delivery strategy refers to the general program design used to deliver efficiency measures to final customers. Depending on the delivery strategy used, the incremental costs incurred by the customer, utility, and society at large can differ significantly. For example, the incremental costs associated with direct installation programs include installation labor costs, whereas those associated with downstream rebate program do not. The incremental cost differences related to program delivery strategy, therefore, need to be isolated and understood separately from other factors that influence measure costs. In general, resource acquisition programs use three main program delivery strategies: direct installation, upstream incentives to manufacturers, and downstream rebates to final customers.

Cost cases

As discussed briefly above, establishing the “baseline case” or “cost case” from which the incremental cost of high-efficiency measures are then calculated is a central aspect of MCS studies. The cost cases used in the previous MCS studies in California are listed and described below. These cost cases will be used as a starting point for defining those that will be used in the 2010-2012 MCS.

13 A common example of deferred-cost analysis occurs in early replacement (i.e. retrofit) of commercial lighting

systems in which costs are incurred at the time of retrofit that would otherwise have to be incurred at the end of the remaining useful life of the existing equipment. Add-on examples include measures that are full cost by definition in that they are usually installed on top of existing end-use equipment. Such measures are often control related (variable-speed drives) though not always (window film).



Code base case describes a new equipment installation or an ROB situation where the customer must make an end-use equipment purchase. In this case, the baseline from which incremental costs are calculated is the average price of equipment that meets the minimum efficiency level defined by any applicable building codes (i.e. Title 24) or appliance efficiency standards (Title 20 or federal standards).

Market average base case describes a new equipment installation or an ROB situation where there are no applicable building codes or appliance standards. In this case, the baseline from which incremental costs are calculated is the average price of equipment that is being installed by customers in the current market outside of utility programs. For example, the market average base case for “high performance” or “premium” T-8 linear fluorescent lamps is likely to be “standard” T-8 lamps rather than lower-efficiency T-12 lamps (which are still available in the market), reflecting the fact that the vast majority of lamps installed on the margin are T-8s rather than T-12s.

Customer average base case describes an ROB situation where there are no applicable building codes or appliance standards, and the baseline from which incremental costs are calculated is the average price of replacing the existing end-use equipment with a specified efficiency level below that of the efficient technology. This baseline efficiency level has, in the past, been tied to the minimum efficiency level available in the market; however, recent CPUC decisions and evaluation recommendations indicate the basis for such baselines should be industry standard practice. In practice this will require estimation of the average efficiency of technologies based on a specific share of the market, though market share data may not always be available.

Dual baseline case describes an early replacement situation where existing equipment is replaced before the end of its expected useful life (EUL) with a new, high-efficiency version of the same equipment. In this case, the baseline from which incremental costs are calculated takes the form of a “dual baseline”, where the baseline cost case is defined differently over the remaining useful life (RUL) of the existing equipment compared to the period between the RUL of the existing equipment and the EUL of the measure. This case requires a life-cycle and present-value perspective since the two cost outlays occur at two different points in time.



A complete list of all in-scope deemed measures to be analyzed for the 2010-2012 MCS (draft and final)

Precise definitions for all in-scope deemed measures to be used for pricing and data collection purposes, including but not limited to technology-specific capacity ranges and efficiency levels (draft and final)



Precise definitions of the cost data segmentation and cost cases to be used for all in-scope deemed measures and identification of all the corresponding DEER database fields (draft and final)

A ranking of in-scope deemed measures from highest priority for updating to lowest priority for updating based on a transparent methodology and ranking criteria (draft and final)


2.5 Task 5: Develop Data Collection Strategies

Using the analytic boundaries and information gathered in Tasks 1, 2 and 4, we will then focus on developing detailed data collection strategies and plans for each in-scope deemed measure in Task 5. There are a number of different data collection approaches that will be considered and evaluated for any given in-scope measure. At the highest level, these data collection approaches can be categorized generally into those best conducted using external resources (i.e. specialized subcontractors) and those that can be conducted by internal resources (i.e. Itron and/or KEMA). Below we provide brief descriptions of the various data collection options that will be considered for in-scope deemed measures in the 2010-2012 MCS. Note that while we consider the options below to be a fairly comprehensive list, other viable and appropriate data collections options may present themselves during the course of the study and be integrated into the final data collection plans.

2.5.1 Data Collection Options Using External Resources

Obtain wholesale price lists. As described previously, obtaining wholesale-level prices combined with the application of retail/contractor mark-ups can be a more cost-effective and, in some cases, more accurate method of estimating incremental measure costs, particularly for measures which are sold to end users largely by contractors rather than retailers (e.g. residential and commercial HVAC, residential and commercial windows, and commercial lighting). The detailed data required for any given product typically includes manufacturer name, model number, high-volume wholesale price, low-volume wholesale price, suggested retail price, as well as a host of other attributes unique to each product.

Conduct prototypical costing. This involves developing a hypothetical “market average” application of a set of measures and site conditions and then developing cost estimates based on that prototypical project (or set of projects), rather than attempting to derive meaningful average cost estimates across a variety of actual project cost records. For certain types of custom energy



efficiency measures, site-specific characteristics can drive differences in installed measures costs as much, or more so, than differences in wholesale prices or contractor mark-ups (e.g. high-efficiency chillers, high-efficiency cooling towers, daylighting systems, food service equipment). In these cases, the use of prototypical project costing can be used to remove variations in incremental measures costs stemming from site-specific factors.

Conduct component costing. This approach essentially attempts to build up wholesale equipment prices from the component level and understand differences in wholesale prices for efficient equipment from a materials and labor perspective. This type of component-level perspective can help reduce the uncertainty of projecting incremental cost estimates for emerging technologies for which current retail prices are not readily available in sufficient quantity to support direct estimation of average incremental cost.

2.5.2 Data Collection Options Using Internal Resources

Purchase point-of-sales (POS) retail price data. For certain mass market measures such as CFLs and appliances, it is possible to directly purchase large samples of actual point-of-sale price data from third party marketing firms. Such POS data is a relatively inexpensive and time-efficient way to assemble relatively large samples of retail price points. Indeed, POS data have served as the primary data used in the Residential Market Share Tracking (RMST) studies conducted by Itron annually between 1999 and 2007 and will again serve as the primary data used in the current RMST study (WO023). Indeed, the current RMST study recently purchased POS data for CFLs (in February 2011) and is planning to make another POS data purchase for CFLS early next year (February 2012). As such, those data sets will be immediately available to the MCS study team and will be integrated into our data collection and analysis plans for CFLs (at a minimum). We will also coordinate with the RMST team on any proposed POS data purchases for other in-scope measures (e.g. residential appliances).

Conduct retail shelf surveys. Another data collection option for mass market measures is to conduct field surveys of retail prices at major retailers where surveyors observe and record product prices at a sample of retail businesses. This approach has been used extensively in the previous MCS studies in California as well as in ex post program evaluations. This approach requires developing population and sample frames for the type of retailers in question, developing data collections forms and protocols for the surveyors, and developing quality-control processes for data entry and reporting. This approach can be used as a primary data collection strategy and/or as a strategic supplement to purchasing POS data.

Mine existing data sets. There are several existing data sets that have not been explicitly mined for purposes of collecting measure cost data. These existing data sets include but are not limited to: 1) 2006-2008 ex post program evaluations (particularly for calculated incentive programs) and 2) primary data underlying 2010-2012 IOU ex ante work papers or actual costs and ED’s ex



ante work paper reviews. For in-scope measures where primary data collection is difficult and/or costly, mining existing data sets may prove to be a valuable (albeit, in some cases, second best) alternative, particularly for niche measures such as many of those found in highly-customized industrial efficiency projects.

Conduct surveys of vendors, contractors, and/or program implementers. For certain types of measure cost data (e.g. retail mark-up percentages, volume discount percentages, installation labor hours), conducting telephone surveys of equipment vendors, contractors, builders, and program implementers can be an effective and fairly low-cost data collection approach.

Use online data collection tools and sources. Previous MCS studies in California have leveraged the increasing presence of online retailers and electronic product catalogues to develop online data collection tools and techniques. This approach has proven to be a low-cost alternative for collecting retail prices compared to conducting shelf surveys and purchasing POS data and could be used to either strategically supplement shelf survey results and POS data and/or serve as validation benchmarks.

Use secondary sources. A variety of high-quality secondary sources is readily available and can be leveraged directly for certain types of measure cost data and certain type of in-scope measures. Specifically, the RS Means publications provide location-specific labor rate estimates for specific types of projects that are internally consistent. The US Department of Energy’s MotorMaster+ software and associated database contains extensive cost data for high-efficiency commercial and industrial motors. Similarly, Vaughen’s National Average Price Guide for Electric Motors and Pumps contains extensive cost data as well as associated labor hour estimates for various types of system installations and retrofits.



A comprehensive memorandum presenting the preferred data collection strategies for all in-scope measure groups, including but not limited to explicit identification of data to be collected from subcontractors, data to be collected by Itron and/or KEMA staff, data to be purchased, and data to be collected from secondary sources. The limitations and key analysis issues associated with each preferred data collection strategy will be delineated and previewed (draft and final)

Task orders for all data collection to be conducted by subcontractors, including written scopes of work, budgets, and schedules (draft and final)




2.6 Task 6: Implement Data Collection Strategies

Once measure-specific data collection plans for deemed measures have been developed and approved, the focus of Task 6 will be to implement and manage those measure-specific plans to ensure effective use of time and resources and compliance with appropriate quality control processes.

For data collection activities conducted by specialized subcontractors, Task 6 will include developing and implementing subcontractor-specific oversight and quality control processes. These oversight and quality control processes are likely to be a combination of strategically-designed interim deliverables and regular (perhaps weekly) check-in meetings and progress reports.

For field and telephone-based survey activities, Task 6 will include the development of sample frames and sample designs for each survey activity. Since these survey activities are likely to target retailers, equipment vendors, distributors, and contractors, the most readily available data source for developing sample frames is Dunn & Bradstreet. However, we will also evaluate all other reasonable alternative sources, including but not limited to using IOUs’ Customer Information System (CIS) data to establish a population and sample frame for specific types of equipment retailers. Once the data sources for each sample frame has been established, we will then develop corresponding sample designs to meet the specific research objectives of each survey. These sample designs will include a clearly delineated stratification approach, sample size targets, and expansion approach. In parallel to the development of sample frames and sample designs, we will develop customized survey instruments and/or data collection forms for each survey activity. These survey instruments and data collection forms will be designed to efficiently: 1) verify the firm’s business type, 2) solicit unbiased estimates of the cost data in question, and 3) provide built-in quality control checks to minimize misreporting and otherwise spurious data entry and/or responses. For field surveys, we will also develop a telephone-based recruitment survey that may potentially include offering small financial incentives to firms that initially decline to participate.

All telephone-based survey activities will be implemented by Itron’s CATI survey center, and all telephone survey instruments will be pre-tested to ensure proper question coding (in terms of interviewer scripts and question skip patterns), verify average survey length, and optimize question wording and overall survey flow. Once the fully coded surveys have been pre-tested, Itron’s CATI center supervisor will provide weekly updates on each survey’s disposition and



alert the Itron Project Manager of any sample- or survey-related issues that arise. Once the target sample sizes have been fulfilled, the survey results will be read into a SAS database. For field survey recruitment efforts, Itron’s CATI center will provide daily data dumps of recruited sites to Itron’s field staff coordinator who will then relay recruitment information to field staff as appropriate.



Memorandum describing proposed sample frames and sample designs for each survey activity, including but not limited to identification of the recommended data sources for the sample frame, stratification approach, sample size targets, expected confidence and precision, and expansion approach (draft and final)

Survey instruments and/or data collection forms for each survey activity (draft and final)

Bi-weekly progress reports to ED Project Manager on all data collection activities through completion of data collection activities

Note that each of these deliverables (with the exception of the progress reports) will be produced first in draft form and circulated to the project oversight team for review. The final deliverables will integrate and address comments raised by the project oversight team as deemed appropriate by the ED Project Manager.

2.7 Task 7: Conduct Data Analysis

In this task, all of the raw cost data collected for deemed measures will be cleaned and analyzed in order to ultimately produce incremental cost estimates for all high-priority, in-scope deemed measures. This process will be conducted in four distinct subtasks as described below.

2.7.1 Task 7.1 Data Cleaning

Once all of the raw cost data collected in Task 6 have been compiled and organized into a central database, the next step is to conduct thorough data cleaning processes. These data cleaning processes will be designed with two objectives in mind: 1) to identify and remove erroneous data points associated with spurious data entry and misreporting, and 2) to achieve data completeness as much as possible where gaps exist for a given entry.

With respect to identifying erroneous data points, a variety of approaches will be used. To identify erroneous price data, as an example, summary statistics will be created for each measure that describe the distribution and variance of the price data set in order to quickly identify outlier values. These outlier values will then be targeted for further validation. Another example is



developing and implementing Excel functions or SAS routines that identify erroneous data on equipment characteristics, such as SEER values that fall outside the range of technically possible values (e.g. below 13 or above 30).

With respect to achieving data completeness, an example of the types of approaches that will be used is the development and application of model lookup tables to fill in any missing data on key equipment characteristics such as capacity, efficiency rating, and other product features relevant to retail price analysis. For appliances, such tables are developed and maintained by the CEC and are publically available.14 For HVAC equipment, the Air-Conditioning, Heating, and Refrigeration Institute (AHRI) and the Consortium for Energy Efficiency (CEE) also maintain model lookup tables for Energy Star and CEE-qualified equipment. For commercial lighting, lookup tables are currently being developed under the scope of WO24 (Commercial Saturation Study) and will be available to the Measure Cost Study team.

The MCS team will develop data cleaning protocols and document the data points that were removed from the cost analysis to ensure transparency.

2.7.2 Task 7.2 Cost Modeling

With the cleaned cost database available, we will then conduct the data analysis and cost modeling required to estimates robust, defensible ex ante incremental measure cost estimates for in-scope measures. Depending on the amount and character of the cost data collected for a given measure, a number of different cost modeling approaches could be applied.

In some cases, the simple averages provide the best possible estimate of the costs of the base and high-efficiency technologies. However, when sample sizes are small, using the average price could result in significant bias from outliers. Additionally, differences in average prices can also be biased due to differences in prices between different sources that have nothing to do with differences in technology efficiency. These biases can be addressed by removing outliers from the analysis, using strictly matched pair analysis, or using other forms of weighted averages.

In other cases, however, neither simple averages nor weighted averages can adequately control for important price differences between base case and high-efficiency technologies not related to energy efficiency performance. Where data sets are large, regression models can be developed that predict prices as a function of several key technology attributes, including capacity, efficiency, and a host of technology-specific features that affect price.

For measures where significant samples of wholesale price data are available and collected, wholesale price estimates will first be developed (either through simple averages, weighted

14 See http://www.appliances.energy.ca.gov/.



averages, or regression models), followed by application of a technology-specific retail markup percentage to generate the expected retail or contractor-level final price.

Since the choice of the cost modeling approach has a significant impact on the results, we will first assess the quantity, type, and quality of the cost data collected and recommend, based on our findings, the preferred cost modeling approach that will be applied to each in-scope measure. These recommendations will be provided to the project oversight group for comment and vetting in order to arrive at final decisions for the cost modeling approach to be applied to each in-scope measure.

2.7.3 Task 7.3 Cost Forecasting

For technologies that are expected to experience significant price changes over the short-term (i.e. as identified in Task 4), we will then apply the findings and methods developed in Task 1 to forecast incremental measure costs over the 2013-2014 time period. For any such measures, we will provide both the current-year measure cost estimate as well as the forecasted measure cost estimate in such a way as to clearly distinguish the two values in the final database.

2.7.4 Task 7.4 Uncertainty Analysis and Benchmarking

As the final step in our overall quality control strategy, we will benchmark the cost results for deemed measures, to the extent possible, with the existing values in DEER and other ex ante incremental cost estimates available in other jurisdictions (e.g. NYSERDA, NREL). These benchmarking results will be combined with the regression statistics, where available, to assess uncertainty and precision in the cost modeling results. This analysis will be used in turn to “sanity check” the cost results wherever possible and appropriate and, if warranted, make refinements to the cost modeling assumptions and approaches.

2.7.5 Task 7.5 Database of Recommended Values

Finally, the complete set of recommended ex ante incremental cost values for all high-priority, in-scope deemed measures will be compiled into a single database in a platform and format to be determined by the ED Project Manager in consultation with the DEER team. At a minimum, the measure naming and key database fields will be synchronized with the DEER database and technology tables.



Final, cleaned database of all raw cost data collected for high-priority, in-scope deemed measures (format TBD)



Memorandum presenting comprehensive mapping of recommended cost modeling approaches to all in-scope deemed measure groups (draft and final)

Final database of recommended incremental cost values for all high-priority, in-scope deemed in format TBD by DEER team

Note that the memorandum described above will be produced first in draft form and circulated to the project oversight team for review. The final memorandum will integrate and address comments raised by the project oversight team as deemed appropriate by the ED Project Manager.

2.8 Task 8: Reporting

In Task 8, we will develop a comprehensive project report and corresponding appendices that documents the data collection methods and sources, cost modeling methods, and key assumptions used to generate the final set of recommended ex ante incremental measure cost values. To ensure that the overall content of the report satisfies the CPUC’s needs, we will first develop a detailed report outline and solicit review and comment on that outline from the project oversight team. Similarly, we will circulate draft versions of the report and appendices for review and comment to the project oversight team and integrate and address comments as deemed appropriate by the ED Project Manager in the final project report.

Additionally, this task will include preparation for and participation in a public workshop (date to be determined by ED) to present the data, methods, and results of the study and respond to questions from the public. As with the project report, we will circulate draft versions of presentation materials for review and comment from the project oversight team and integrate and address issues raised as deemed appropriate by the ED Project Manager.



Detailed outline of project report and appendices (draft and final)

Workshop presentation materials (draft and final)

Complete project report and appendices (draft and final)




2.9 Task 9: Integration of Final Ex Ante Measure Costs into DEER

In parallel to the preparation of the final project report, we will oversee integration of the final recommended ex ante incremental measure cost values for all in-scope deemed measures into the DEER database and related user-interface software. The focus of this task will be on systematically verifying that the incremental cost values and associated database fields properly reflect the results delivered in Task 7 and are properly integrated with corresponding energy savings, peak savings, EULs, and NTG ratios.



Memorandum describing the approach used to verify successful integration of the cost results for all in-scope deemed measures into the DEER database and associated software (draft and final)

Note that this deliverable will be produced first in draft form and circulated to the project oversight team for review. The final deliverable will integrate and address comments raised by the project oversight team as deemed appropriate by the ED Project Manager.

2.10 Task 10: Project Management

The project management task will include Itron’s day-to-day management of internal resources and subcontractors (including progress reporting, invoicing, and work assignments) as well as regular conference calls and in-person meetings with the ED Project Manager and the project oversight team (frequency TBD).



Monthly progress reports will accompany invoices sent to CPUC

Weekly reports of accomplishments (by subtask), planned activities for the following week, and issues needing resolution will be posted using the CPUC’s Project Status Reporting (PSR) system each week

Regular project coordination conference calls with subcontractors (frequency TBD)

Regular Itron/KEMA/ED project status conference calls (frequency TBD)

Itron, Inc. 3-1 Project Schedule

3 Project Schedule and Stakeholder Engagement

This section provides an overview of the project schedule and a brief discussion of the expected level of stakeholder engagement.

3.1 Project Schedule and Milestones

Table 3-1 provides an overview of the project schedule and approximate dates of key deliverables from each project task.

Table 3-1: Project Timeline by Task

Activity Start Date End Date

Task 1. Needs Assessment and Related Research December 2011 February 2012

Task 2. Identify Qualified Subcontractors September 2011 December 2011

Task 3. Custom and New Construction Cost Research December 2011 December 2012

Task 4. Define Measure List and Analysis Scope December 2011 February 2012

Task 5. Develop Data Collection Strategies December 2011 February 2012

Task 6. Implement Data Collection Strategies January 2012 December 2012

Task 7. Conduct Data Analysis January 2012 July 2013

Task 8. Reporting July 2012 January 2013

Task 9. Integration of Final Ex Ante Measure Costs into DEER January 2013 July 2013

Task 10. Project Management June 2011 July 2013

Critical path items which might impact this schedule are the:

Finalization of the in-scope measure list and measure definitions

Finalization of the cost data segmentation, cost cases, and corresponding DEER database fields and measure naming conventions

These critical path items may directly impact the development and implementation of measure-specific data collection plans. That said, all efforts will be taken to develop data collection plans that can account for and easily adapt to minor changes to the in-scope measure list and measure definitions.


Itron, Inc. 3-2 Project Schedule

It is important to note that it is expected that some of the activities in Tasks 2, 3, 4, and 5 will occur in parallel, since it is likely that the data collection strategies for some measures will be readily apparent and be approved for implementation early in the process (e.g. for CFLs) whereas the data collection strategies for other measures will undoubtedly require more time to develop, approve, and be cleared for implementation. Similarly, it is expected that some of the activities in Tasks 5 and 6 will occur in parallel for the same reasons discussed above.

3.2 Stakeholder Engagement and Early Feedback

The IOUs will be provided with the opportunity to review and provide comments on this research plan. The timeline for this research plan has incorporated the delivery of interim findings to provide early feedback to the IOUs. The specific forms of early feedback that will be provided to the IOUs are shown below in Table 3-2.

Table 3-2: Schedule of Early Feedback Deliverables

Early Feedback Product January

2012 July 2012

January 2013

July 2013

Memo summarizing adopted cost data collection approaches by measure group and data type X

Memo summarizing cost data availability and quality to date by measure group and data type X

Memo summarizing proposed cost modeling approaches by measure group and data type

X

Memo summarizing adopted cost modeling approaches by measure group and data type

X

Draft IMC estimates for selected priority DEER measures X

Draft IMC estimates for all priority DEER measures X

Draft IMC estimates for selected non-DEER measures X

Draft IMC estimates for all priority non-DEER measures X

Note that it is also planned to have ongoing meetings and discussions with the IOUs to solicit feedback on approaches and the need for interim results to support the IOUs’ planning needs. Through this process, the MCS study team will attempt to meet the needs of the IOUs with respect to providing timely and useful interim results.

Itron, Inc. 4-1 Task Budgets

4 Task Budgets

Table 4-1 provides an overview of the initial budget allocation by task. Note that these initial budget allocations are likely to be revised as the project moves forward and all in-scope data collection strategies and costing methodologies are approved and finalized.

Table 4-1: Initial Budget Allocation by Task

Task number/title Initial Budget Allocation

Task 1. Needs Assessment and Related Research $55,000

Task 2. Identify Qualified Subcontractors $57,000

Task 3. Custom and New Construction Cost Research $450,000

Task 4. Define Measure List and Analysis Scope $50,000

Task 5. Develop Data Collection Strategies $250,000

Task 6. Implement Data Collection Strategies $900,000

Task 7. Conduct Data Analysis $200,000

Task 8. Reporting $100,000

Task 9. Integration of Final Ex Ante Measure Costs into DEER $50,000

Task 10. Project Management $150,000

Contingency (9.5%) $238,000

Total $2,500,000

It should also be noted that we do not intend to seek full authorization of the planned project budget up front. Rather, we intend to seek staged budget authorizations for both Itron and its subcontractors in order to minimize the possibility of cost-overruns and to allow ED to re-allocate resources to other EM&V activities should some of the activities proposed for this study prove to be less costly, lower priority, and/or out of scope.

Table 4-2 breaks down the total task budgets presented above into three separate budget authorizations and delineates the anticipated relative contributions of Itron (and its subcontractors) and KEMA. Table 4-2 also maps the tasks described in this research plan to the



associated task numbers in Itron and KEMA’s respective Work Orders. In this table, the Phase I, II, and III stages represent the following:

Phase I represents the task funding already authorized in Itron and KEMA’s initial Work Orders.

Phase II represents the next budget authorization that Itron will seek, which will authorize the full budget allocations for Tasks 1, 4, and 5 and initial portions of the budget allocations for Tasks 3, 6, and 7. Importantly, Phase II includes adjustments to the task funding established in the original Itron and KEMA Work Orders based on actual spending to date and anticipated effort going forward.

Phase III represents the final budget authorization that Itron will seek, which will authorize the remaining portions of the budgets for Tasks 3, 6, and 7 and the full budgets for Tasks 8 and 9.

Note that the staged budget authorizations shown in Table 4-2 are subject to revision and refinement as deemed necessary and appropriate by ED and Itron through the course of the study.


Table 4-2: Anticipated Staged Budget Authorizations

Work Order Tasks Research Plan Task

Phase I Phase II Phase III

Total

Itron Original WO

KEMA Original WO

Itron Additions/Reductions

KEMA Additions/Reductions

Itron Additions

Task 0. Work Order Development Task 10 $20,000 $20,000

Task 1. Work Order Management Task 10 $25,000 $50,000 $55,000 $130,000

Task 2. Research Plan and Project Schedule Development

2.1 Requests for Qualifications Task 2 $30,000 $15,000 $20,000 ‐$8,000 $57,000

2.2 Research Plan Development Task 5 $70,000 $30,000 ‐$20,000 ‐$30,000 $50,000

2.3 Needs Assessment and Related Research Task 1 $55,000 $55,000

Task 3. Research Plan Implementation

3.1 Early Data Collection Task 3 $105,000 $80,000 ‐$70,000 ‐$80,000 $35,000

3.2 Coordinate Data Collection Task 5 $125,000 ‐$70,000 $55,000

3.3 Custom and NC Research Task 3 $120,000 $295,000 $415,000

3.4 Define Measure List Task 4 $50,000 $50,000

3.5 Develop Data Collection Strategies Task 5 $125,000 $20,000 $145,000

3.6 Implement Data Collection Strategies Task 6 $200,000 $80,000 $700,000 $980,000

3.7 Conduct Data Analysis Task 7 $50,000 $18,000 $150,000 $218,000

3.8 Reporting Task 8 $100,000 $100,000

3.9 Integration into DEER Task 9 $50,000 $50,000

Total $375,000 $125,000 $535,000 $0 $1,350,000 $2,262,000

Contingency (9.5%) $238,000

ex ante measure cost study research plan - … · related to particular types of measures that have...

Documents