integrating processimpactevals

Main Headquarters: 120 Water Street, Suite 350, North Andover, MA 01845 With offices in: NY, ME, TX, CA, OR www.ers-inc.com

MERGE AHEAD: Integrating Process and Impact Evaluations

2014 ACEEE Summer Study Patrick Hewlett, P.E. ERS August 21, 2014

Example: Consider a small-business direct install program that underwent concurrent but separate process and impact evaluations. Process finding: “The total potential energy savings associated with franchises is significantly greater than with individually owned businesses. To better achieve program savings goals, the utilities should engage franchise accounts with multiple eligible facility locations.”

Impact finding: “Franchises were found to have higher free-ridership than non-franchises and should therefore be carefully screened during the incentive application process.”

WHY INTEGRATE?

8/26/2014 2

Status quo and barriers to integration An integrated approach for three phases:

Evaluation planning Data collection Reporting

Joint recommendations: Quantified Harmonized

A blueprint for PAs

OVERVIEW

8/26/2014 3

STATUS QUO

8/26/2014 4

Traditional Process approach: • Big-picture assessment of program effectiveness • Delivery, marketing, outreach, incentive structures,

customer satisfaction

Traditional impact approach:

• Focused on producing one number: net savings • Detailed file review, site visits with M&V,

engineering analysis

Efficiency program improvement through recommendations

Timing Process evaluations typically occur before impact Actionable feedback for PAs, can be implemented more quickly

Budget Two evaluations might not be funded by same source Levels of funding are often different Additional budget required to allow for strategic planning and joint

recommendation development

Objective Traditional process and impact evaluations not thought to intersect However, recommendations benefit from an intersection

BARRIERS TO INTEGRATION

8/26/2014 5

Process recommendations not informed by comprehensive file review and field findings

Impact recommendations often not grounded in

program costs, logistics

CONSEQUENCES

8/26/2014 6

Result: unrelated, isolated recommendations Program hesitant to act on recommendations that

might be out-of-touch or idealistic.

AN INTEGRATED APPROACH

8/26/2014 7

Traditional Process

Approach

Program benchmarking

Interviews with program staff

Define research objectives

Application review

Interviews with customers, contractors

Compliance: age, EUL, cost, payback

Tracking system review

Process findings

Estimation best practices

Implementation

Customer/contractor experience

Process recommendations

Integrated Approach

Benchmarking to define program needs

Set joint research objectives

Plan for joint data collection

Collaborative project file review

Initial application/ algorithm findings

Joint survey questions

Field data collection on compliance factors

Discrepancy analysis driving prioritized, joint recommendations based on savings, cost

Traditional Impact

Approach

Areas of uncertainty

Define research objectives

Review program technologies

Project file review

Algorithm review

On-site data collection and M&V

Interviews with customers, contractors, nonparticipants

Savings results

Realization rates

Net-to-gross

Impact discrepancies

Impact recommendations

• Program benchmarking • Interviews with program staff • Define research objectives

• Benchmarking to define program needs • Set joint research objectives • Plan for joint data collection

EVALUATION PLANNING

8/26/2014 8

Traditional Process

Approach Integrated Approach

Traditional Impact

Approach

• Areas of uncertainty • Define research objectives • Review program technologies

• Application review • Interviews with customers, contractors • Compliance: age, EUL, cost, payback • Tracking system review

• Joint survey questions • Collaborative project file review • Initial application/ algorithm findings • Field data collection on compliance factors

DATA COLLECTION

8/26/2014 9

Traditional Process

Approach Integrated Approach

Traditional Impact

Approach

• Project file review • Algorithm review • On-site data collection and M&V • Interviews with customers, contractors, nonparticipants

• Estimation best practices • Path to implementation • Customer, contractor experience • Process recs

• Discrepancy analysis driving prioritized, joint, actionable recs based on savings, cost

REPORTING

8/26/2014 10

Traditional Process

Approach Integrated Approach

Traditional Impact

Approach

• Savings results • Realization rates • Net-to-gross ratios • Impact discrepancies • Impact recs

A key benefit of an integrated approach is quantified and harmonized recommendations.

Quantified recommendations Allows program to prioritize recommendations by savings

and/or costs Impact team can quantify the program’s shortcomings by

savings magnitude, highlighting areas of greatest need Process team research informs the costs and barriers of

implementation and proposes solutions Result: program administrators able to weigh

recommendations, prioritize by need, and more immediately act.

AN INTEGRATED APPROACH

8/26/2014 11

QUANTIFIED RECOMMENDATIONS

# ProjectsImpact on kWh RR

Impact on kWh RR

# Projects

-8%2%

-8%0%

-6%1%

0%0%

0%0%

-1%0%

-1%0%

-2%1%

-6%8%

-2%3%

-6%0%

-2%0%

102 -41% 15% 41

Unsubstantiated savings 5 0

Totals

4 2

Difference in controls 9 0

Estimating Performance

Incorrect load profile 6 2

Difference in operating hours 17 21

Deemed savings assumed

7 2

Incorrect technology 3 0

2 1

Inspection

Inoperable equipment 1 0

Quantity or size error

Measure Screening

Ineligible measure 13 0

Incorrect baseline 8 2

Interactivity

Category

Negative Positive

Administrative Administrative error 27 11

8/26/2014 12

QUANTIFIED RECOMMENDATIONS

# ProjectsImpact on kWh RR

Impact on kWh RR

# Projects

-8%2%

-8%0%

-6%1%

0%0%

0%0%

-1%0%

-1%0%

-2%1%

-6%8%

-2%3%

-6%0%

-2%0%

102 -41% 15% 41

Unsubstantiated savings 5 0

Totals

4 2

Difference in controls 9 0

Estimating Performance

Incorrect load profile 6 2

Difference in operating hours 17 21

Deemed savings assumed

7 2

Incorrect technology 3 0

2 1

Inspection

Inoperable equipment 1 0

Quantity or size error

Measure Screening

Ineligible measure 13 0

Incorrect baseline 8 2

Interactivity

Category

Negative Positive

Administrative Administrative error 27 11

8/26/2014 13

Savings tracking and measure screening issues led to a 19% drop in kWh RR

QUANTIFIED RECOMMENDATIONS

# ProjectsImpact on kWh RR

Impact on kWh RR

# Projects

-8%2%

-8%0%

-6%1%

0%0%

0%0%

-1%0%

-1%0%

-2%1%

-6%8%

-2%3%

-6%0%

-2%0%

102 -41% 15% 41

Unsubstantiated savings 5 0

Totals

4 2

Difference in controls 9 0

Estimating Performance

Incorrect load profile 6 2

Difference in operating hours 17 21

Deemed savings assumed

7 2

Incorrect technology 3 0

2 1

Inspection

Inoperable equipment 1 0

Quantity or size error

Measure Screening

Ineligible measure 13 0

Incorrect baseline 8 2

Interactivity

Category

Negative Positive

Administrative Administrative error 27 11

8/26/2014 14

Operating hours assumptions and deemed savings had relatively high swings, but is action needed?

Process finding: “Database is fully populated and includes appropriate fields for tracking applications.” Impact recommendation: “Eliminate all duplicate records in the tracking dataset and always update tracking system with the most current estimates of savings.” Integrated recommendation: “Database design conforms to best practices. Apparently, QC processes are inadequate because tracking errors degraded the realization rate by 6%. Convene a team of database administrators, program implementers, and the contractor to map out the tracking database and identify specific checkpoints in the system for reducing these errors.”

HARMONIZED RECOMMENDATIONS

8/26/2014 15

HARMONIZED RECOMMENDATIONS

8/26/2014 16

Process finding: “Customers prove eligibility through extensive screening of project applications, which drives down participation.” Impact recommendation: “Ineligible customers have been approved due to inaccurate account information as determined through field data collection. Each project should require an on-site pre-inspection to ensure eligibility.” Integrated recommendation: “Develop a benchmarking system to screen customer-claimed usage by square footage and facility type. Require field inspection for those customers that do not meet the benchmark, which is expected to occur about 15% of the time.”

An integrated evaluation approach requires strategic planning for joint objectives. Assignment of roles, responsibilities Agreement on method of valuing recommendations Joint interviews with program staff Coordination of delivery schedules and check-ins

Regular communication among teams Leading to joint recommendations

Process research informs feasibility of implementation Impact research highlights areas of greatest need

Result: actionable, grounded recommendations

prioritized by need.

A BLUEPRINT FOR PAS

8/26/2014 17

CONTACTS Patrick Hewlett, P.E.

ERS

phewlett@ers-inc.com

(978) 478-5305

Sue Haselhorst

ERS

shaselhorst@ers-inc.com

(978) 478-5325

8/26/2014 18