washington state low income weatherization program evaluation calendar year 2011 draft results...

Washington State Low Income Weatherization Program Evaluation Calendar Year 2011DRAFT Results

Prepared by:Rick Kunkle

July 2013

Purpose of Evaluation

Identify and document Weatherization Program outcomes, benefits and costs to:

• Assure prudent use of funds (accountability)

• Improve the quality and effectiveness of program services

• Assess progress toward Weatherization Program outcomes as measured by key performance measures

Weatherization Program Overview

Serves low income families by installing energy efficiency measures and making health and safety improvements and necessary repairs

• Services provided in Washington since 1977

• Managed by the Housing Improvement and Preservation Unit in the Department of Commerce

• Commerce contracts with 28 local agencies to deliver services

• Funded by a mix of state and local sources

• The American Recovery and Reinvestment Act (ARRA) significantly increased funding, primarily in 2010 and 2011

Previous Evaluation Findings

• Oak Ridge National Laboratory completed an impact evaluation for Washington in 2001 showing energy savings compared favorably with other states

• WSU Energy Program completed Washington evaluations for 2006 and FY2010

• Benefits exceeded costs

• Production increased significantly in FY2010 from ARRA funding

• Recommendations for improving data collection, estimates of benefits, and cost tracking and allocation

• Commerce has made investments and progress in addressing evaluation recommendations

Evaluation Approach

The evaluation covers calendar year 2011

• Project Data: from the Weatherization information Data System (WIDS) for projects with final inspections in 2011

• Program Data: from Commerce including expenditures, agency work plans, and historical data

Bottom Line Results for 2011

• Expenditures were $48 million. This is more than twice 2005-2008 annual expenditures. Over half was from ARRA

• Production was 7,451 units. This is a little less than the peak in 2010, but still more than twice pre-ARRA production. Two-thirds of production was multi-family units

• Over 12,000 people lived in housing that was weatherized

• On average, 9 improvement measures were installed in each weatherized unit

• Estimated energy savings is $1.17 million/year. On average this is $157/unit. Other benefits are estimated to be $198/unit

• The Program benefit-cost ratio is estimated to be 1.12, meaning that benefits slightly exceed costs

What is in this Presentation

• Program Delivery

• Program Benefits

• Program Costs

• Program Cost-Effectiveness

• Summary and Recommendations

Key Findings – Program Delivery

Historical Production Trends

Monthly Production Trends

Production by Agency in 2011

Agency Service Delivery

Type of Agency

Delivery Method

# of agencies % of FY10 production

Funding Sources

Number of Funding Sources

Project Delivery Time

Average Days to Completion by Agency

Heating Fuel

Weatherization Measures by Unit

Weatherization Measure Categories

Top Weatherization Measures

Top Weatherization Measures by Building Type

Tier 2 Weatherization Measures

Household Demographics

Household Poverty Level

Accomplishments, Challenges, Recommendations

Each year, local agencies produce a work plan that summarizes their processes, procedures, accomplishments, and challenges

• Accomplishments:

• Energy savings, comfort, health and safety and housing repair benefits of their weatherization work

• Partnerships, cooperation and referrals

• Challenges

• Decreasing funding along with increasing costs from wage and other requirements (reporting, monitoring, training, etc.)

• Recommendations

• Clearer communication and more transparent decision-making about funding availability and allocations

• Streamlining program requirements

Quality Assurance Inspections Specifications Correction Factor

Key Findings – Program Benefits

Energy Savings Analysis in WIDS

Deemed savings estimates:

• Developed savings coefficients by measure, building type, fuel source, and heating zone

• Used savings coefficients with measure data in WIDs in savings calculations for each measure

• Accounted for interactions between measures that save heating energy

• Energy savings are calculated for each weatherization project with caps and checks

• Energy cost savings based on average Washington energy costs by fuel source

• Energy benefits calculations use weighted average measure life – 34 years

Total Energy SavingsFuel Source Building Type

Total Energy Savings by MeasureCategory Measure

Energy Savings per Unit by Agency

Energy Cost SavingsFuel Source Building Type

Energy Cost Savings per Unit

Other Benefits CalculationsThis is the same approach used for the FY2010 Evaluation

• Utility benefits > estimated from best practices review

• Reduce delinquent bills and bad debt write-off

• Service shut offs and reconnects

• Participant benefits > estimated from best practices review

• Water/wastewater savings

• Increased property value

• Fewer moves

• Health, safety and comfort benefits

• Societal benefits

• Reduce greenhouse gas and pollutant emissions > directly estimated

• Improvements to the economy > estimated from evaluation of Weatherization Program for Pacific Power

Program Benefits

$/Household/Yr Mid Percent of Total

Utility 16 4%

Participant 103 29%

Societal - Economic 60 17%

Societal - Emissions 19 5%

Other Total 198 56%

Energy 157 44%

Total 355 100%

Key Findings – Program Costs

Program Expenditures by Fund Source

Program Expenditures by Category

Direct Project Cost by Fund Source

Direct Project Cost by Measure Category

Direct Average Unit Cost by Agency

Distribution of Direct Unit Cost

Cumulative Direct Unit Cost Curve

High Cost Projects

Average Direct Unit Cost

Total Direct Cost Expenditures

Total Unit Cost Estimate

Why are per unit costs higher in 2011 compared to FY2010?

Many of the reasons could be associated with ARRA

• More comprehensive weatherization

• Wage requirements

• Reporting requirements

• Quality assurance requirements

• Others?

Key Findings – Program Cost Effectiveness

Per Unit Cost Divided by Energy Savings

Simple Energy Payback

Cumulative Direct Unit Cost and Energy Savings

Cost-Effectiveness Analysis Approach

• Cost-Effectiveness = total benefits per unit/total costs per unit

• Program benefits occur over time and are converted to a present value

• 30 year average measure life

• 2.3% discount factor (OMB)

• 1% fuel escalation factor

• A high, mid and low scenario are considered to account for uncertainty in the benefits estimates

• Alternate approaches with different costs and benefits are also considered

Cost-Effectiveness Analysis Results($/unit costs)

Present Value Mid Low High

Emissions Benefit $421 $369 -

Economic Benefit $1,313 $685 $1,967

Utility Benefit $338 $76 $674

Participant Benefit $2,233 $920 $4,588

Total Non-Energy $4,305 $2,050 $7,229

Energy Benefit $3,934 $2.992 $4,571

Total Benefit $8,239 $5,042 $11,800

Total Cost $7,362 $7,362 $7,362

Benefit-Cost Ratio 1.12 0.68 1.60

Cost-Effectiveness Alternate Approaches

($/unit costs)Present Value Energy Only Plus

No Admin or T&TA $

Direct Cost Only

Plus No H&S and WRR

Emissions Benefit 0 0 0 0

Economic Benefit 0 0 0 0

Utility Benefit 0 0 0 0

Participant Benefit 0 0 0 0

Total Non-Energy 0 0 0 0

Energy Benefit $3,934 $3,934 $3,934 $3,934

Total Benefit $3,934 $3,934 $3,934 $3,934

Total Cost $7,362 $6,414 $4,780 $3,724

Benefit-Cost Ratio 0.53 0.61 0.82 1.06

Summary and Recommendations

Some Interesting Results for 2011

• Higher production during ARRA is due to large multi-family units

• Top 9 producers account for 80% of production

• City and county government agencies account for over half the production, but less than half the agencies

• Compared to FY2010 there is evidence of more comprehensive weatherization

• Ceiling, wall, and floor insulation and air sealing account for 55% of the estimated energy savings

• Large multi-family units account for 60% of the units, but only a quarter of the energy savings and a quarter of total direct costs

• The 10% of highest cost units account for 32% of total direct costs and 27% of total energy savings

Conclusions to Think About

• The total program unit cost estimate increased in 2011 compared to FY2010

• Large multi-family units contribute to higher production, more people served, and lower costs, but they also have fewer measures installed and produce less energy savings

• High cost units contribute significantly to total Program costs, but they also have more measures installed and greater energy benefits

• Energy cost benefits are less than other benefits

• While high production agencies had lower unit costs, they did not have lower unit costs/energy savings

Recommendations• We need to develop better ways to track

Program expenditures for Commerce

• The Program unit cost estimate includes an estimate of local Program operations costs that did not go directly to unit weatherization. We need to check the accuracy of this estimate and identify what these cost are for and how to track them

• Identify ways to reduce the trend towards higher costs, which reduces Program cost-effectiveness if benefits do not increase

• How do we measure success? Identify key Program metrics and use WIDS data or other sources to document Program success

• Consider looking at the benefits from high cost units

Next Steps

• Review and discuss the evaluation results

• Make needed revisions and corrections to the evaluation

• Complete an executive summary for the evaluation (text document)

• Share the evaluation with the weatherization network and stakeholders

• Apply the experience from the 2011 evaluation to plan for and conduct the 2012 evaluation