seminar 55 simulation calibration
TRANSCRIPT
Seminar 55
Simulation Calibration
Autotune Calibration
Joshua New, Ph.D. Oak Ridge National Laboratory
[email protected] 865-241-8783
Learning Objectives • Describe how ASHRAE Guideline 14 defines calibration criteria for
energy simulation
• Describe how high performance computing (HPC) resources can be used to efficiently distribute simulation runs across multiple servers
• Describe how machine learning algorithms can be used to support the development of efficient calibration techniques
• Describe the disadvantages of each of the three calibration techniques presented
• Describe the advantages of each of the three calibration techniques presented
• Describe realistic scenarios for model calibration that can be utilized by practitioners today
ASHRAE is a Registered Provider with The American Institute of Architects Continuing Education Systems. Credit earned on completion of this program will be reported to ASHRAE Records for AIA members. Certificates of Completion for non-AIA
members are available on request.
This program is registered with the AIA/ASHRAE for continuing professional education. As such, it does not include content that may be deemed or construed to be an approval or endorsement by the AIA of any material of construction or any method or manner of handling, using, distributing, or dealing in any material or product. Questions related to specific
materials, methods, and services will be addressed at the conclusion of this presentation.
Acknowledgements
• Amir Roth – DOE’s BTO
• Aaron Garrett – JSU
• Jibonananda Sanyal – ORNL
• Richard Edwards – UT
• Mahabir Bhandari – ORNL
• Som Shrestha – ORNL
• Buzz Karpay – Karpay Associates
• XSEDE
• OLCF
Outline/Agenda
• Motivation
• What is Autotune?
– Calibration as search
• How does it work?
– Methods for speeding up the search
• How good is it?
– Calibration process and accuracy
• How can I use it?
– Deployment as web service
5
ASHRAE G14
Requires
Using Monthly utility data
CV(RMSE) 15%
NMBE 5%
Using Hourly utility data
CV(RMSE) 30%
NMBE 10%
3,000+ building survey, 23-97% monthly error
Motivation
.
.
.
E+ Input
Model
6
The Autotune Idea Automatic calibration of software to data
Problem/Opportunity:
~3000 parameters per input file
2 minutes per simulation = 83 hours
7
Calibration as Search
• EnergyPlus is a desktop app
• Writes files during a simulation
• Use RAMdisk
• Balance simulation memory
vs. result storage
• Validate simulation output
• Bulk write data to disk
• Design of Experiments for
Uncertainty Quantification
• In-Situ data analysis
• Scalable Architecture for Big
Data Mining
• 270TB of simulation data
8
Supercomputers for Buildings CPU
Cores Wall-clock
Time (mm:ss) Data Size
EnergyPlus Simulations
16 18:14 5 GB 64
32 18:19 11 GB 128
64 18:34 22 GB 256
128 18:22 44 GB 512
256 20:30 88 GB 1,024
512 20:43 176 GB 2,048
1,024 21:03 351 GB 4,096
2,048 21:11 703 GB 8,192
4,096 20:00 1.4 TB 16,384
8,192 26:14 2.8 TB 32,768
16,384 26:11 5.6 TB 65,536
32,768 31:29 11.5 TB 131,072
65,536 44:52 23 TB 262,144
131,072 68:08 45 TB 524,288
• Linear Regression
• Non-Linear Regression
• Feedforward Neural
Network
• Support Vector Machine
Regression
• K-Means with Local
Models
• Gaussian Mixture Model
with Local Models
• Self-Organizing Map with
Local Models
• Regression Tree (using
Information Gain)
• Time Modeling with Local
Models
• Recurrent Neural Networks
• Genetic Algorithms
• Ensemble Learning
(combinations of
multiple algorithms)
9
Suite of Machine Learning
Integrated mixture of
Commercial, Research, and
Open Source software
Data Preparation
30x LS-SVMs
validation folds 1-10
input orders 1-3
MLSuite Architecture
MLSuite XML
PBS
Linux #1
Super-computer
#1
Linux #218
Super-computer
#2 …
• EnergyPlus – 2-10 mins for an annual simulation
!- ALL OBJECTS IN CLASS
Version,
7.0; !- Version
!- SIMULATIONCONTROL ===
SimulationControl,
No, !-Do Zone Sizing Calc
No, !-Do System Sizing Calc
…
11
MLSuite Example
• ~E+ - 4 seconds AI agent as surrogate model, 90x speedup, <5% error; “brittle” <156 input changes
• EnergyPlus is slow
– Full-year schedule
– 2 minutes per simulation
• Use abbreviated 4-day schedule instead
– Jan 1, Apr 1, Aug 1, Nov 1
– 10 – 20 seconds per simulation
Monthly Electrical Usage
r = 0.94
Hourly Electrical Usage
r = 0.96
12
Getting more for less
Thickness Conductivity Density Specific Heat
Bldg1 0.022 0.031 29.2 1647.3
Bldg2 0.027 0.025 34.3 1402.5
(1+2)1 0.0229 0.029 34.13 1494.7
(1+2)2 0.0262 0.024 26.72 1502.9
• Average each component
• Add Gaussian noise
• … “AI inside of AI”
How are offspring produced?
13
Evolutionary Computation
Island Hopping
14
4 of 19 experiments 1. Surrogate Modeling 2. Sensor-based Energy
Modeling (sBEM) 3. Abbreviated Schedule 4. Island-model evolution
Evolutionary Process
Leveraging HPC resources to calibrate models on commodity for optimized building efficiency decisions
Industry and building owners DOE-EERE BTO XSEDE and DOE Office of Science
Features:
Works with “any” software
Tunes 100s of variables
Customizable distributions
Matches 1+ million points
Uses commodity hardware
ASHRAE
G14 Requires
Autotune Results
Monthly utility data
CVR 15% 0.32%
NMBE 5% 0.06%
Hourly utility data
CVR 30% 0.48%
NMBE 10% 0.07%
Commercial Buildings
Within
30¢/day (actual use
$4.97/day)
Residential
home
15
Final Calibration Accuracy
Working Internal Website
60+ fields (optional)
Determine Inputs to Calibrate Restaurant Hospital Large Hotel Large Office
Medium Office
Midrise Apartment
Primary School
Quick Service
#Inputs 49 227 110 85 81 155 166 54
#Groups 49 146 71 45 38 82 113 54
Secondary School
Small Hotel Small Office Stand-alone
Retail Strip Mall
Super Market
Warehouse TOTAL
#Inputs 231 282 72 59 113 78 47 1809
#Groups 128 136 61 56 89 73 45 1143
Provide Data
Calibrated Results Metric Value
Input error average 24.38
Input error maximum 66.12
Input error minimum 0.09
Input error variance 228.53
CV(RMSE)
CH4:Facility [kg](Monthly) 9.95
CO2:Facility [kg](Monthly) 15.42
CO:Facility [kg](Monthly) 20.40 Carbon Equivalent:Facility [kg](Monthly) 14.42
Cooling:Electricity [J](Hourly) 1577.96
Electricity:Facility [J](Hourly) 10.48
…
NMBE
CH4:Facility [kg](Monthly) -9.57
CO2:Facility [kg](Monthly) -14.78
CO:Facility [kg](Monthly) -19.52 Carbon Equivalent:Facility [kg](Monthly) -13.83
Cooling:Electricity [J](Hourly) 592.77
Electricity:Facility [J](Hourly) -9.52
Electricity:Facility [J](Monthly) -9.52
143+ outputs
IDF + CSV = XML
Performance and Availability
ASHRAE
G14 Requires
Autotune Results
Monthly utility data
CVR 15% 0.32%
NMBE 5% 0.06%
Hourly utility data
CVR 30% 0.48%
NMBE 10% 0.07%
Results from 24 Autotune calibrations (3 building types - 8, 34, 79 tuned inputs each)
ASHRAE
G14 Requires
Autotune Results
Monthly utility data
CVR 15% 1.20%
NMBE 5% 0.35%
Hourly utility data
CVR 30% 3.65%
NMBE 10% 0.35%
Results from 20,000+ Autotune calibrations (15 types – 47-282 tuned inputs each)
FY15 project to begin integration of Autotune web service as OpenStudio application
Free to use. Pay for cloud computing.
Bibliography • Garrett, Aaron and New, Joshua R. (2014). "A Scientific Study of Automated Calibration applied to
DOE Commercial Reference Buildings." ORNL internal report ORNL/TM-2014/709, December 31,
2014, 114 pages
• Ostrouchov, George, New, Joshua R., Sanyal, Jibonananda, and Patel, Pragnesh (2014).
"Uncertainty Analysis of a Heavily Instrumented Building at Different Scales of Simulation." In
Proceedings of the 3rd International High Performance Buildings Conference, Purdue, West
Lafayette, IN, July 14-17, 2014.
• Sanyal, Jibonananda, New, Joshua R., Edwards, Richard E., and Parker, Lynne E. (2014).
"Calibrating Building Energy Models Using Supercomputer Trained Machine Learning Agents." In
Journal on Concurrency and Computation: Practice and Experience, March, 2014.
• Garret, Aaron and New, Joshua R. (2013). "Trinity Test: Effectiveness of Automatic Tuning for
Commercial Building Models." ORNL internal report ORNL/TM-2013/130, March 7, 2013, 24
pages.
• Edwards, Richard E., New, Joshua R., and Parker, Lynne E. (2012). "Predicting Future Hourly
Residential Electrical Consumption: A Machine Learning Case Study." In Journal of Energy and
Buildings, volume 49, issue 0, pp. 591-603, June 2012.
• Bhandari, Mahabir S., Shrestha, Som S., and New, Joshua R. (2012). "Evaluation of Weather
Data for Building Energy Simulations." In Journal of Energy and Buildings, volume 49, issue 0, pp.
109-118, June 2012.
• Garrett, Aaron and New, Joshua R. (2012). "An Evolutionary Approach to Parameter Tuning of
Building Models (Experiments 1-17)." ORNL internal report ORNL/TM-2012/418, April 2012, 68
pages.
