Issues Related to Parameter Estimation in
Model Accuracy Assessment
DDDAS: June 6-7, 20131
Tom Henderson&
Narong Boonsirisumpun
ICCS 2013Barcelona, Spain
6 June 2013
Funded in part by grant AFOSR-FA9550-12-1-0291.
DDDAS: June 6-7, 2013 2
Major Objectives:
1. Develop Bayesian Computational Sensor Networks– Detect & identify structural damage– Quantify physical phenomena and sensors– Characterize uncertainty in calculated
quantities of interest (real and Boolean)
DDDAS: June 6-7, 2013 3
Major Objectives (cont’d):
2. Develop active feedback methodology using model-based sampling regimes
– Embedded and active sensor placement– On-line sensor model validation– On-demand sensor complentarity
DDDAS: June 6-7, 2013 4
Major Objectives (cont’d):
3. Develop rigorous uncertainty models; stochastic uncertainty of:
– System states– Model parameters– Sensor network parameters (e.g., location)– Material damage assessments
DDDAS: June 6-7, 2013 5
DDDAS Aspects
Addresses 3 of 4 DDDAS components:
• Applications modeling• Advances in mathematics and statistical
algorithms, and• Application measurement systems and
methods
General Framework
1.World 2. Observations
3.Model
Code
4. Explanations& Predictions
Observe,Measure
AnalyzeControl
Inform
Generate
Validate
Constrain
Verify
DDDAS: June 6-7, 2013 6
Uncertainty Quantification
VVUQ for Sensor Networks
DDDAS: June 6-7, 2013 7
DDDAS: June 6-7, 2013 8
Model Validation
Issues:• Input uncertainty: parameters, initial
conditions, etc.• Model discrepancy: fails to capture
physics, scale, etc.• Cost of computation
Note: This and next 2 slides based on “Assessing the Reliability of Complex Models,” NRC Report, 2012.
DDDAS: June 6-7, 2013 9
Model Validation
• Identify sources of uncertainty• Identify information sources• Assess quality of prediction• Determine resources required to
improve validity
DDDAS: June 6-7, 2013 10
Model Adequacy Measure
Predict:• quantity of interest (QoI)• with acceptable tolerance for intended
application• with uncertainty range attached
e.g., V(x) = 5 +/- 2 with 90% confidence
DDDAS: June 6-7, 2013 11
Our Long-term Goal
Bayesian inference network analysis of:
• Computational uncertainty results • Information from large knowledge bases:
– Maintenance log data– Human knowledge
DDDAS: June 6-7, 2013 12
Model Accuracy Assessment
(Figure based onOberkampf [1])
DDDAS: June 6-7, 2013 13
Model Accuracy Assessment(MAA)
• Compare 7 parameter estimation approaches:
– Inverse method– LLS– MLE– EKF– PF– Levenberg-Marquardt– Minimum RMS Error
DDDAS: June 6-7, 2013 14
Model Accuracy Assessment(MAA)
• Can statistics produced by estimation technique characterize adequacy of the model?
– Which method gives the best k estimate?– Which is least sensitive to noise?– Which has lowest time complexity?
DDDAS: June 6-7, 2013 15
PDE Model: 2D Heat Flow
Truncation Error:
DDDAS: June 6-7, 2013 16
Inverse Method
At each location:
Yields global estimate:
DDDAS: June 6-7, 2013 17
Linear Least Squares
C is the Laplacian term and d is the temporal derivative:
Yields global estimate:
DDDAS: June 6-7, 2013 18
Maximum Likelihood Estimate
Take derivative of log likelihood function of T:
DDDAS: June 6-7, 2013 19
Extended Kalman Filter
from temperature equation at each point
from
where
DDDAS: June 6-7, 2013 20
Particle Filter Method
- Sample p particles from range of distribution- Use weight function to re-calculate particle probabilities- Re-sample particles from new distribution
Continue until change in range is small
DDDAS: June 6-7, 2013 21
Levenberg-Marquardt Method
Use Jacobian:
Solve for k as:
Minimum RMS Method
DDDAS: June 6-7, 2013 22
DDDAS: June 6-7, 2013 23
Model ofPhenomenon Phenomenon
Simulated Data
Measured Data
SensorsAlgorithms& Code
Thermal Data Processing
DDDAS: June 6-7, 2013 24
Model ofPhenomenon Phenomenon
Simulated Data
Measured Data
Algorithms& Code
Regular Mesh
Temperatures
Thermal Data Processing
Sensors
DDDAS: June 6-7, 2013 25
Model ofPhenomenon
TestGeneration
Simulated Data
KnownSolution
Data
Algorithms& Code
PDE’s, Material Points,other
Sequential, parallel, multi-grid,adaptive mesh refinement
Verification: Ensure Code Implements Model
NoiseModels
?
DDDAS: June 6-7, 2013 26
Model ofPhenomenon
Phenomenon
Simulated Data
Measured Data
SensorsAlgorithms& Code
ParameterEstimation
Adjust Model Parameters
Model Parameter Estimation
DDDAS: June 6-7, 2013 27
Model ofPhenomenon Phenomenon
Simulated Data
Measured Data
SensorsAlgorithms& Code
Validation: Make sure Model matches Phenomenon
?
Adjust Model
DDDAS: June 6-7, 2013 28
Example Result(LLS, simulated)
DDDAS: June 6-7, 2013 29
EKF Tracking Results
DDDAS: June 6-7, 2013 30
EKF Predictive Results
DDDAS: June 6-7, 2013 31
UQ in LLS Prediction
DDDAS: June 6-7, 2013 32
Flat Heat Plate Schematic
2D Thermal Data
Raw Thermal Data
DDDAS: June 6-7, 2013 33
FLIR T420 high performance IR camera
320x240 pixel array170x170 over plateSubsampled (smoothed) down to 17x17 array
DDDAS: June 6-7, 2013 34
K Estimate Results
DDDAS: June 6-7, 2013 35
RMS Error of Prediction
DDDAS: June 6-7, 2013 36
Time Cost of Methods
DDDAS: June 6-7, 2013 37
Extrapolative Prediction
DDDAS: June 6-7, 2013 38
Summary of Results
• Given the validation criterion that predicted temperature is within 2 degrees of measured temperature, the accuracy requirements are met.
• Distributions are determined for the thermal diffusivity parameter.
DDDAS: June 6-7, 2013 39
Our Current WorkStudy of Bayesian Computational Sensor Networks for Structural Health Monitoring Using Ultrasound
DDDAS: June 6-7, 2013 40
Our Current Work (cont’d)
DDDAS: June 6-7, 2013 41
Current High-Level Goals
1. Develop Uncertainty Quantification for data driven structural health analysis process.
Dongbin Xiu has joined the University of Utahand we have started discussions on this.
2. Quantify the effect of subject matter judgments with respect to inferences about VVUQ outcomes.
Develop Bayesian inference network methods.
DDDAS: June 6-7, 2013 42
Current Specific Goals
1. Determine prior joint pdf’s describing knowledge of model parameter distributions.
2. Provide proof of robustness and stability of models under the various sources of perturbation (algorithmic, data, etc.).
DDDAS: June 6-7, 2013 43
Current Specific Goals
3. Quantify validation processes to assess the appropriateness of the calibrated model for predictions of quantities of interest (e.g., damage existence, damage extent, model and sensor parameter values).4. Obtain piezoelectric active sensor network experimental results on metallic plates
DDDAS: June 6-7, 2013 44
Questions?