Issues Related to Parameter Estimation in

Model Accuracy Assessment

DDDAS: June 6-7, 20131

Tom Henderson&

Narong Boonsirisumpun

ICCS 2013Barcelona, Spain

6 June 2013

Funded in part by grant AFOSR-FA9550-12-1-0291.

DDDAS: June 6-7, 2013 2

Major Objectives:

1. Develop Bayesian Computational Sensor Networks– Detect & identify structural damage– Quantify physical phenomena and sensors– Characterize uncertainty in calculated

quantities of interest (real and Boolean)

DDDAS: June 6-7, 2013 3

Major Objectives (cont’d):

2. Develop active feedback methodology using model-based sampling regimes

– Embedded and active sensor placement– On-line sensor model validation– On-demand sensor complentarity

DDDAS: June 6-7, 2013 4

Major Objectives (cont’d):

3. Develop rigorous uncertainty models; stochastic uncertainty of:

– System states– Model parameters– Sensor network parameters (e.g., location)– Material damage assessments

DDDAS: June 6-7, 2013 5

DDDAS Aspects

Addresses 3 of 4 DDDAS components:

• Applications modeling• Advances in mathematics and statistical

algorithms, and• Application measurement systems and

methods

General Framework

1.World 2. Observations

3.Model

Code

4. Explanations& Predictions

Observe,Measure

AnalyzeControl

Inform

Generate

Validate

Constrain

Verify

DDDAS: June 6-7, 2013 6

Uncertainty Quantification

VVUQ for Sensor Networks

DDDAS: June 6-7, 2013 7

DDDAS: June 6-7, 2013 8

Model Validation

Issues:• Input uncertainty: parameters, initial

conditions, etc.• Model discrepancy: fails to capture

physics, scale, etc.• Cost of computation

Note: This and next 2 slides based on “Assessing the Reliability of Complex Models,” NRC Report, 2012.

DDDAS: June 6-7, 2013 9

Model Validation

• Identify sources of uncertainty• Identify information sources• Assess quality of prediction• Determine resources required to

improve validity

DDDAS: June 6-7, 2013 10

Model Adequacy Measure

Predict:• quantity of interest (QoI)• with acceptable tolerance for intended

application• with uncertainty range attached

e.g., V(x) = 5 +/- 2 with 90% confidence

DDDAS: June 6-7, 2013 11

Our Long-term Goal

Bayesian inference network analysis of:

• Computational uncertainty results • Information from large knowledge bases:

– Maintenance log data– Human knowledge

DDDAS: June 6-7, 2013 12

Model Accuracy Assessment

(Figure based onOberkampf [1])

DDDAS: June 6-7, 2013 13

Model Accuracy Assessment(MAA)

• Compare 7 parameter estimation approaches:

– Inverse method– LLS– MLE– EKF– PF– Levenberg-Marquardt– Minimum RMS Error

DDDAS: June 6-7, 2013 14

Model Accuracy Assessment(MAA)

• Can statistics produced by estimation technique characterize adequacy of the model?

– Which method gives the best k estimate?– Which is least sensitive to noise?– Which has lowest time complexity?

DDDAS: June 6-7, 2013 15

PDE Model: 2D Heat Flow

Truncation Error:

DDDAS: June 6-7, 2013 16

Inverse Method

At each location:

Yields global estimate:

DDDAS: June 6-7, 2013 17

Linear Least Squares

C is the Laplacian term and d is the temporal derivative:

Yields global estimate:

DDDAS: June 6-7, 2013 18

Maximum Likelihood Estimate

Take derivative of log likelihood function of T:

DDDAS: June 6-7, 2013 19

Extended Kalman Filter

from temperature equation at each point

from

where

DDDAS: June 6-7, 2013 20

Particle Filter Method

- Sample p particles from range of distribution- Use weight function to re-calculate particle probabilities- Re-sample particles from new distribution

Continue until change in range is small

DDDAS: June 6-7, 2013 21

Levenberg-Marquardt Method

Use Jacobian:

Solve for k as:

Minimum RMS Method

DDDAS: June 6-7, 2013 22

DDDAS: June 6-7, 2013 23

Model ofPhenomenon Phenomenon

Simulated Data

Measured Data

SensorsAlgorithms& Code

Thermal Data Processing

DDDAS: June 6-7, 2013 24

Model ofPhenomenon Phenomenon

Simulated Data

Measured Data

Algorithms& Code

Regular Mesh

Temperatures

Thermal Data Processing

Sensors

DDDAS: June 6-7, 2013 25

Model ofPhenomenon

TestGeneration

Simulated Data

KnownSolution

Data

Algorithms& Code

PDE’s, Material Points,other

Sequential, parallel, multi-grid,adaptive mesh refinement

Verification: Ensure Code Implements Model

NoiseModels

?

DDDAS: June 6-7, 2013 26

Model ofPhenomenon

Phenomenon

Simulated Data

Measured Data

SensorsAlgorithms& Code

ParameterEstimation

Adjust Model Parameters

Model Parameter Estimation

DDDAS: June 6-7, 2013 27

Model ofPhenomenon Phenomenon

Simulated Data

Measured Data

SensorsAlgorithms& Code

Validation: Make sure Model matches Phenomenon

?

Adjust Model

DDDAS: June 6-7, 2013 28

Example Result(LLS, simulated)

DDDAS: June 6-7, 2013 29

EKF Tracking Results

DDDAS: June 6-7, 2013 30

EKF Predictive Results

DDDAS: June 6-7, 2013 31

UQ in LLS Prediction

DDDAS: June 6-7, 2013 32

Flat Heat Plate Schematic

2D Thermal Data

Raw Thermal Data

DDDAS: June 6-7, 2013 33

FLIR T420 high performance IR camera

320x240 pixel array170x170 over plateSubsampled (smoothed) down to 17x17 array

DDDAS: June 6-7, 2013 34

K Estimate Results

DDDAS: June 6-7, 2013 35

RMS Error of Prediction

DDDAS: June 6-7, 2013 36

Time Cost of Methods

DDDAS: June 6-7, 2013 37

Extrapolative Prediction

DDDAS: June 6-7, 2013 38

Summary of Results

• Given the validation criterion that predicted temperature is within 2 degrees of measured temperature, the accuracy requirements are met.

• Distributions are determined for the thermal diffusivity parameter.

DDDAS: June 6-7, 2013 39

Our Current WorkStudy of Bayesian Computational Sensor Networks for Structural Health Monitoring Using Ultrasound

DDDAS: June 6-7, 2013 40

Our Current Work (cont’d)

DDDAS: June 6-7, 2013 41

Current High-Level Goals

1. Develop Uncertainty Quantification for data driven structural health analysis process.

Dongbin Xiu has joined the University of Utahand we have started discussions on this.

2. Quantify the effect of subject matter judgments with respect to inferences about VVUQ outcomes.

Develop Bayesian inference network methods.

DDDAS: June 6-7, 2013 42

Current Specific Goals

1. Determine prior joint pdf’s describing knowledge of model parameter distributions.

2. Provide proof of robustness and stability of models under the various sources of perturbation (algorithmic, data, etc.).

DDDAS: June 6-7, 2013 43

Current Specific Goals

3. Quantify validation processes to assess the appropriateness of the calibrated model for predictions of quantities of interest (e.g., damage existence, damage extent, model and sensor parameter values).4. Obtain piezoelectric active sensor network experimental results on metallic plates

DDDAS: June 6-7, 2013 44

Questions?