Lecture 1: Motivation and Prototypical Examples “Essentially all models are wrong, but some are useful,” George E.P. Box, Industrial Statistician

Predictive Science Components: All involve uncertainty

•  Experiments

•  Models

•  Numerical Simulation

•  Quantity of Interest: usually a statistical quantity; e.g., mean

Example: Weather Models Physical Processes: •  Temperature •  Precipitation •  Winds •  Chemistry of aerosol species

Equations of Atmospheric Physics:

Example: Weather Models

Equations of Atmospheric Physics:

Phenomenological Model for Sources:

where

Example: Weather Models

Sources of Uncertainty: •  Model errors or discrepancies •  Input uncertainties •  Numerical errors and uncertainties •  Measurement errors and uncertainties

Steps:

•  Model Calibration: Involves the assimilation or integration of data to quantify and update input uncertainties.

•  Model Prediction: Here one computes the response along with statistics, error bounds, or PDF; extrapolation is important and difficult.

•  Estimation of the Validation Regime:

• Goal: Construct best estimate parameters and responses or quantities of interest with best estimate reduced uncertainties.

Example: Weather Models Sources of Uncertainty: •  Model errors or discrepancies •  Input uncertainties •  Numerical errors and uncertainties •  Measurement errors and uncertainties

Ensemble Forecasts:

•  Run multiple simulations with differing parameter values or initial conditions drawn from appropriate pdf.

•  A 50% chance of rain means that given present atmospheric conditions, half of simulations predict measurable rain amount at random point in specified area.

Example 2: Pressurized Water Reactors (PWR)

Models:

•  Involve neutron transport, thermal-hydraulics, chemistry

•  Inherently multi-scale, multi-physics

Example 2: Pressurized Water Reactors (PWR) 3-D Neutron Transport Equations:

Challenges:

•  Linear in the state but function of 7 independent variables:

•  Very large number of inputs or parameters; e.g., 100,000

•  ORNL Code: Denovo;

•  Codes can take hours to days to run.

Example 2: Pressurized Water Reactors (PWR) Thermo-Hydraulic Model: Mass, momentum and energy balance for fluid

Challenges:

•  Nonlinear coupled PDE with nonphysical parameters due to closure relations;

•  CASL code: COBRA-TF – Difficult to access primary parameters and inputs.

•  Codes can take minutes to days to run.

Note: Similar equations for gas

Example 2: Pressurized Water Reactors (PWR) UQ Challenges:

•  Specify bounds on void fraction distributions and boiling transitions that guarantee specified performance levels and safety margins.

•  Specify conditions that limit CRUD on the outside of fuel cladding to within prescribed levels.

•  Determine new cladding materials, fuel materials, and fuel pin geometries that provide an average specified improvement in performance and increased resistance to damage.

•  Determine conditions that produce specified levels of radiation damage, mechanical thermal fatigue, and corrosion.

Example 3: HIV Model for Characterization and Control Regimes T1 = �1 � d1T1 � (1� ")k1V T1

T2 = �2 � d2T2 � (1� f")k2V T2

T ⇤1 = (1� ")k1V T1 � �T ⇤

1 �m1ET ⇤1

T ⇤2 = (1� f")k2V T2 � �T ⇤

2 �m2ET ⇤2

V = NT �(T⇤1 + T ⇤

2 )� cV � [(1� ")⇢1k1T1 + (1� f")⇢2k2T2]V

E = �E +bE(T ⇤

1 + T ⇤2 )

T ⇤1 + T ⇤

2 +KbE � dE(T ⇤

1 + T ⇤2 )

T ⇤1 + T ⇤

2 +KdE � �EE

HIV Model:

Compartments:

Example 3: HIV Model for Characterization and Control Regimes HIV Model: Used for characterization and control treatment regimes.

T1 = �1 � d1T1 � (1� ")k1V T1

T2 = �2 � d2T2 � (1� f")k2V T2

T ⇤1 = (1� ")k1V T1 � �T ⇤

1 �m1ET ⇤1

T ⇤2 = (1� f")k2V T2 � �T ⇤

2 �m2ET ⇤2

V = NT �(T⇤1 + T ⇤

2 )� cV � [(1� ")⇢1k1T1 + (1� f")⇢2k2T2]V

E = �E +bE(T ⇤

1 + T ⇤2 )

T ⇤1 + T ⇤

2 +KbE � dE(T ⇤

1 + T ⇤2 )

T ⇤1 + T ⇤

2 +KdE � �EE

Parameters: Most are unknown and must be estimated from data

�1 Target cell 1 production rate ⇢1 Ave. virions infecting type 1 cell

�2 Target cell 2 production rate ⇢2 Ave. virions infecting type 2 cell

d1 Target cell 1 death rate bE Max. birth rate immune effectors

d2 Target cell 2 death rate dE Max. death rate immune effectors

k1 Population 1 infection rate Kb Birth constant, immune effectors

k2 Population 2 infection rate Kd Death constant, immune effectors

c Virus natural death rate �E Immune effector production rate

� Infected cell death rate �E Natural death rate, immune effectors

" Population 1 treatment efficacy NT Virions produced per infected cell

m1 Population 1 clearance rate f Treatment efficacy reduction

m2 Population 2 clearance rate

Example 3: HIV Model for Characterization and Control Regimes

HIV Model: Several sources of uncertainty including viral measurement techniques

0 200 400 600 800 1000 1200 1400 1600 1800200

400

600

800

1000

1200

Patient #6

CD

4+ T−c

ells

/ ul

0 200 400 600 800 1000 1200 1400 1600 1800

102

104

time (days)

viru

s co

pies

/ml

40050

Figure 2: Patient 6 CD4+ T-cell and viral load data, including censor points (lines at L1 =400, L2 = 50) for viral load, and periods of on-therapy (solid lines on axis) and periods of oÆ-therapy (dashed line on axis).

Of the 45 patients considered in this paper, sixteen (those numbered 2, 4, 5, 6, 9, 10, 13, 14, 15,23, 24, 26, 27, 33, 46, and 47) spend 30–70% time oÆ treatment. Of these only patients 9, 15, and47 do not spend appreciable time oÆ treatment during the early half of their observation period.

Due to the linear range limits described above, the clinical viral load assays eÆectively havelower and upper limits of quantification. The upper limit is typically readily handled by repeatedlydiluting the sample until the resulting viral load measurement is in range and then scaling. Thelower limit, or left censoring point, however, directly influences the observed data. When a datapoint is left-censored (below the lower limit of quantification), the only available knowledge is thatthe true measurement is between zero and the limit of quantification L? for the assay. Those athand have two limits of quantification, L1 = 400 copies/ml for the standard and L2 = 50 copies/mlfor the ultra-sensitive assay. These are illustrated in sample data from patient 6 shown in Figure2, where censored data points are those appearing identically on the horizontal censoring linesL1 = 400, L2 = 50. A statistical methodology for handling this type of censored data is describedbelow in Section 3.2.

The observation times and intervals vary substantially between patients. The sample data inFigure 2 also reveal that observations of viral load and CD4 may not have been made at thesame time points, so in general for patient number j we have CD4+ T-cell data pairs (tij1 , y

ij

1 ), i =1, . . . , N

j

1 and (potentially diÆerent) viral RNA data pairs (tij2 , y

ij

2 ), i = 1, . . . , N

j

2 .

6

Example: Upper and lower limits to assay sensitivity

Experimental Uncertainties and Limitations Examples: Experimental results are believed by everyone, except for the person who ran the experiment, Max Gunzburger, Florida State University.

•  Pharmaceutical and disease treatment strategies often too dangerous or expensive for human tests or large segments of the population.

•  Climate scenarios cannot be experimentally tested at the planet scale. Instead, components such as volcanic forcing tested using measurements such as the 1991 Mount Pinatubo data.

•  Subsurface hydrology data very limited due to infeasibility of drilling large numbers of wells. Result: significant uncertainty regarding subsurface structures.

Model Errors Examples: Essentially, all models are wrong, but some are useful, George E.P. Box, Industrial Statistician

•  Numerous components of weather and climate models --- e.g., aerosol-induced cloud formation, greenhouse gas processes --- occur on scales that are much smaller than numerical grids used to solve the atmospheric equations of physics. These processes represent highly complex physics that is only partially understood.

•  Many biological applications are coupled, complex, highly nonlinear, and driven by poorly understood or stochastic processes.

Input Uncertainties Note: Essentially, all models are wrong, but some are useful, George E.P. Box, Industrial Statistician

•  Phenomenological models used to represent processes such as turbulence in weather, climate and nuclear reactor models have nonphysical parameters whose values and uncertainties must be determined using measured data.

•  Forcing and feedback mechanisms in climate models serve as boundary inputs. These parameterized phenomenological relations introduce both model and parameter uncertainties.

Numerical Errors Note: Computational results are believed by no one, except the person who wrote the code, Max Gunzburger, Florida State University.

•  Roundoff, discretization or approximation errors; e.g., mesh for nuclear subchannel code COBRA-TF is on the order of subchannel between rods.

•  Bugs or coding errors;

•  Bit-flipping, hardware failures and uncertainty associated with future exascale and quantum computing;

•  Grids required for numerical solutions of field equations in applications such as weather or climate models (e.g., 50~km) are much larger than the scale of physics being modeled (e.g., turbulence or greenhouse gases).

Steps in Uncertainty Quantification

Note: Uncertainty quantification requires synergy between statistics, mathematics and application area.

Modeling Issues

Computer Model

Conceptual Model Computer Simulation

Programming

Analysis

Model Verification

Model Validation

Model Qualification Reality

Verification Process

Verification Test

Conceptual Model

Computational Model

Computational Solution

`Correct’ Answer

• Analytic solutions

• Highly resolved numerical solutions

• Benchmark solutions

Verification: The process of determining that a model implementation accurately represents the developer’s conceptual description of the model and the solution to the model.

Note: Verification deals with mathematics

Validation Process

Validation Process

Real World

Conceptual Model

Computational Model

Computational Solution

`Correct’ Answer Provided by Experimental Data

• Benchmark cases

• System analysis

• Statistical analysis

Validation: The process of determining the degree to which a model is an accurate representation of the real world from the perspective of the intended model users.

Note: Validation deals with physics and statistics

Validation Metrics

Experiment Model

`Viewgraph’ Norm Input

Res

pons

e

Experiment Model

Deterministic

Input

Res

pons

e

Experiment Model

Numerical Error

Input

Res

pons

e

Experiment Model

Experimental Uncertainty

Input

Res

pons

e Experiment Model

Nondeterministic Computation