an overview on uncertainty management and sensitivity ... · pdf filemodel g formally, the...

17/07/2015

An overview on uncertainty management and

sensitivity analysis problems

Bertrand Iooss

Hanoi - July 2015

Course available on: http://www.gdr-mascotnum.fr/iooss1.html#courses

http://www.gdr-mascotnum.fr/iooss1.html#courses



Hanoi course 2015 – Uncertainty management and sensitivity analysis - B. Iooss 2

A few words about my company : EDF = Électricité de France

220 hydraulic dams - 58 nuclear power plants - Solar energy - Wind power - …


Uncertainty problems in energy production systems

Many uncertainties for the energy production and the safety due to:

• hazards (demand, weather, …),

• incomplete system knowledge (ageing, physics, …),

• internal agressions (failures, …)

• external agressions (earthquake, …)

In order to better understand, prove the safety and optimize its industrial

processes, EDF R&D develops some physical numerical simulation codes


• Large database of component failures

CLASSICAL STATISTICS Des avis d’experts

STATISTIQUE BAYESIENNE

Small database of system failures

Expert Judgment

BAYESIAN STATISTICS

• Pas de REX de défaillances

• Un modèle physique

• • No observed failure

• • Physical model

SIMULATION UNCERTAINTY SIMULATION UNCERTAINTY

Examples: … Chemical accident scenario

… Dam failure

… Waste repository

Statistics for Industrial Risk Management

EDF Research & Development – Department of Industrial Risk Management


Introduction to

uncertainty

management


Starting point: Uncertainties everywhere in a modeling chain!

Main problem: credibility of predictions

Numerical

model

Input

data

Variables

of interest Algorithm

Physical

model

Physical

phenomenon

Simplifications

Numerical

approximation

Parameters

physicist

Computer

scientist

mathematician

statistician

Coding Errors

Model uncertainties

Stochastic

uncertainties

Numerical

uncertainties

Epsitemic

uncertainties

Real observations


Example 1: particle dispersion in atmosphere (1/3)

Topography and location of sources

Domain of study: 10 km around an industrial site

2 arbitrary sources (at ground level) :

* source 1 : tracer (gas)

* source 2 : iodine (particles)

Projection for 4 days

Meteorological data: wind, temperature,

humidity, rain

Rugosity of the ground (vegetation)

Accidental scenario of pollutant release

Source 1 Source 2

Topographie (m)

[ Source : CEA ]


Computation of wind field (direction and amplitude)

Visualization of the wind with flux lines



10 km

Plume under the particular form Concentration plume (at 10 m level)

Use of a computer code of lagrangian particle dispersion

(solving the Euler equations of fluid mechanics )

Visualisation of gas concentrations en gaz after a 5 hours’ release

gaz

aérosols


Results are strongly sensitive to meteorological data


Example 2: Seismic fragility curve (industrial

facilities protection to earthquakes)

Study on a

mock-up

(SMART)

Building a numerical

model (finite-element

based structural

mechanics)

Validation on

experiments

Computing the

sesimic fragility

curves :

Proba (failure | PGA)

PGA = Peak ground Acceleration


Example 3: Optimization of photovoltaic energy

production

inclination

elevation Distance inter stands

Time consuming computer code (electrical model) with:

6 hours per model run

3 « controllable » input variables:

elevation (0,4m to 1m),

inclination (0° to 50°),

distance inter-stands (2,5m to 10m)

2 uncertain variables: meteo and albedo

In output : the production (kWh) of a stand


Similar safety and uncertainty issues in Computational

Science & Engineering and Nature sciences

Climate Modeling :

Prediction

Car and

plane:

Conception

Nuclear industry :

Conception,

Maintenance, risks

Oil, gas, CO2:

Production optimization

Astrophysics:

Understanding


• Modeling phase:

– Improve the model

– Explore the best as possible different input combinations

– Identify the predominant inputs and phenomena in order to priorize R&D

• Validation phase:

– Reduce prediction uncertainties

– Calibrate the model parameters

• Practical use of a model:

– Safety studies: assess a risk of failure (topics of rare events inference)

– Conception studies: optimize system performances and robustness

Summary: Main stakes of uncertainty management


Global methodology of

uncertainty

management


The global methodology of uncertainty management

Coming back

(feedback)

Step C : Propagation

of uncertainty sources

Step C’ : Sensitivity analysis,

Ranking

Model

Input

variables

Uncertain : x

Fixed : d

Variables

of interest

Z = G(x,d)

Decision criterion

e.g.: probability < 10-b

Step A : Specification of the problem

Quantity of

interest

e.g.: variance,

quantile ..

Step B:

Quantification

of uncertainty

sources

Modeled by probability

distributions

x = (x1, x2, …, xp)

Z = G(x,d)


Step A - Which (parametric) uncertainty sources?

• Epistemic uncertainty

– It is related to the lack of knowledge or precision about a parameter which is deterministic in itself (or can be considered deterministic under some accepted hypotheses). E.g. a characteristic of a material.

• Stochastic (or aleatory) uncertainty

– It is related to the real variability of a parameter, which cannot be reduced (e.g. the discharge of a river in flood risk assessment of a riverside area). The parameter is stochastic in itself.

• Reducible vs non-reducible uncertainties

– Epistemic uncertainties are (at least theoretically) reducible

– Instead, stochastic uncertainties are (in general) irreducible (the discharge of a river will never be predicted with certainty)


Step A – Focus on the quantity of interest

Inputs : X

Output : Z

What is really interesting in our study?

m

s

Mean, median, variance,

(moments) of Z

Ex: in the design stage

Pf

treshold

Quantiles (extremes), probability of

treshold exceedence

Ex: in the certification stage

Numerical

model

G

Formally, the quantity of interest is a particular feature of the pdf of Z

Min Max

Ex: in the prior stage

of a new product design

Z


Step B - Quantification of uncertainty sources

Different cases with respect to available information

1. A lot of data (n >> 10)

– Fitting of probability distributions

– Statistical hypothesis tests (often parametric tests)

2. Few data (n < 10)

– Hypothesis on parametric probability distribution

– Non-parametric tests : less powerfull, wide bounds

– Expert judgement, then Bayesian inference

3. No data

– Expert judgment techniques

– Maximum entropy principle

Measure of the “vagueness” of

the information on X

provided by f(x)

Information Maximum Entropy pdf

Uniform

Exponential

Normal


Some comments (Step B). Dependency

Taking into account the dependency between inputs is a crucial

issue in uncertainty analysis Using copulas structure CDF of the vector X

as a function of the marginal CDF of X1 … Xn:

Example: All bivariate densities here

have the same marginal pdf’s

(standard Normal) and the same

Spearman rank coeff. (0.5)


Step C - Uncertainty propagation: main principles

Propagate uncertainties from X to Z, via the deterministic function G (•)

• Conceptually simple problem, but with sometimes a complex

implementation

• Choice of method strongly depends on the quantity of interest

=> importance of step A

This quantity of interest is linked to decisional issues

Two kinds of problems :

• Central tendancy (ex. mean) or

dispersion (variance)

– Metrology

• High quantile, « probability of

failure »

justification of a safety

criterion

Analytical methods

sometimes applicable

Numerical methods

(optimization, Monte Carlo

sampling)


A particular quantity of interest: the “probability of failure”

• G models a system (or a part of it) in operating conditions – Variable of interest Z a given state variable of the system (e.g. a temperature, a

deformation, a water level etc.)

• Following an « operator » point of view – The system is in safe operating condition if Z is above (or below) a given “safety”

threshold

• System “failure” event: – Classical formulation (no loss of generality) in which the threshold is 0 and the

system fails when Z is negative

– Structural Reliability Analysis (SRA) “vision”: Failure if C-L < 0 (Capacity – Load)

• Failure domain:

• Problem: estimating the mean of the random

variable “failure indicator”:


Focus on step C’:

Sensitivity analysis

Y = f(X)


Main objectives of sensitivity analysis

• Reduction of the uncertainty of the model outputs by prioritization of the

sources

• Variables to be fixed in order to obtain the largest reduction (or a fixed

reduction) of the output uncertainty

A purely mathematical variable ordering

• Most influent variables in a given output domain

if reducibles, then R&D prioritization

else, modification of the system

• Simplification of a model

• determination of the non-influent variables, that can be fixed without

consequences on the output uncertainty

• building a simplified model, a metamodel


Typical engineering practice : One-At-a-Time (OAT) design

Main remarks :

OAT brings some information, but potentially wrong

Exploration is poor: Non monotonicity ? Discontinuity ? Interaction ?

X1

X2

P1 P2

P3

Experiment P1 : f (X1[low],X2[low])

Experiment P2 : f (X1[high],X2[low])

Experiment P3 : f (X1[low],X2[high])


Sensitivity analysis notions

• Sensitivity, for example

Answer to: how the output varies with respect to potential perturbations of the

inputs?

• Contribution = sensitivity x importance, for example

Gives the weight of an input (or group of inputs) on the uncertainty of the output

iXY

)( i

i

XX

Y

DX DY

f

X Y


Local methods

Quadratic summation

Contribution of Xi on the output Y :

Computation of derivatives via finite differences, exact differentiation,

or automatic differentiation

p

i

i

i

i

XY

XYXC

1

2

X

2

2

X

2

i

i)(

p

i

ii

Xi

XXX

YYY

1

00 )(0

XX


X1

X2

Screening without hypothesis on function: Morris’

method

• Discretization of input space

• Computation of one

elementary effect for each

input

P1

• Needs p + 1 experiments

• OAT (One-at-A-Time)

P2 P3


X1

X2

1

5

4

3

2

• OAT design is repeated R times

(total: n = R*(p+1) experiments)

• It gives an R-sample for each

elementary effect

• Sensitivity measures:

Morris’ method


Morris: Sensitivity measures

• is a measure of the sensitivity:

Important value important effects (in mean)

sensitive model to input variations

• is a measure of the interactions

and of the non linear effects:

important value different effects in the R-sample

effects which depend on the value:

• of the input Xi => non linear effect

• or of the other inputs => interaction

(the distinction between the two cases is impossible)


Morris: Example

*

Test case: non monotonic funtion of Morris

20 factors

210 simulations

Graph (mu*, sigma)

Distinction between 3

groups:

1. Negligible effects

2. Linear effects

3. Non linear effects

and/or with interactions

1

3

2


Sensitivity analysis for one scalar output

Sample ( X, Y (X ) ) of size n > p , preferably n >> p

Preliminary step: graphical vizualisation (for ex: scatterplots)

Remark: it can be a Monte Carlo sample, a quasi-Monte Carlo sample or

any other designs


Graphical representation: Scatterplots

YX

YX

),cov(

yx

N

i

ii

ss

yyxxN

1

))((1

0.00 0.02 0.04 0.06 0.08 0.10

0

2

4

6

8

10

i3

p23

N runs

Plot the output with respect to each input

Example :

N = 300 Measure the linear character of the cloud


Flood model - Scatterplots – Output S

Monte Carlo sample – N = 100

Q = river flowrate: pdf ~ Gumbel on [500,3000]

Ks = friction coefficient: pdf ~ normal on [15,50]

Zv = downstream river bed heigth: pdf ~ triangular on [49,51]

Hd = dyke heigth: pdf ~ triangular on [7,9]

Cb = bank heigth: pdf ~ triangular on [55,56]


Flood model - Scatterplots – Output Cp


Major drawback: only first order relations between inputs are analyzed

and not their interactions (=> need other data analysis tools)


Interaction analysis tool: Cobweb plot




? Linear relation ?

Oui

Linear

regression

between Xi

and Y

Regression

coefficients

(R²)

Sensitivity

indices Sobol’ indices

Non ? Monotonic relation ?

Oui Non

Regression on

ranks

(R²*)

Preliminary step: graphical vizualisation (for ex: scatterplots)

Quantitative sensitivity analysis methodology


Sensitivity indices in case of linear inputs/output relation

Independent inputs

Sample: n realizations of

• Standard Regression Coefficient (SRC) :

Sign of bi gives the direction of variation of Y / Xi

• Similar to the linear correlation coefficient (Pearson)

• Validity of the linear model

via regression diagnostics and R² :

• Theoretically, we have , useful to interpret SRC

p

i

ii XY1

0 :modellinear of Hypothesis bb

)Var(

)Var(:

Y

XXSRC i

ii b

)( with , XX fYY

n

i

i

n

i

ii

YY

YY

R

1

2

1

2

2

ˆ

1

pX,X ...,1X

p

i

iXR1

22 SRC


Flood model - Output S


31% 15% 18% 29% 6%

Sensitivity indices (SRC2)

The model is linear (R²=0.99)

SRC coefficients are sufficient for sensitivity analysis




? Linear relation ?

Oui

Linear

regression

between Xi

and Y

Regression

coefficients

(R²)

Sensitivity



Oui Non

Regression on

ranks

(R²*)

Preliminary step : graphical vizualisation (for ex: scatterplots)





? Linear relation ?

Oui

Linear

regression

between Xi

and Y

Regression

coefficients

(R²)

Sensitivity



Oui Non

Regression on

ranks

(R²*)



)Var(

)]Var[E(

Y

XYS

i

i


Sensitivity indices without model hypotheses

Sobol indices definition:

• First order sensitivity indices:

• Second order sensitivity indices:

• ...

)Var(Y

VS i

i

)Var(Y

VS

ij

ij

)()()()(Var 12

1

YVYVYVY p

p

ji

ij

p

i

i

Functional ANOVA [Efron & Stein 81] (hyp. of independent Xi s) :

... , )E(Var

)]E([Var)( where

jijiij

ii

VVXXYV

XYYV


Graphical interpretation

2 4 6 8 10 12 14

02

46

810

per1

p23

0 20 40 60 80

02

46

810

kd1

p23

0 50 100 150

02

46

810

kd2

p23

0.00 0.02 0.04 0.06 0.08 0.10

02

46

810

i3

p23

First order Sobol’ indices measure the variability of the conditional

expectations (mean trend curves in the scatterplots)

Null

index

Small

index

High

index


1i

iS

1i

iS

Always

Additive model

i

iS1 Measure the degree of interactions between variables

k

i j k

ijk

i j

ij

p

i

i SSSS ,...,2,1

1

...1

Sobol indices properties

Examples : p =4 gives 4 indices Si, 6 indices Sij , 4 indices Sijk, 1 indice Sijkl

General case : 2p-1 indices to be estimated

Total sensitivity index:

[ Homma & Saltelli 1996 ]

i

j kj

ijkijiTi SSSSS ~

,

1...


Flood model


Sobol indices computation

• Indices for Xi (1st order and total) :

• Formulations of the conditional variances:

)(),(CovEE)]Var[E()( '

~,~,

22

iiiiiiiiii XXfXXfdXXYdXXYXYYV

),(),(Cov)]Var[E()( ~

'

~,~~ iiiiii XXfXXfXYYV

XXX ofcopy t independenan ' and )(Let ~, ii XX

)Var(1 and

)Var(

~

Y

VS

Y

VS i

Ti

ii




? Linear relation ?

Yes

Linear

regression

between Xi

and Y

Regression

coefficients

(R²)

Sensitivity

indices

Sobol’ indices

No ? Monotonic relation ?

Yes No

Regression on

ranks

(R²*)


Quantitative sensitivity analysis methodology [Saltelli et al. 00, Helton et al. 06 ]

?cpu time?

Large Negligible

Metamodel

N > 10 p

Monte Carlo

N > 1000 p

Small Quasi-MC,

FAST, RBD

N > 100 p


Uncertainties management for cpu time consuming models

Physical phenomena Computer code

p input variables

X = (X1, …, Xp )

observed

experiences

simulated

experience

s

Metamodel

Predicted

experiences

Identification of input

parameters values

Adequation between

observed and simulated

experiences

Use of the metamodel :

C’:Sensitivity analysis

Metamodel

)(XfY SRSR

Distribution of

the inputs Distribution

of the output

C:Uncertainty propagation

(via Monte Carlo methods)

B’:Calibration

A useful solution : the metamodel (model of the numerical model)

Y (X)

Time consuming

Ŷ (X)

Negligible cost


Flood model – Output Cp

From the 100-size Monte Carlo sample, a kriging metamodel is fitted

Predictivity of the kriging metamodel : Q2 = 99%

Sobol’ indices estimation

• Direct estimation (« modèle »): n =1e5 and 100 replicates

=> N = n x (p+2) x 100 = 7e7 evaluations of the model

• « métamodèle » estimation

=> N = 100 evaluations of the model and 7e7 evaluations of the metamodel


Application example


Application example: hydrogeological modeling

Site (2 ha) near Moscow

From 1943 to 1974 : radioactive waste repositories

1990 : site recognition with 20 piezometers

Upper aquifer contamination in 90Sr

Collaboration Kurchatov Institute/CEA

Questions:

- impact assessment of the contamination on the environment

- degree of rehabilitation of the site

[ Volkova et al. 2008 ]


Computation of the spatio-temporal evolution of 90Sr concentration in an ancient radwaste disposal site between 2002 and 2010

Goal: Estimate the contamination impact on the environment Identify the influent inputs on predicted outputs

p = 20 uncertain inputs X:

Permeability, dispersivity, Kd, infiltration intensity, …

Numerical modeling:

Hydrogeological transport (Darcy’s law) scenario of 90Sr with the MARTHE software

Output of interest:

Concentration values Y

Uncertainty management in hydrogeological modeling

30 mn / run

Step A

Quantity of interest :

Distribution, variance


Step B: Quantification of uncertainty sources

Statistical modeling of the uncertainty of each input

Different cases:

1. A lot of data

– Fitting of probability distributions

– Statistical hypothesis test

MARTHE case : hydraulic conductivity (lognormal distribution)

2. Few data (n < 10)

– Bayesian inference

3. No data

– Bibliography

– Expert judgment techniques

MARTHE case : dispersivity, permeability, … (uniform distributions)


Step B: Quantification of uncertainty sources

Input variables of the model nominal value, type of distribution and parameters


August 2002 December 2010

Concentration maps

Step C: Uncertainty propagation

0 2 4 6 8 10 12 0 2 4 6 8 10

Histogram & density - p23

Density

0.0

0

0.0

2

0.0

4

0.0

6

0.0

8

0.1

0

0.1

2

Histogram & density – p31K

Density

0.0

0.2

0.4

0.6

0

.8

1.0

1.2

1.4

N = 300 runs from Latin Hypercube Sample concentration distributions at the 20 piezometers (Bq/l)

Initial concentration


Outputs with larger R2 (linear relation )

- p23 (R2 = 0,78) p104 (R²=0,68)

- p4-76 (R2 = 0,71)

Outputs with larger R2* (monotonic relation)

- p4-76 (R2* = 0,95)

- p107 (R2* = 0,92)

- p104 (R2* = 0,91)

p102K (R2* = 0,90)

p23 (R2* = 0,90)

p29K (R2* = 0,83)

Most influent inputs

- Distribution coefficient, layer 2

- Distribution coefficient, layer 1

- Infiltration intensity

- permeability, layer 2

Step C’: Regression-based approach

PROBLEM: 14 outputs are non monotonic and we have only 300 simulation runs


Gaussian process (Gp) metamodel : Predictivity coefficient: Q2 = 93% - Linear regression : Q2 = 68% Sobol indices estimation + confidence intervals This can be done for several outputs …

(en %) SRCi²

(linear regression)

Gaussian process

IC- 90%

Gaussian

process

per1 2 8 [ 5 ; 11 ]

kd1 52 69 [ 56 ; 83 ]

i3 13 13 [ 10 ; 17 ]

Sensitivity analysis results for one scalar output « p104 »

[ Marrel et al. 2009 ]


Group 1 : kd1

(distribution coef. of layer 1)

Group 2 : kd2

(distribution coef. of layer 2)

Group 3 : i3

(infiltration intensity)

- group 1

- group 3

- group 2

Step C’: sensitivity analysis for 20 scalar outputs

Main influent inputs

Spatial location map

… but the results are difficult to synthesize, then to interpret


We have simulated N = 300 maps (Monte Carlo runs) with p = 20 random inputs

Considering functional (spatial) output

Concentration in 90Sr predicted in 2010

9 output simulated

maps

4096 pixels

Discretized spatial output can be considered as a functional 2D output


Sensitivity analysis when model ouputs are functions

Elementary cases (sensitivity analysis on each scalar output):

– Very small CPU time consuming model

– Linear or monotonic model

Difficult cases:

– Complex/Non linear model need of Sobol indices

– CPU time expensive model need of metamodel


Sensitivity analysis for spatial outputs: methodology

• Computer code f (.) :

Input: X = ( X 1 , … , X p ) random vector

Output for input x* : y = f (x* , z ) , z Dz R2

In practice, Dz is discretized in nz points (here: 64 x 64 = 4096 points)

(X,Y (X,z) ) = input-output sample of size N

• Decomposition of Y (z) on an orthogonal function basis (fixed basis)

For example, a wavelet basis is well-suited if there are discontinuities

• Modeling of the decomposition coefficients by a kriging metamodel

Selection procedure of the most important coefficients

• Prediction: x* => prediction of coeff. => spatial output map reconstruction

Sensitivity analysis :

Spatial maps of sensitivity indices Functional metamodel


Step 1 : Wavelet decomposition of each map (300 maps)

Step 2 : Kriging metamodeling of the main wavelet coefficients (the most variables) in fonction of X ; constant for the other coef.

Step 3 : Prediction for a new input x*

x* => prediction of the coefficients => spatial output map reconstruction

Example of use: Sensitivity analysis (spatial map of sensitivity indices)

Sensitivity indices (Sobol indices, based on functional ANOVA): The estimation of Sobol indices require a large number of simulations computation of the Sobol indices by the way of the metamodel

Metamodel fitting: methodology for spatial output

j kj

ijkijiTiij

iXi SSSSS

YVar

XYEVarS i ...;...;;

)(

)]([

,

1st order 2nd order Total indices


N = 300 simulations

p = 20 random input variables

K = 4096 pixels

k = 100 wavelet coefficients

modeled by Gaussian process

Mean predictivity (functional metamodel): Q2 = 72%

Estimation of first order and total Sobol’ indices maps by Monte Carlo (22000 runs with the fonctional metamodel)

20 maps of sensitivity indices

Application on our test case (hydrogeological pollution)

[ Marrel et al. 2011 ]


Spatial maps of Sobol sensitivity indices of first order, for 6 inputs

Spatial output: results of sensitivity analysis

Permeability layer 1 Permeability layer 2 Permeability layer 3

Kd layer 1 Kd layer 2 Strong infiltration intensity


Conclusion


(quantity of interest = variability of the output)

Three types of answers:

1. Screening

- classical design of experiments

- numerical design of experiments (Morris, sequential bifurcation)

2. Quantitative measures of global influence

- correlation/regression on values/ranks

- functional variance decomposition (Sobol)

3. Deep exploration of sensitivities

- smoothing techniques (param./non parametric)

- metamodels

Overall classification of sensitivity analysis methods

X1

X3

X2

Y


Synthesis: Classification of sensitivity analysis methods

p 10p 100p

Linear 1st

degree

Monotonic +

interactions

Monotonic

without

interaction

Metamodel

2p

Morris

( p = number of input variables )

0

Non

monotonic

Number of

model f

evaluations

Super screening

Variance decomposition

Design of experiment

Rank regression

Linear regression

Complexity/regularity

of model f

Calculations of all types of indices

(Sobol, distribution-based, …)

+ main effects E(Y | Xi )

Screening

Sobol indices

Monte-Carlo sampling

DGSM


- E. de Rocquigny, N. Devictor and S. Tarantola (eds). Uncertainty in industrial practice,

Wiley, 2008.

- B. Iooss and P. Lemaître. A review on global sensitivity analysis methods -

http://fr.arxiv.org/abs/1404.2405

- Faivre et al., Analyse de sensibilité et exploration de modèles – Applications aux

sciences de la nature et de l’environnement, Editions Quaé, 2013

- J.C. Helton, J.D. Johnson, C.J. Salaberry et C.B. Storlie: Survey of sampling-based

methods for uncertainty and sensitivity analysis. Reliability Engineering and System

Safety, 91:1175–1209, 2006

- Kleijnen, The design and analysis of simulation experiments, Springer, 2008

- A. Saltelli, K. Chan & E.M. Scott, Sensitivity analysis, Wiley, 2000

- A. Saltelli et al., Global sensitivity analysis - The primer. Wiley, 2008

Bibliography



an overview on uncertainty management and sensitivity ... · pdf filemodel g formally, the...

Documents