![Page 1: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/1.jpg)
Global Sensitivity Analysis to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access to methods and workflows!Francesca Pianosi and Thorsten Wagener Department of Civil EngineeringUniversity of Bristol!! NE/J017450/1
credible.bris.ac.uk
![Page 2: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/2.jpg)
Global Sensitivity Analysis is a set of statistical techniques that provide a structured approach to tackle several types of uncertaintyassociated with the development and application of numerical models!GSA is useful for: :: more efficient model calibration :: better understanding of model response:: prioritizing efforts for uncertainty reduction (e.g. acquisition of new data):: assessing robustness to modeling assumptions :: …!!
![Page 3: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/3.jpg)
NUMERICAL MODEL!
resolution!
parameters!
forcing inputs!
structure(equations)!
FC!LP!BETA!
time!
output!
pdf!
forcing inputs!
parameters!
interactions!
resolution!
Global Sensitivity Analysis investigates how the variation in the output of a numerical model can be attributed to variations of its input factors!!!
![Page 4: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/4.jpg)
Outline!
:: Key concepts underlying GSA techniques !:: Examples of GSA applications :: SAFE: a Matlab/Octave/R toolbox for GSA !
![Page 6: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/6.jpg)
Steps of sampling-based GSA!
3 POST PROCESSING!
2 MODEL EVALUATION!
1 INPUT SAMPLING!
x1 x2 x3!
sens
itivi
ty!
!!
1 !!
0.5!!
0!
Sensitivity indices!
Latin-hypercubeQuasi-random sampling…!
FC!LP!BETA!output!
pdf!
![Page 7: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/7.jpg)
Steps of sampling-based GSA!
3 POST PROCESSING!
2 MODEL EVALUATION!
1 INPUT SAMPLING!
x1 x2 x3!
sens
itivi
ty!
!!
1 !!
0.5!!
0!
Sensitivity indices!
Sensitivity measured by: :: multiple-start derivatives/differences (e.g. Morris method, DELSA, …)!
:: correlation between inputs and outputs !
:: properties of input distributions after conditioning outputs (Monte Carlo filtering)!!
:: properties of output distribution after conditioning inputs (e.g. Sobol’ method, density-based methods, …)!
![Page 8: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/8.jpg)
Steps of sampling-based GSA!
3 POST PROCESSING!
2 MODEL EVALUATION!
1 INPUT SAMPLING!
x1 x2 x3!
sens
itivi
ty!
!!
1 !!
0.5!!
0!
Sensitivity indices!
!:: Computational cost of post-processingis negligible wrt to model evaluation !:: Required number of model evaluations grows with number of input factors !:: Growth rate differs from method to method!
![Page 9: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/9.jpg)
Choice of GSA method: classification system!
Correlation & Regression Analysis!
Variance- based !&!Density-based!
Multiple-starts derivatives!
Monte-Carlofiltering!
Num
ber o
f mod
el e
valu
atio
ns! >1
0 x
M!
>100
x M
!>1
000
x M!
PAWN!
FAST!
VBSA (Sobol’)!
EET (Morris)!
Regional Sensitivity Analysis!
CART!
Screening! Ranking! Mapping!Specific purpose!
M = number of input factors!
Pianosi et al. 2016 Env.Mod&Soft!
implemented in the SAFEToolbox!
![Page 11: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/11.jpg)
Supporting calibration of a land surface model!
!Which parameters mostly affect the model performance? Which parameters have littleinfluence and can be set todefault values?!
latent heat!
soil heat!
surface skin temperature!
soiltemperature!
sensibleheat!
water on canopy!
transpiration!
rainfall!
evaporation!
rainfall!
soilmoisture!
drainage!
evaporation!
runoff!
runoff!
ENERGY!WATER!
The Joint UK Land Environment Simulator (JULES)!
Application tothe Santa Rita creosotesite in the US!
with J. Iwema, R. Rosolem
![Page 12: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/12.jpg)
Supporting calibration of a land surface model!
1-9: parameters: b sathh satcon sm_sat sm_crit sm_wilt hcap hcon albsoil 10-12: initial conditions: tsar_tile sthuf t_soil!
0.2
0.4 1. Sensible heat
sens
itivity
2. Latent heat
1 2 3 4 5 6 7 8 9 10 11 12
0.2
0.4 3. Soil moisture (TDT)
sens
itivity
1 2 3 4 5 6 7 8 9 10 11 12
4. Soil moisture (CRNS)
Sensitivity of RMSE of different simulated variables to uncertain parameters and initial conditions!
with J. Iwema, R. Rosolem
![Page 13: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/13.jpg)
Supporting calibration of a land surface model!
0.2
0.4 1. Sensible heat
sens
itivity
2. Latent heat
1 2 3 4 5 6 7 8 9 10 11 12
0.2
0.4 3. Soil moisture (TDT)
sens
itivity
1 2 3 4 5 6 7 8 9 10 11 12
4. Soil moisture (CRNS)
Sensitivity of RMSE of different simulated variables to uncertain parameters and initial conditions!
with J. Iwema, R. Rosolem
1-9: parameters: b sathh satcon sm_sat sm_crit sm_wilt hcap hcon albsoil 10-12: initial conditions: tsar_tile sthuf t_soil!
![Page 14: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/14.jpg)
Supporting calibration of a land surface model!
samples that improve model performances wrt default set-up default set-up values!
with J. Iwema, R. Rosolem
parameters! initial conditions!
b sathh satcon sm−sat sm−crit sm−wilt hcap hcon albsoil tstar−tile sthuf t−soil
0.3
11.1
0.1
833.3
0
0.1
0.4
0.7
0
0.7
0
0.7
1000019.1
2999900.8
0.1
3
0.1
0.5
260
320
0
1
260
320
![Page 15: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/15.jpg)
Supporting calibration of a land surface model!
parameters! initial conditions!
b sathh satcon sm−sat sm−crit sm−wilt hcap hcon albsoil tstar−tile sthuf t−soil
0.3
11.1
0.1
833.3
0
0.1
0.4
0.7
0
0.7
0
0.7
1000019.1
2999900.8
0.1
3
0.1
0.5
260
320
0
1
260
320
with J. Iwema, R. Rosolem
samples that improve model performances wrt default set-up default set-up values!
![Page 16: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/16.jpg)
Investigating uncertainties in a flood inundation model!
!How important is the choiceof the model’s spatial resolution for flood simulations with respect to other uncertain factors? !
with J. Savage, P. Bates, J. Freer
LISFLOOD-FP model applied to Imera Basin, Italy!
![Page 17: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/17.jpg)
Investigating uncertainties in a flood inundation model!
Flood extent =percentage of wet cells (water depth > 0.10 m)!!!Uncertain input factors:!- spatial Resolution - Channel friction (parameter)- Floodplain friction (parameter)- Forcing Hydrograph (boundary condition)- DEM: Digital Elevation Model!!!
uncertainty in predicted flood extent!
time (hours)!
with J. Savage, P. Bates, J. Freer
![Page 18: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/18.jpg)
most important contributor to uncertainty (Sobol’)!
time (hours)!
Investigating uncertainties in a flood inundation model!
Flood extent =percentage of wet cells (water depth > 0.10 m)!!!Uncertain input factors:!- spatial Resolution - Channel friction (parameter)- Floodplain friction (parameter)- Forcing Hydrograph (boundary condition)- DEM: Digital Elevation Model!!!
uncertainty in predicted flood extent!
time (hours)!
with J. Savage, P. Bates, J. Freer
![Page 19: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/19.jpg)
Investigating uncertainties in a flood inundation model!
!!!!!!!Uncertain input factors:!spatial Resolution ! !Channel friction !Floodplain friction Forcing Hydrograph !DEM!!
with J. Savage, P. Bates, J. Freer
![Page 20: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/20.jpg)
Finding key drivers of slope failure!
What are the dominant drivers of landslides in a slope with properties known at different level of certainty?!!Slope properties (geometry&soil):!!!!!!Design-storm > deeply uncertain:!!!!!!
with S. Almeida and L. Holcombe
evaporation
runoff
rainfall
water table
slip circle
the Combined Hydrology And Slope Stability Model (CHASM)!www.chasm.info!
Slope angle (degrees)! [27,30]!
Thickness of top soil (m)! [2,6]!
etc. (25 in total)!
Rain
fall!
Time!
Duration!
Intensity!
![Page 21: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/21.jpg)
Finding key drivers of slope failure!
Results of CART analysis!
Cohesion/Thickness
top soil
Stable
>2.0 <2.0
>11 <11
>1.5 <1.5
>3 <3
>47 <47
Stable
>18 <18
Depth WT
>80 <80
Rainfall intensity
<5 >5
Stable
<3.2 >3.2
Stable
Thickness top soil
Stable
Fail
Fail
Rainfall duration
Cohesion/Thickness
top soil
Rainfall duration
Rainfall intensity
Stable >7.5 <7.5
Cohesion/Thickness
top soil
Fail Stable
Rainfall intensity
Fail
with S. Almeida and L. Holcombe
![Page 22: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/22.jpg)
Finding key drivers of slope failure!
Results of CART analysis!!:: the dominant drivers of landslides in this slope are:!1 cohesion of the top soil2 thickness of the top soil3 rainfall duration 4 rainfall intensity5 depth of water table!!
Cohesion/Thickness
top soil
Stable
>2.0 <2.0
>11 <11
>1.5 <1.5
>3 <3
>47 <47
Stable
>18 <18
Depth WT
>80 <80
Rainfall intensity
<5 >5
Stable
<3.2 >3.2
Stable
Thickness top soil
Stable
Fail
Fail
Rainfall duration
Cohesion/Thickness
top soil
Rainfall duration
Rainfall intensity
Stable >7.5 <7.5
Cohesion/Thickness
top soil
Fail Stable
Rainfall intensity
Fail
with S. Almeida and L. Holcombe
![Page 23: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/23.jpg)
Finding key drivers of slope failure!
Results of CART analysis!!:: the dominant drivers of landslides in this slope are:!1 cohesion of the top soil2 thickness of the top soil3 rainfall duration 4 rainfall intensity5 depth of water table!!:: thresholds for these drivers that would lead to slope failure!!
Cohesion/Thickness
top soil
Stable
>2.0 <2.0
>11 <11
>1.5 <1.5
>3 <3
>47 <47
Stable
>18 <18
Depth WT
>80 <80
Rainfall intensity
<5 >5
Stable
<3.2 >3.2
Stable
Thickness top soil
Stable
Fail
Fail
Rainfall duration
Cohesion/Thickness
top soil
Rainfall duration
Rainfall intensity
Stable >7.5 <7.5
Cohesion/Thickness
top soil
Fail Stable
Rainfall intensity
Fail
with S. Almeida and L. Holcombe
![Page 25: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/25.jpg)
Characteristics of the SAFE Toolbox!
:: It works under Matlab/Octave (an R version is also available)!!:: flexible, modular structure
!> easy to integrate with models running outside matlab!:: tutorial scripts (workflows) to get started … more in our introductory paper on Env. Mod & Soft (2015)!
![Page 26: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/26.jpg)
Uptake in academia!
Freely available for non-commercial use since December, 2014www.bris.ac.uk/cabot/resources/safe-toolbox/!!Introductory paper published on Env. Mod & Soft in May, 2015!!About 300 academic users so far!!!About 300 academic users so far!
![Page 27: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/27.jpg)
Uptake in industry!
Download requests for a closed-code 3-months trial version by: - E.ON Energy (Uncertainty in yield prediction for wind farms)- Pfizer (Physiological Based Pharmacokinetic models)- EDF (Thermochemical Heat Storage)!Ongoing collaboration with:!- Risk Management Solutions
!> support calibration of rainfall-runoff models !> find key drivers of loss models
- Airbus!> uncertainty in aircraft design models
- JBA Trust !> support long-term investment plans for flood risk reduction!
!
![Page 28: Global Sensitivity Analysis to support model calibration ... · to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access](https://reader035.vdocuments.net/reader035/viewer/2022070803/5f0322117e708231d407b394/html5/thumbnails/28.jpg)
Conclusions!
Our group @UoB is very active in GSA research, contributing to both methodological advances and development of GSA tools!!The SAFE Toolbox is freely available for non-commercial use from: www.bris.ac.uk/cabot/resources/safe-toolbox/!