project acronym: pamina - tu clausthal · project acronym: pamina milestone 2.1.d.11: sensitivity...
TRANSCRIPT
Project acronym: PAMINA
Milestone 2.1.D.11:
Sensitivity Analysis Benchmark Based on the Use of Analytic and
Synthetic PA Cases (Topic Report)
Reference: FP6-036404
Version: 1.0
RTDC: 2
Work package: 2.1.D
Author: Elmar Plischke (TU Clausthal, Germany),
Klaus-Jürgen Röhlig (TU Clausthal, Germany),
Anca Badea (JRC Petten, Netherlands),
Ricardo Bolado Lavín (JRC Petten, Netherlands),
Per-Anders Ekström (Facilia SA, Sweden),
Stephan Hotzel (GRS Cologne, Germany)
Date of working paper: 28/05/2009
PAMINA Milestone M2.1.D.11: Sensitivity AnalysisBenchmark Based on the Use of Analytic and
Synthetic PA Cases (Topic Report)
Elmar Plischke∗, Klaus-Jurgen Rohlig†, Anca Badea‡,Ricardo Bolado Lavın§, Per-Anders Ekstrom¶, Stephan Hotzel‖
May 28, 2009
Contents1 Introduction 3
2 Problem formulation 4
3 Sensitivity Indices 4
4 Overview of the Tested Algorithms 54.1 Correlation Ratios: Graphical method extended . . . . . . . . . . . . . . . . . . 64.2 Polynomial Fit: If a linear model is not enough . . . . . . . . . . . . . . . . . . 84.3 Conditional linear fit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94.4 Methods using special input sampling . . . . . . . . . . . . . . . . . . . . . . . 10
4.4.1 Ishigami-Homma-Saltelli/Sobol´ . . . . . . . . . . . . . . . . . . . . . . 104.4.2 Fourier Amplitude Sensitivity Test/Random Balance Design . . . . . . . 10
4.5 Effective Algorithm: Combining given data with Fourier amplitude . . . . . . . . 13
5 The analytical benchmark cases 135.1 Ishigami function: A model with three input parameters and higher order effects . 145.2 A discontinuous switch example . . . . . . . . . . . . . . . . . . . . . . . . . . 17
∗[email protected]†[email protected]‡[email protected]§[email protected]¶[email protected]‖[email protected]
2
5.3 A linear model with dependent input data . . . . . . . . . . . . . . . . . . . . . 215.4 Sobol’ g function: A model with eight input parameters . . . . . . . . . . . . . . 27
6 The PA case example: The GTM Level-E model 306.1 Peak of total dose rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316.2 Time of occurrence of the peak . . . . . . . . . . . . . . . . . . . . . . . . . . . 336.3 Time-dependent total dose rate . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
7 Conclusions 39
References 42
A A short UA/SA implementation 44
B More benchmark results 44
1 IntroductionThe PAMINA task 2.1.D–Techniques for Sensitivity and Uncertainty Analysis–compares therelative advantages and disadvantages of different methods for applying Sensitivity Analysis(SA) to performance assessment calculations. This report is part of a benchmark study aimedat testing a wide range of Sensitivity Analysis techniques on test cases. In a previous MilestoneReport [11], we issued the plan for such a benchmark study. In this Milestone Report, we reporton the results on this benchmarks study by analysing nonlinear techniques for SA. In order togain experience with the tools and techniques of SA we first apply them to analytical test models,the results of which are reported in Section 5. As a more application-oriented model, we study ageosphere transport model in Section 6.Uncertainty Analysis (UA) and Sensitivity Analysis (SA) form an integral part of gaining knowl-edge of computational models used for the analysis and prediction in many parts of engineeringand applied sciences. UA is handled by computing standard statistical indicators (mean, vari-ance) based on a given sample of corresponding input data and output data. Asides from thesestandard indicators, Sensitivity Analysis indicators should provide hints to questions like:
• Which uncertain parameters mostly contribute to the output uncertainty?
• Are there any parameters whose uncertainties have negligible effects on the output?
• Is there a set of uncertain parameters which has a combined effect on the output variabilitywhile the individual influences are not noticeable?
In most practical cases, SA is performed using indicators based on linear regression techniques,including rank-based techniques. However, for complex models where no linear or monotonousdependency is apparent, such a SA is not very powerful. Especially, the dependency on higherorder terms cannot be fully explained when using only linear regression techniques.
3
When the model under inspection is a linear one, variance-based Sensitivity Analysis indicatorsyield similar results in terms of information about parameter importance as Pearson CorrelationCoefficients, Partial Correlation Coefficients or Standard Regression Coefficients. Additionally,for nonlinear, non-monotonous models those indicators provide information not retrievable fromindicators based on global linear regression techniques.For our purposes, we distinguish two types of sensitivity indicators, named first order effects(also called: main effects, Sobol’ indices, correlation ratios, importance measures) which can beused in a Factor Prioritisation (FP) setting by detecting direct influences from the input parame-ters to the output parameter, and total effects which can be used in a Factor Fixing (FF) settingby detecting indirect influences. In a FP setting most influential parameters are identified forwhich further research (for reducing their lack-of-knowledge uncertainty) improves the modelsignificantly, in a FF setting unessential parameters are identified which may be fixed to a con-stant value without changing the overall model behaviour. For further discussion see [14, Section1.2].
2 Problem formulationWe consider a computational model y = f(x1, . . . , xk) with k (scalar) input parameters xi and a(scalar) output y. However, the values of the input parameters are not exactly known. We assumethat this uncertainty can be handled by using random variables Xi of known distributions. Thenthe model output is also a random variable Y = f(X1, . . . , Xk). To determine its properties oneuses the tools of Uncertainty Analysis and Sensitivity Analysis. These tools require realisationsof the input distributions and model evaluations, e.g., to compute the mean as an estimate forthe expected value of Y . We will denote one realisation of a parameter set with (x1, . . . , xk),multiple realisations are shown in matrix notation X = (xji)j=1,...,n,i=1,...,k.
3 Sensitivity IndicesFor a short introduction to Sensitivity Analysis, see [1]. We will draw most of our attentionto methods described therein in Section 5.2 (Monte Carlo based methods) which contains adiscussion of regression techniques and in Section 5.3 (Variance decomposition based methods)where the Sobol’ indices are introduced and methods for their estimation are presented.For convenience, some details for variance decomposition are mentioned here (cf. [19]). Thevariance of Y can be expressed in terms of the conditional variance
V[Y ] = E[V[Y |XI ]] + V[E[Y |XI ]] (1)
where XI = (Xi)i∈I is a random vector, and I ⊂ {1, . . . , k} is an index set of “interest-ing” factors. E[Y |XI ] is the conditional expectation of Y given XI and V[Y |XI ] = E[(Y −E[Y |XI ])
2|XI ] is the conditional variance, respectively. These two are random variables of XI .
4
The Sensitivity Index (SI) with respect to the index set I is then given by
SI =V[E[Y |XI ]]
V[Y ]or SI = 1− E[V[Y |XI ]]
V[Y ].
If the index set contains just one element i ∈ I , the SI is called first order effect or main effectof i. If I contains all but one index i 6∈ I then ST i = 1− SI is the total effect of i. Analogously,the total effect of an index set I is defined as STI = 1 − SIC where IC = {1, . . . , k} \ I is thecomplement set of I . The name “total effect” should not distract from the fact that in case ofdependent random vectors the total effect does not include the part of the variance of Y whichescapes through the dependency of XIC on XI .The Sensitivity Index has the following properties:
• SI ∈ [0, 1].
• If SI = 1 then Y is a function of XI .
• If XI and Y are independent then SI = 0.
For the first order and total effects of single factors i the following results hold true:
• If X1, . . . , Xk are independent then∑k
i=1 Si ≤ 1 and∑k
i=1 ST i ≥ 1.
• If f(x1, . . . , xk) is an additive function in all of the parameters xi (in particular, if f islinear) then Si = ST i and
∑ki=1 Si = 1.
For the derivation of some of the methods presented below, the variance of the conditional ex-pectation of Y satisfies the following limit,
V[E[Y |XI ]] = limE[ϕ(XI)−E[Y |XI ]]→0
E[(EY − ϕ(XI))
2] (2)
for a sequence of (square-integrable) functions ϕ : R` → R of XI where ` = card I ≤ kdenotes the number of factors in I (which is the dimension of the random vector XI). Settingϕ(x) = E[Y |XI = x] yields equality in (2). This special choice of the function ϕ is a non-parametric regression curve.With (2) in mind the first order effect Si is the fraction of the variance of Y that is explained bya functional dependency on Xi alone, while the total effect ST i is the fraction of the variance ofY that is not explained by a functional dependency on all parameters but Xi.
4 Overview of the Tested AlgorithmsSome of the methods which we present in the next subsections have not caught much attentionin the current literature. These are “cheap methods” in the sense that the Sensitivity Indices canbe estimated from a given sample of realisations of the input variables and its associated set of
5
model outputs which were, e.g, already used for Uncertainty Analysis. Hence these methods areof special value for practitioners when keeping efficiency in mind.The formula (2) implies that an estimator for the Sensitivity Index with respect to the index set Iis given by
SI =
∑nj=1(y(xIj)− y)2∑nj=1(yj − y)2
, y =1
n
n∑j=1
yj, (3)
where y is a model prediction based on the input data from the parameter group I with E[(y −E[Y |XI ])
2] small. Clearly, one cannot consider all possible functions for y. Instead, we usedifferent model classes which are “rich” enough to provide a good estimate of E[Y |XI ]. If yis constructed from a linear regression model then we have already noted that the result of (3)(which is, in this case, the standard regression coefficient) is not necessarily good in case ofnon-linear, non-monotonic output data. We may therefore try a class of locally constant models(Correlation Ratios), models performing higher-order polynomial model fitting or models usinglocally linear regression techniques. Other regression techniques, like kernel density estimators,have not been used in this benchmark.A different approach to computation of SA requires special sampling schemes. We will brieflydiscuss the used methods based upon repetitive model evaluations (Sobol’) or upon a frequencyresponse setting (FAST). The use of other sampling schemes like winding stairs sampling or al-ternative orthogonal transformations (Walsh-Hadamard) was not within the scope of the bench-mark.
4.1 Correlation Ratios: Graphical method extendedOne straight-forward method of estimating V[E[Y |Xi]] is to consider E[Y |Xi ∈ Im] where{Im,m = 1, . . . , `} is a partition of the whole input parameter range for Xi into ` subsam-ples. Then from an estimate of the conditional variance for yj = f(xj1, xj2, . . . , xji, . . . , xjk),j = 1, . . . , n we obtain
ym =1
nm
∑xji∈Im
yj, nm = card{xji ∈ Im},
Si =
∑`m=1 nm(ym − y)2∑n
j=1(yj − y)2(CR-VCE)
where card A denotes the number of elements in the set A. Figure 1 demonstrates this processof taking local means.Alternatively, based on decomposition (1) one can also compute the mean of the conditionalvariances instead of estimating the variance of the conditional means,
s2m =
1
nm − 1
∑xji∈Im
(yj − ym)2,
Si = 1− (n− 1)∑`
m=1 s2m
(n− `)∑n
j=1(yj − y)2. (CR-ECV)
6
−2 −1.5 −1 −0.5 0 0.5 1 1.5 20
0.5
1
1.5
2
2.5
xi
yA Non−Monotonic Scatter Plot
dataconditional meanmean
Figure 1: Correlation Ratios: Computing the variance of conditional means.
For the number of subsamples in a partition, [6] suggests ` = b√nc. The TU Clausthal imple-
mentations use the upper integer bound, ` = d√ne (which we will call “rule of thumb” in the
following text). Hence we can expect ` realisations in each of the ` subsamples. However, thetests performed by R. Bolado and A. Badea, JRC-Petten, for this benchmark exercise suggestthat this choice can be sub-optimal, see the notes [2] in the electronic appendix.Using a higher-dimensional partition, correlation ratio methods are also able to compute higher-order effects. Unfortunately, for computing the interaction between multiple factors as for totaleffects this method suffers from the curse of dimensionality: The number of subsamples in apartition grows with the power of the length of the index set. Moreover, in this situation it isunclear if there are enough realisations available in each subsample of a high-dimensional space.With respect to (3), in (CR-VCE) we are using the step function y : x 7→ E[Y |Im], x ∈ Im whichcan be rewritten by utilising the characteristic function of Im,
y(x) =∑m=1
1m(x)E[Y |Im], 1m(x) =
{1 if x ∈ Im,
0 if x 6∈ Im
For the estimation of Si via Correlation Ratio this yields
n∑j=1
(y(xij)− y)2 =n∑j=1
(∑m=1
1m(xij)E[Y |Im]− y
)2
=∑m=1
nm(E[Y |Im]− y)2.
In the original publication [9] the Correlation Ratio yηxi is defined as the square root of theSensitivity Index.
7
−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
Normalized Input2
Nor
mal
ized
Out
put
Non−Monotonic Model
DataPolyfit
Figure 2: Polynomial Fit: Computing the variance for the model-predicted output.
4.2 Polynomial Fit: If a linear model is not enoughAnother cheap approach consists of fitting a polynomial model of the given input data to thegiven output data (with hidden error term),
Y = β0 + β1Xi + β2X2i + β3X
3i + . . . βMX
Mi
Then we compute the goodness-of-fit
Si =
∑nj=1(y|pM (Xi)(xji)− y)2∑n
j=1(yj − y)2(FIT)
where y|pM (Xi)(·) is the model predicted output from a polynomial model in Xi with maximalpower M . This maximal power should be chosen large enough to capture sudden changes of theoutput. However, one has to consider the problem of over-fitting the data. The value in (FIT)already gives an estimate for the first order Sensitivity Index. Further details can be found in [7].In Figure 2 the polynomial fit with M = 10 is applied to the same set of data as in Figure 1.With respect to (3), the used model y is a polynomial y|pM (Xi)(x). This global polynomial fitclearly is of limited use when there are discontinuities in the output.When computing higher-order effects using polynomial regression we have to consider productsof powers of the input factors. A feasible way to handle this mass of monomials is to restrictthe sum of the powers of the individual factors by M (i.e, to prescribe a maximal length of the
8
0 0.2 0.4 0.6 0.8 10
1
2
3
4
5
6
7
8
9Non−monotonic scatterplot
Input
Out
put
dataconditional linear fit
Figure 3: Conditional Linear Fit: Piecewise linear model-predicted output.
associated multi-indices). As an example, the computation of ST1 for a k = 3 parameter modelwith M = 2 uses a polynomial fit of the form
Y = β00 + β10X2 + β01X3 + β20X22 + β11X2X3 + β02X
23 =
∑|α|≤M
βα ~Xα, ~X = [X2X3].
As the design matrices for the regression obtained by this method get very large we need a leastsquares algorithm that can cope with intermediate results which have close-to-singular precision.
4.3 Conditional linear fitInstead of fitting a polynomial to the data of the whole input parameter space as in Subsection 4.2,we can perform the model fitting conditioned on some suitable partition as in Subsection 4.1. Forlocal interpolation purposes, low order polynomials should suffice. In fact, we use linear models,see Figure 3. Compared to the previous figures, a slightly different function has been used tocreate the data. As the local approach allows for jump discontinuities, it also helps in handlingnon-continuous output. However, the computational effort of higher-order effects (in particular,total effects) is of polynomial order in the length of the factor group I .The TU Clausthal implementation of the conditional linear fit uses a fixed partition size of ` = 5for first order effects and of ` = 5k−1 for total effects unless otherwise noted. The subsamplesare determined by partitioning the ranked data into equally sized intervals.
9
4.4 Methods using special input samplingAsides from the direct computation of the Sensitivity Indices a large amount of algorithms havebeen developed that need special sampling schemes. Overviews of the available algorithms canbe found in [13, 16, 14]. These methods offer better estimates compared to the cheap methods.However, there are some drawbacks and pitfalls which one should be aware of when using thesealgorithms. We will report on this in Section 5.
4.4.1 Ishigami-Homma-Saltelli/Sobol´
These methods use two input samples, the basic sample X = (xji)j=1,...,n, i=1,...,k and the alter-native sample X′ = (x′ji)j=1,...,n, i=1,...,k with the associated output from the model evaluationsY = f(X) = (f(xj1, . . . , xjk))j=1,...,n and Y′ = f(X′) = (f(x′j1, . . . , x
′jk))j=1,...,n.
For each input factor i, a new sample Xi is created by replacing the ith column of X′ with thatof X. The first order Sensitivity Indices are computed by determining the correlation coefficientbetween the model output Yi = (f(x′j1, . . . , x
′j(i−1), xji, x
′j(i+1), . . . , x
′jk))j=1,...,n associated with
the input sample Xi and the basic model output Y, in matrix notation given by
Si = %(Yi,Y) =(Yi − Yi)
T(Y − Y)√(Yi − Yi)T(Yi − Yi)
√(Y − Y)T(Y − Y)
. (IHS)
There are many variants of this formula in use which exploit that the output variables Y , Y ′
and Yi have the same expectation. To obtain total effects, we compute the correlation coefficientbetween the modified model output Yi and the output Y′ from the alternative sample, ST i =1− %(Yi,Y
′). The total number of model evaluations is given by n(k+ 2) and provides us withfirst order effects and total effects. If there are dependencies between the input data then the rowinsertion has to take marginal probabilities into account.For the Sobol´ method, a special quasi-random sampling scheme named LPτ is used as inputsample generator which has favourite convergence properties for Monte-Carlo integration com-pared to simple random sampling. There are further constraints1 on the basic sample size and onthe maximal number of parameters when using LPτ .These methods have the drawback that they may produce negative values or, for total effects,values larger than 1. These are meaningless values as fractions of variance are estimated.
4.4.2 Fourier Amplitude Sensitivity Test/Random Balance Design
For Fourier Amplitude Sensitivity Tests (FAST) the parameter realisations are chosen along asearch curve with a special frequency behaviour. This introduces an artificial time-scale via theplacement of the realisations in the sample. A common choice for this sample is
xji = Gωi(sj + ri), ri ∈ [0, 1], sj = j/n, i = 1, . . . , k, j = 1, . . . , n,
Gω : R→ [0, 1], s 7→ 1/π arccos(cos(2πωs))(4)
1Due to number-theoretical reasons the performance is best for power-of-two sample sizes.
10
where ri is a random shift parameter and ωi ∈ N is an integer frequency assigned to the ith inputfactor. The frequency selection is handled by a special algorithm [3, 17, 4] to avoid frequencyinterference. As a vital constraint all the frequencies including higher harmonics2 up to a givenmaximum M have to stay below the Nyquist frequency, M
∑ωi < bn2 c. For small sets of
parameters the choice ωi = ωi−10 also works well. In this case, the basic frequency ω0 controls
the precision of the algorithm. Moreover, this frequency selection scheme also allows for thecomputation of higher order effects and total effects, see below.Note that with (4) the generated sample is quasi-uniformly [0, 1] distributed. If other distributionsare needed then the sample has to be modified by suitable transformations, e.g., via inversecumulative distribution functions. To keep this presentation short, we only consider uniform[0, 1] input distributions and therefore need no parameter transformations to other distributiontypes.After the frequencies are assigned to all input factors, the output is analysed for resonances usinga Fast Fourier Transformation, see Figure 4 for an illustration and also Section 5.1 for furtherdetails on this particular example. If the complex discrete Fourier coefficients of Y = (yj)j=1,...,n
are given by
cm =n∑j=1
yjζ(j−1)mn , ζn = e−
2πin , m = 0, . . . , n− 1, (5)
then the part of the output attributed to the frequency ωi (and hence to the input factor i) is foundin the set {cmωi |m = 1, . . . ,M} where the maximal higher harmonic M is usually 4 or 6. Thefraction of the variance attributed to the frequency ωi (and hence to the input factor i) is given by
Si = 2
∑Mm=1 |cmωi|2∑m6=0 |cm|2
. (FAST)
However, if the output depends non-continuously on input parameters then the quadratic conver-gence properties of the series in (FAST) are lost and higher values of M are required.Analogously to the cheap methods the formula can be derived from (3) by using a model predic-tion based on the frequency ωi giving a regression depending on the sampling sequence (sj),
Si =
∑nj=1(y|ωi(sj)− y)2∑n
j=1(yj − y)2, y|ω(s) =
1
n
(c0 + 2
M∑m=1
Re(cmωe
2πimωs))
, y = 1nc0.
The application of Parseval’s Theorem then yields (FAST).To compute total effects with FAST we use the frequency scheme ωi = ωi−1
0 where the basicfrequency ω0 = 2M + 1 is given by the maximal harmonic. Each frequency ω ≤M
∑k−1`=0 ω
`0 =
12(ωk0 − 1) in the Fourier spectrum of the output Y can be uniquely decomposed into
ω =k∑i=1
αi(ω)ωi−10 , αi(ω) ∈ {−M, 1−M, . . . ,−1, 0, 1, . . . ,M − 1,M}.
2The maximal harmonic M is also called interference factor.
11
0 200 400 600 800 1000 1200 1400 1600 1800 2000−4
−2
0
2
4Ishigami test function
Index
Inpu
ts
0 200 400 600 800 1000 1200 1400 1600 1800 2000−20
−10
0
10
20Ishigami test function
Index
Out
put
100 200 300 400 500 600 7000
20
40
Power Spectrum of Output
Frequency
%V
aria
nce
x1
x2
x3
Figure 4: FAST: Prescribed frequencies in the inputs, resonances in the output.
If αi(ω) 6= 0 then ω contributes to the total effect of input factor i. Hence to compute this valuewe can use two different approaches
ST i = 2
∑αi(ω)6=0 |cω|2∑m 6=0 |cm|2
, ST i = 1− 2
∑αi(ω)=0 |cω|2∑m6=0 |cm|2
, ω ∈{
1, . . . ,1
2(ωk0 − 1)
}.
For computing higher order effects the combined zero patterns of the αi(ω) can be exploited.In a different frequency selection scheme named “Extended FAST” (EFAST) a factor i of interestis assigned to a relative large frequency ωi � 1 and all others are assigned to low frequencies(say, ωj 6=i = 1). EFAST can be used to compute total effects as all frequencies below ω0 =ωi −M
∑j 6=i ωj do not contribute to the variance from factor i up to the M th order. Hence
ST i = 1− 2
∑ω0−1m=1 |cmωi |2∑m6=0 |cm|2
(EFAST)
Clearly, a new sample is needed for each of the factors. But this set-up also allows for thecomputation of the first order effect Si via (FAST). Moreover, is has a smaller memory footprintcompared to the full resolution FAST described above.The Random Balance Design (RBD) [20] uses only the frequency ω = 1 and generates a one-dimensional sample U with realisations uj = G1(sj) where s = (sj)j=1,...,n is equidistantlyspaced in [0, 1]. Then k random permutations πi : {1, . . . , n} → {1, . . . , n} are generated, andXi = πi(U) is a sample for the ith input parameter. To find the first order Sensitivity Indicesfor the ith factor, the output Y = f(X) = f(X1, . . . ,Xk) is sorted with respect to the inverse
12
of the ith permutation, π−1i (Y), and then analysed via a Fourier transformation using (5) with
yj replaced by π−1i (Y)j and (FAST). This method can only estimate first order effects. Higher
order effects can be estimated by introducing groups of different frequencies.
4.5 Effective Algorithm: Combining given data with Fourier amplitudeWe can modify the idea behind RBD so that it can be applied to given data: Instead of generatingrandom permutations, we construct permutations πi from the columns Xi of the given data matrixX such that each πi(Xi) has a zig-zag-like shape and therefore has a power spectrum where thefrequency ω = 1 is predominant. These permutations are obtained by sorting and shuffling theinput data. In particular, let x = (xj)j=1,...,n be a vector of realisations of the random variableXi. To keep the notation short, we drop the dependency on i. We order x = (xj) increasinglyand obtain an ordered vector (x(j)) with x(1) ≤ x(2) ≤ · · · ≤ x(n). Now, taking all odd indicesfrom (x(j)) in increasing order followed by all even indices in decreasing order gives us a vector(x[j]) with
x[j] =
{x(2j−1), j ≤ n+1
2,
x(2(n+1−j)), j > n+12,
j = 1, 2, . . . , n
that satisfies
x[j] ≤ x[j+1] if j ≤ n+12, x[j] ≥ x[j+1] if j > n+1
2.
This shows that the entries in the vector (x[j]) are in zig-zag order. There exists a permutationπi with πi((xj)) = (x[j]). As in RBD, this permutation is also applied to the output πi(Y).The Fourier transformation of the permuted output is analysed for frequency responses. Furtherdetails can be found in [10].This approach is called “Effective Algorithm” for the computation of SI (EASI). It has beendeveloped in the course of the PAMINA project. This benchmarking exercise is also used to testits performance.
5 The analytical benchmark casesA first round of PA benchmark studies were performed by the members of the PAMINA 2.1.Dwork group, see Appendix B. In order to unify the results and to draw more attention to thenonlinear SA indicators we asked in a second simulation round for selected benchmarks caseswith a prescribed setting. This setting consists of 25 runs at sample sizes of 100, 300, 1000, 3000,and 10000, computing mean, variance,R2,R2∗, and first and total order effects (where available).The choice of the SI algorithms was left to the participants of this second round. Contributionswere received from Facilia (Sweden), GRS Cologne (Germany), JRC Petten (The Netherlands),and TU Clausthal (Germany). In the following we sometimes mark the contributions of theseparticipants with the abbreviations FCL, GRS, JRC, and TUC, respectively.
13
Most of the following graphics are shown in form of box plots derived from the available 25 runsper sample size. The box plots show the lower and the upper quartile, the median value is markedwith a dot. The whiskers in the plots are lines illustrating the data range. Outliers are detectedusing three-times the inter-quartile range.
5.1 Ishigami function: A model with three input parameters and higherorder effects
The Ishigami test function is a three-parameter model. It is in so far interesting as the second andthird input factors have a Pearson Correlation Coefficient of zero. A variance-based SA retrievesa 44% first order effect for the second input factor, but the third input factor shows no first ordereffect. Only when estimating total effects, the third factor is attributed to 24% of the outputvariance.The Ishigami function is given by
Y = sinX1 + 7 sin2X2 + 0.1X43 sinX1
where Xi ∼ U(−π, π) are uniformly distributed in [−π, π]. The values of R2 ≈ R2∗ ≈ 20%imply that the results from a standard or rank-transformed linear regression are not very powerful.Hence we have to look into nonlinear SA methods.Figure 4 displays inputs and outputs from this model prepared for the FAST method. From theupper graphics showing the input we see that ω3 = 1, ω2 = 11 (by counting the peaks), andω1 = 112 = 121. Note that the scatter-plot of the ω1 input data shows Moire-patterns whichindicates that the sampling size, n = 2000 (Nyquist frequency 1000), is small compared to themaximal frequency in use 5(1 + 11 + 121) = 665. Considering the model output in the centerof Figure 4, we find a periodic behaviour which is not directly related to the input frequencies.The power spectrum of the output in the lower part of the figure shows more details. First ordereffects are coloured blue, second order effects green and third order effects are coloured red. Wefind noticeable first order effects for frequencies 44 (x4
2), 121 (x1), and 363 (x31). Second order
effects group around the first order effects of x1, for frequencies 119 and 123 (x1x23), 117 and
125 (x1x43), 361 and 365 (x3
1x23). A well-equipped eye might also spot frequencies 115 and 127
(x1x63), but as the maximal harmonic is M = 5 this part of the variance is mis-classified as third
order effect.As x2 enters as fourth power into the model and x3 enters the model only indirectly in combina-tion with x1, their influences are not detected by linear regression techniques.Let us now consider the performance of different algorithms for this example. Figures 5, 6 showthe results of the first order effects for x1 and x2, respectively. Nine different algorithms wereused. The TUC implementation of the FAST method errs on too few realisations for sizes 100and 300, EFAST only for sample size 100. For a FAST analysis of a k = 3 parameter modelwith a maximal harmonic M = 3 the TUC implementation needs at least 2M(1 + (2M + 1) +· · · + (2M + 1)k−1) = 6 · (1 + 7 + 49) = 342 realisations, for EFAST 2kM(2M + 1) = 126realisations.
14
100 300 1k 3k 10k
0
0.1
0.2
0.3
0.4
0.5
Fra
ctio
n of
Var
ianc
e
Sample Sizes
Ishigami S1 algorithm comparison
EASIFITCLMVCEECVRBDIHSFASTEFAST
Figure 5: TUC results – Box plots for S1.
100 300 1k 3k 10k
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Fra
ctio
n of
Var
ianc
e
Sample Sizes
Ishigami S2 algorithm comparison
EASIFITCLMVCEECVRBDIHSFASTEFAST
Figure 6: TUC results – Box plots for S2.
15
100 300 1k 3k 10k
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Fra
ctio
n of
Var
ianc
e
Sample Sizes
Ishigami S1 IHS methods
TUCFCLJRC
100 300 1k 3k 10k
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Fra
ctio
n of
Var
ianc
e
Sample Sizes
Ishigami S2 IHS methods
TUCFCLJRC
100 300 1k 3k 10k
−0.2
−0.1
0
0.1
0.2
0.3
Fra
ctio
n of
Var
ianc
e
Sample Sizes
Ishigami S3 IHS methods
TUCFCLJRC
Figure 7: Ishigami-Homma-Saltelli algorithm comparison.
The first five algorithms for each sample size are cheap methods working on the same data setgenerated with simple random sampling. Their performance is nearly the same. The indicatorsgenerated via the correlation ratio method which uses the mean of conditional variances (ECV)differ from those generated by calculating the variance of the conditional mean (VCE).The Ishigami-Homma-Saltelli (IHS) algorithm seems to be the only algorithm which producesunbiased estimates. But the sample size n is only the basic sample size for use in the IHS methodso that a total of (3 + 2)n = 5n model evaluations are needed. However, there is no explanationfor the wide variance compared to the cheap methods when estimating S1. An overview of theperformance of the different IHS implementations is found in Fig. 7. The TUC version uses aformula [18] that reduces the error introduced via cancellation, hence the true values should bebetter approximated. However, this theoretical result does not become apparent in the figure.The behaviour of the IHS algorithm changes drastically when using Sobol’s LPτ sequence aspseudo random number source, see Figure 11. Then even for small sample sizes good estimatesare computed. Here, the basic sample size is rounded to the next power of 2.The RBD method –although using an artificially generated sample– seems to have no better prop-erties than the cheap methods, see also Figure 9 where all Fourier-based methods are compared.A comparison of correlation ratio implementations from TUC, GRS-Cologne and JRC-Petten forall parameters is found in Figure 8. Note that the different correlation ratio implementations usesimple random sampling (ECV,VCE,SRS,CR2P,CR,CP5S), Latin hypercube sampling (LHS),and Latin hypercube sampling with the selection of the conditional mean in each subinterval(LHS-M). However, the use of different input sampling schemes does not produce significantlydifferent results. The CR2P, CR and CP5S methods study different subsample sizes: CR2P usesa two-interval partition, while CR5S requests a partition which is constructed in such way that
16
every subsample contains five realisations, and CR uses a subsample size resembling the rule-of-thumb ` = d
√ne. Here, the subsample-size-five setting overestimates the true values while
the two-intervals setting produces an underestimation, all other estimators produce consistentresults. For S3, the estimation of true zero values via CR methods is also difficult, only ECV andCR2P produce unbiased results.If FAST methods are available then they produce exact estimates for moderately-sized samples.For the computation of S2 via EFAST(TUC) strange things happen: The range of the computedestimates is not reduced by increasing the sample size. Maybe there are some resonances internalto the model that react to joint input frequencies ω1 = ω3 = 1. Figure 9 shows the comparisonof Fourier-based methods from TUC and Facilia. Facilia’s implementation of EFAST uses asimple frequency selection for sample size 100, hence now the same-sized box plot appears asfor the TUC implementation with sample sizes larger or equal to 300. For larger sample sizes,the Facilia version of EFAST uses a different frequency selection scheme utilising different smallfrequencies and produces much better results which are on par with FAST.The results for the total effects of x3 are presented in Figure 10. The number of available al-gorithms for the computation of total effects is smaller than the number of algorithms for thecomputation of main effects. The cheap estimators seem to be biased but consistent, IHS is un-biased but has large variations: even negative estimates are generated for sample size 100 (whichare clipped out from the graphics). The FAST methods produce estimates that are nearly unin-fluenced by the random frequency shift, there are only little differences in the performance of thedifferent implementations.For the Sobol’ method TUC used the next power of 2 for the sample size which produces goodestimates, while Facilia used the given value as basic sample size which produces less accurateresults for small sample sizes, see Figure 11. As the Sobol’ method uses a special samplingscheme there is only one estimate available per sampling size. The figure therefore shows allmain and total effects in one graphics. Note that S2 = ST2 so that two values are plotted on thesame spot.
5.2 A discontinuous switch exampleIf a computational model has input parameters that drastically change the output behaviour thenthese discontinuities may impose numerical problems for the used SA algorithms. Hence weanalyse the following test function,
Y =
{X2, if X1 >
12
−X2, if X1 ≤ 12
, X1, X2 ∼ U(0, 1).
The expected values are S1 = 34, S2 = 0, ST1 = 1, ST2 = 1
4. As R2 ≈ R2∗ ≈ 56% the results
from a standard or rank-transformed linear regression are only of limited use.Figure 12 shows the results of the estimation of S1 using the TUC computations. We concludethat in this case only the conditional linear model (CLM), the correlation ratio methods (VCE,ECV), the IHS algorithm, and EFAST can cope with the non-linearity, all other algorithms sys-tematically produce too low estimates. Using the correlation ratio methods, for all but the 3000
17
100 300 1k 3k 10k
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5F
ract
ion
of V
aria
nce
Sample Sizes
Ishigami S1 Correlation Ratios
ECVVCESRSLHSLHS−MCR2PCRCR5S
100 300 1k 3k 10k
0
0.1
0.2
0.3
0.4
0.5
0.6
Fra
ctio
n of
Var
ianc
e
Sample Sizes
Ishigami S2 Correlation Ratios
ECVVCESRSLHSLHS−MCR2PCRCR5S
100 300 1k 3k 10k
−0.1
−0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
Fra
ctio
n of
Var
ianc
e
Sample Sizes
Ishigami S3 Correlation Ratios
ECVVCESRSLHSLHS−MCR2PCRCR5S
Figure 8: Correlation Ratio methods for the Ishigami function.
18
100 300 1k 3k 10k
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7Fr
actio
n of
Var
ianc
e
Sample Sizes
Ishigami S2 Fourier−based methods
EASI(TUC)EASI(FCL)RBD(TUC)RBD(FCL)EFAST(TUC)EFAST(FCL)FAST(TUC)
Figure 9: Fourier-based methods for the Ishigami function.
100 300 1k 3k 10k0
0.1
0.2
0.3
0.4
0.5
0.6
Frac
tion
of V
aria
nce
Sample Sizes
Ishigami ST3
algorithm comparison
FITCLMIHS(TUC)IHS(FCL)IHS(JRC)FASTEFAST(TUC)EFAST(FCL)
Figure 10: Box plots for ST3.
sample size the number of subsamples in the partition given by the rule of thumb is even so thatthe discontinuity at x1 = 0.5 is resolved by the associated partition. One sees that the perfor-mance of the CR methods is slightly different for the 3000 sample size: While the estimates forthe other sampling sizes are nearly unbiased, the estimate for the size-3000 sample underesti-mates the true value. Figure 16 shows the results of other CR methods. For the computation
19
102
103
104
−0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
Sample Sizes
Frac
tion
of v
aria
nce
Ishigami Sobol’ methods TUC results
S1S2S3ST1ST2ST3
102
103
104
−0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Sample SizesFr
actio
n of
var
ianc
e
Ishigami Sobol’ methods Facilia results
S1S2S3ST1ST2ST3
Figure 11: First and total effects using Sobol’s method for the Ishigami function.
100 300 1k 3k 10k
0.65
0.7
0.75
0.8
0.85
0.9
Fra
ctio
n of
Var
ianc
e
Sample Sizes
Switch example S1 algorithm comparison
EASIFITCLMVCEECVRBDIHSFASTEFAST
Figure 12: TUC results – Box plots for S1.
20
of S1 via CR the use of Latin Hypercube sampling schemes is advantageous, but there are nodifferences in the results of S2 for varying sampling schemes (SRS,LHS,LHS-M). As alreadynoted in the previous example the CR estimates based on a subsample size of five realisations oron two intervals produce inferior results.Returning to Figure 12 the conditional linear model (CLM) uses a partition with an even numberof intervals, M = 6. Hence this method benefits from the same effect as the CR methods.The results for the TUC implementations of FAST and EFAST differ considerably. While EFASTconverges slowly FAST remains stubbornly at a low level. This is due to the fact that for thesample of size 10,000 EFAST uses the maximal harmonic M = 35 and frequency ω = 71while FAST is restricted to maximal harmonic M = 8 and frequencies ω1 = 17, ω2 = 1. Otherimplementations of Fourier-based methods suffer also from the fixed maximal harmonic M ,see Figure 15 which shows that the algorithms with the same value of M produce equivalentestimates.Figure 14 shows first and total effects estimated via IHS methods. The variations are large whencompared to other methods. However, unbiased estimates are produced. Compared to theseresults from the IHS methods, the Sobol’ algorithm gives almost the correct estimates even forsmall sample sizes, cf. Fig. 17. Even a sampling size which is not a power-of-2 produces noeye-catching effects.For a two-parameter model the total effects are given by ST i = 1 − S3−i, i = 1, 2. Thereforethe explicit calculation of total effects is not necessary. Nevertheless, Figure 13 shows computedtotal effects. Again, the polynomial fit and the FAST algorithm with bounded maximal frequencyfail to catch the exact value. Figure 17 shows that the total effects computed by the Sobol’ methodare in good agreement with the theoretical values.
5.3 A linear model with dependent input dataIn theory, independent input parameters are required for performing variance-based SA. It is notclear what happens with the SA algorithms in the presence of dependencies between the inputparameters or to what extend the results can be interpreted. This example highlights some of theproblems encountered when processing dependent data. The function under inspection is givenby the linear model Y = X1 + X2 where the input parameters have a joint probability densityfunction given by
p(X1, X2) =
{2 if 0 ≤ X1, X2 ≤ 0.5 or 0.5 ≤ X1, X2 ≤ 1,
0 otherwise.
The expected values are Si = V[E[Y |Xi]]V[Y ]
= 1314
= 0.9285714 . . . hence ST i = 1 − S3−i = 114
=0.0714285 . . . , i = 1, 2. In a linear model with independent parameters we would expect
(a) the sum of all main effects to be 1, and
(b) the total effects to be larger or equal to the main effects.However, we are not dealing with an independent input parameter setting. Figures 18 and 19show the results for the Sensitivity Indices S1 and S2. Since the model is symmetric in x1 and
21
100 300 1k 3k 10k
0.1
0.15
0.2
0.25
0.3
0.35
Fra
ctio
n of
Var
ianc
e
Sample Sizes
Switch ST2
algorithm comparison
FITCLMIHS(TUC)IHS(FCL)IHS(JRC)FASTEFAST(TUC)EFAST(FCL)
Figure 13: TUC results – Box plots for ST2.
100 300 1k 3k 10k
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
Fra
ctio
n of
Var
ianc
e
Sample Sizes
Switch S1 IHS methods
TUCFCLJRC
100 300 1k 3k 10k
−0.2
−0.15
−0.1
−0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
Fra
ctio
n of
Var
ianc
e
Sample Sizes
Switch S2 IHS methods
TUCFCLJRC
100 300 1k 3k 10k
0.1
0.15
0.2
0.25
0.3
0.35
Fra
ctio
n of
Var
ianc
e
Sample Sizes
Switch ST2
IHS methods
TUCFCLJRC
Figure 14: Ishigami-Homma-Saltelli algorithm comparison.
22
100 300 1k 3k 10k
0.64
0.66
0.68
0.7
0.72
0.74
0.76
0.78
0.8
0.82F
ract
ion
of V
aria
nce
Sample Sizes
Switch S1 Fourier−based methods
EASI(TUC)EASI(FCL)RBD(TUC)RBD(FCL)EFAST(TUC)EFAST(FCL)FAST(TUC)
Figure 15: Fourier-based methods for the Switch example.
x2 the results should also be the same. This is the case where the methods for calculation of theSensitivity Index require no additional information for parameter transformations, i.e., for thecheap methods. For these, the results show a fair agreement with the expected results, and sincethe model is linear the results are already valid for small sample sizes. The problems arise in caseof Sobol’, IHS, RBD, FAST or EFAST methods as the sample generation not only has to satisfythe needs of a special sampling scheme but also to realise the joint probability. Dependingon the input parameter transformation which may use the marginal distribution p(X1|X2) orp(X2|X1) we see different results, in this case the results of S2 are definitely wrong. However,with the careful choice of a parameter transformation one obtains the correct results, as theFacilia implementation of the IHS method and the GRS-Cologne implementations of the specialsampling scheme CR methods (SRS, LHS, LHS-M) show.Moreover, the CR methods, IHS and EFAST seem to be consistent, for the rest of the algorithmsa small bias seems to be present since an increase in the sample size does not lead to betterresults. Figure 20 shows a selection of CR methods.The IHS algorithm shows the largest variation. Again, when using the Sobol’ sequence for thesample generation the quality improves drastically (not shown). For total effects, the cheap meth-ods produce good estimates. For other methods which already gave bad estimates the “wrong”parameter transformation is now in effect for ST1 while ST2 is estimated correctly (not shown).
23
100 300 1k 3k 10k0.45
0.5
0.55
0.6
0.65
0.7
0.75
0.8
Fra
ctio
n of
Var
ianc
e
Sample Sizes
Switch S1 Correlation Ratios
ECVVCESRSLHSLHS−MCR2PCRCR5S
100 300 1k 3k 10k−0.1
0
0.1
0.2
0.3
0.4
Fra
ctio
n of
Var
ianc
e
Sample Sizes
Switch S2 Correlation Ratios
Figure 16: Correlation Ratio methods for the Switch example.
24
102
103
104
−0.2
0
0.2
0.4
0.6
0.8
1
1.2
Sample Sizes
Fra
ctio
n of
Var
ianc
eSwitch Sobol’ methods TUC results
102
103
104
−0.2
0
0.2
0.4
0.6
0.8
1
1.2
Sample Sizes
Fra
ctio
n of
Var
ianc
e
Switch Sobol’ methods Facilia results
S1S2ST1ST2
S1S2ST1ST2
Figure 17: Sobol’s method for the Switch example.
100 300 1k 3k 10k
0.7
0.8
0.9
1
1.1
1.2
1.3
Fra
ctio
n of
Var
ianc
e
Sample Sizes
Dependent Linear Model S1 algorithm comparison
EASI(TUC)EASI(FCL)FITCLMVCEECVRBDIHS(TUC)IHS(FCL)FASTEFAST
Figure 18: TUC and Facilia results – Box plots for S1.
25
100 300 1k 3k 10k0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Fra
ctio
n of
Var
ianc
e
Sample Sizes
Dependent Linear Model S2 algorithm comparison
EASI(TUC)EASI(FCL)FITCLMVCEECVRBDIHS(TUC)IHS(FCL)FASTEFAST
Figure 19: TUC and Facilia results – Box plots for S2.
100 300 1k 3k 10k
0.7
0.75
0.8
0.85
0.9
0.95
Fra
ctio
n of
Var
ianc
e
Sample Sizes
Dependent Linear Model S1 Correlation Ratios
ECVVCESRSLHSLHS−MCR2PCRCR5S
Figure 20: Correlation Ratio methods for the dependent linear model.
26
100 300 1k 3k 10k
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1F
ract
ion
of V
aria
nce
Sample Sizes
Sobol g S1 algorithm comparison
EASIFITCLMVCEECVRBDIHSWINDEFAST
Figure 21: TUC results – Box plots for S1.
5.4 Sobol’ g function: A model with eight input parametersReal-world models have many input parameters. Hence a test case where many input parametersare considered shows if an algorithm is robust enough. A well-studied test function is the non-monotonic Sobol’ g-function which is given by
Y =k∏i=1
|4Xi − 2|+ ai1 + ai
(6)
where k = 8, (ai) = (0, 1, 4.5, 9, 99, 99, 99, 99). The first parameter is most influential, theinfluence decreases through the rest of the parameters until parameters five to eight becomeequally uninfluential. Due to the symmetry in the formula, we have R2 = R2∗ = 0. Hence theresults based on linear regression are of no value for the Sensitivity Analysis.The results for S1 are reported in Figure 21. A full resolution FAST with k = 12 parame-ters needs more than 10,000 realisations so that there are no results available for this particularmethod. Instead, we feature a guest appearance of Jansen’s Winding Stairs algorithm. Its resultsshould be comparable to the IHS method (as both require the same number of model evaluations).Here, the performance of the Winding Stairs algorithm for S1 is slightly better than the resultsfrom IHS.The TUC EFAST implementation needs at least 2kM(2M + 1) ≥ 366, M ≥ 3, realisations towork. Hence the first two sample sizes allow no EFAST(TUC) analysis. For the other samplesizes, the simple frequency scheme of EFAST(TUC) is not well suited: Again, there is no con-vergence to the real value. The performance of other Fourier-based implementations is reportedin Figure 22.The results for S5 are reported in Figure 24. As the fifth parameter is un-influential its Sensi-tivity Index is close to zero. Note that for the first two sample sizes, EFAST produces an exact
27
100 300 1k 3k 10k
0
0.05
0.1
0.15
0.2
0.25
0.3
Fra
ctio
n of
Var
ianc
eSample Sizes
Sobol g S5 Fourier−based methods
EASI(TUC)EASI(FCL)RBD(TUC)RBD(FCL)EFAST(TUC)EFAST(FCL)
100 300 1k 3k 10k0
0.2
0.4
0.6
0.8
Fra
ctio
n of
Var
ianc
e
Sample Sizes
Sobol g S1 Fourier−based methods
EASI(TUC)EASI(FCL)RBD(TUC)RBD(FCL)EFAST(TUC)EFAST(FCL)
Figure 22: Fourier-based methods for Sobol’ g function.
100 300 1k 3k 10k
0.4
0.6
0.8
1
1.2
1.4
1.6
Fra
ctio
n of
Var
ianc
e
Sample Sizes
Sobol g S1 IHS methods
TUCFCLJRC
100 300 1k 3k 10k
−0.15
−0.1
−0.05
0
0.05
0.1
0.15
Fra
ctio
n of
Var
ianc
e
Sample Sizes
Sobol g S5 IHS methods
TUCFCLJRC
100 300 1k 3k 10k
0.4
0.5
0.6
0.7
0.8
0.9
1
1.1
Fra
ctio
n of
Var
ianc
e
Sample Sizes
Sobol g ST1
IHS methods
TUCFCLJRC
Figure 23: Ishigami-Homma-Saltelli algorithm comparison for Sobol’ g function.
28
100 300 1k 3k 10k
−0.2
−0.1
0
0.1
0.2
0.3F
ract
ion
of V
aria
nce
Sample Sizes
Sobol g S5 algorithm comparison
EASIFITCLMVCEECVRBDIHSWINDEFAST
Figure 24: TUC results – Box plots for S5.
zero as there are not enough realisations available. The cheap methods (asides ECV) and RBDoverestimate the exact value.
100 300 1k 3k 10k
−0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
Fra
ctio
n of
Var
ianc
e
Sample Sizes
Sobol g S5 Correlation Ratios
ECVVCESRSLHSLHS−MCR2PCRCR5S
Figure 25: Correlation Ratio methods for Sobol’ g function.
Compared to the IHS method that produces good estimates even for small sample sizes, theWinding Stairs implementation gives the worst estimates of S5 of all tested algorithms.Nearly all Correlation Ratio methods have problems identifying a close-to-zero Sensitivity Index,see Figure 25 where S5 is estimated.There are noticeable differences in the performance of the different implementations of the IHSmethod, see Figure 23. For the Sobol’ method no abnormalities can be spotted (not shown). The
29
estimates of the Sensitivity Indices for the uninfluential factors 5 to 8 are below 0.2% even forthe basic sample size of 128.
6 The PA case example: The GTM Level-E modelAfter discussing the analytical in the previous section we now draw our attention to a complexgeosphere transport model (GTM). In various publications (see [20], and [15] for a review),the PSACOIN Level E code [8] was used both as a benchmark of Monte Carlo simulationsand as a benchmark for Sensitivity Analysis methods. This computational model calculates theradiological dose rate to humans over geological time-scales due to the underground migrationof radionuclides from a hypothetical nuclear waste disposal site through a system of idealisednatural and engineered barriers. The model has a total of 33 parameters, 12 of which are takenas independent uncertain parameters. The distributions of these uncertainties are either uniformor log-uniform distributions, the parameters of which have been selected on the basis of expertjudgement. For a description of these uncertain input parameters see Table 1. These values aretaken from [12] where further information including a mathematical description of the GTMLevel-E model is available. The supplied binary model outputs the dose rate in Sv
awhich stem
from the radioactive Iodine-129 nuclide and the dose rate from the Neptunium-237 decay chain,moreover the maximum of the dose rates up to a given point (for I and Np) and the total doserate per time-step. The change from the Iodine decay to the Neptunium decay chain introducesnon-linearities into the model which are major obstacles for the Sensitivity Analysis. The Level-E model is also discussed in [1, Annex 1]. Here only the influence of Iodine is studied, theNeptunium decay chain is not considered.The issue of time-dependent results deserves some further attention. The Level-E benchmarktherefore provides sensitivity measures for the following entities,
• Peak of total dose rate,
• Time of occurrence of this peak, and
• Total dose rate (time-dependent, 25 time-steps equally distributed over a logarithmic scalefrom 104 to 106 years).
From the experience gained in analysing the analytical test cases, TUC decided to approach theSA problem by the following two paths. On the one hand, a simple random input sample ofsize 75, 000 × 12 was generated and and the associated model output was analysed using cheapmethods, allowing for an analysis of 25 samples of size 3000 each. On the other hand, a basicand an alternative input sample of size 4096×12 were generated using Sobol’s LPτ sequence andthe samples Xi were added to this input set, yielding an overall input sample size of 57, 344×12.Both methods allow for estimates using smaller sample sizes by picking suitable submatrices.Facilia computed first order and total effects using IHS and EFAST methods, and first ordereffects using RBD and EASI methods. For each method, 25 runs of sample sizes 100, 300 and1000 were computed.
30
Table 1: Uncertain input parameters for GTM Level-E.Parameter Description Distribution Range Unit1 T Containment time (source) uniform 102 . . . 103 a2 kI Leach rate for Iodine (source) log-uniform 10−3 . . . 10−2 1/a3 kC Leach rate for Np decay chain (source) log-uniform 10−6 . . . 10−5 1/a4 v1 Water velocity (1st layer) log-uniform 10−3 . . . 10−1 m/a5 l1 Length (1st layer) uniform 100 . . . 500 m6 R1
I Iodine retardation (1st layer) uniform 1 . . . 5 −7 γ1
C Np chain retardation multiplier (1st layer) uniform 3 . . . 30 −8 v2 Water velocity (2nd layer) log-uniform 10−2 . . . 10−1 m/a9 l2 Length (2nd layer) uniform 50 . . . 200 m
10 R2I Iodine retardation (2nd layer) uniform 1 . . . 5 −
11 γ2C Np chain retardation multiplier (2nd layer) uniform 3 . . . 30 −
12 W Stream flow rate (biosphere) log-uniform 105 . . . 107 m3/a
6.1 Peak of total dose rateThe peak of the total dose rate is not directly available as model output. Two simple approachescan be used, either by taking the maximum of the total dose (ignoring effects in-between time-steps) or by taking the maximum of the two peak doses “up to” the latest available time-step(ignoring effects which occur when Iodine as well as the decay chain both significantly contributeto the dose). There are cases where both values differ by a factor of over 5000 which suggestsnumerical problems with the model. For our analysis we have chosen the data from the firstapproach which seems numerically more stable.Figure 26 shows the results obtained with cheap methods when analysing the logarithm of thepeak dose rate. The ECV method immediately catches one’s eye as its bias seems to be minimalcompared to the other methods. The added value of a linear fit for the CLM method cannot standout against the VCE method.If the data were not log-transformed then the Sensitivity Analysis would attribute 15% of thevariance to v1 and 21% of variance to W (opposed to 37% and 46% shown in Fig. 26). Theuntransformed data were analysed by Facilia, see Figure 27 for an illustration. Here we see thatthe methods using special sampling schemes offer no advantage when compared to the cheapEASI method. Sometimes their performance is even worse. Moreover, 1000 realisations arenot enough to capture all the effects of the non-linearity in the model. There were small effectsvisible for the parameter R1
I in the log-transformed output data, for the untransformed outputdata this influence on the output has completely vanished.Let us now discuss the results obtained from the Sobol’ method. The LPτ sequence was takenfrom the GNU Scientific Library, http://www.gnu.org/software/gsl/. Figure 28 shows the re-sults for different basic sample sizes ranging from 100 to 4096. The linear connection betweenthe points is deceiving, there are nonlinear effects between the shown sample sizes. The most-influential parameters W and v1 are identified for small sample sizes. However, to fix a percent-age value the maximal basic sample size, 4096, is still too small.
31
100 300 1k 3k−0.05
0
0.05
0.1
0.15
0.2
Sample Sizes
Fra
ctio
n of
Var
ianc
e
log(peak dose) S1 − Parameter T
EASICLMVCEECV
100 300 1k 3k
−0.05
0
0.05
0.1
0.15
0.2
Sample Sizes
Fra
ctio
n of
Var
ianc
e
log(peak dose) S2 − Parameter k
I
EASICLMVCEECV
100 300 1k 3k
−0.05
0
0.05
0.1
0.15
0.2
Sample Sizes
Fra
ctio
n of
Var
ianc
e
log(peak dose) S3 − Parameter k
C
EASICLMVCEECV
100 300 1k 3k
0.2
0.3
0.4
0.5
0.6
Sample Sizes
Fra
ctio
n of
Var
ianc
e
log(peak dose) S4 − Parameter v1
100 300 1k 3k
0
0.1
0.2
Sample Sizes
Fra
ctio
n of
Var
ianc
e
log(peak dose) S5 − Parameter l1
100 300 1k 3k
0
0.1
0.2
0.3
Sample SizesF
ract
ion
of V
aria
nce
log(peak dose) S6 − Parameter R
I1
100 300 1k 3k
−0.05
0
0.05
0.1
0.15
0.2
Sample Sizes
Fra
ctio
n of
Var
ianc
e
log(peak dose) S7 − Parameter γ
C1
EASICLMVCEECV
100 300 1k 3k
0
0.1
0.2
Sample Sizes
Fra
ctio
n of
Var
ianc
e
log(peak dose) S8 − Parameter v2
EASICLMVCEECV
100 300 1k 3k
−0.05
0
0.05
0.1
0.15
0.2
Sample Sizes
Fra
ctio
n of
Var
ianc
e
log(peak dose) S9 − Parameter l2
EASICLMVCEECV
100 300 1k 3k
0
0.1
0.2
Sample Sizes
Fra
ctio
n of
Var
ianc
e
log(peak dose) S10
− Parameter RI2
100 300 1k 3k
−0.05
0
0.05
0.1
0.15
0.2
Sample Sizes
Fra
ctio
n of
Var
ianc
e
log(peak dose) S11
− Parameter γC2
100 300 1k 3k
0.3
0.4
0.5
0.6
Sample Sizes
Fra
ctio
n of
Var
ianc
e
log(peak dose) S12
− Parameter W
Figure 26: Cheap Sensitivity Analysis of log(peak dose rate).
32
100 300 1.000
0
0.1
0.2
0.3
0.4
0.5
Fra
ctio
n of
Var
ianc
e
Sample Sizes
LevelE S4 Facilia Results
EASIRBDEFASTIHS
100 300 1.000
−0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Fra
ctio
n of
Var
ianc
e
Sample Sizes
LevelE S6 Facilia Results
EASIRBDEFASTIHS
100 300 1.000
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Fra
ctio
n of
Var
ianc
e
Sample Sizes
LevelE S12
Facilia Results
EASIRBDEFASTIHS
Figure 27: Sensitivity Analysis of the peak dose rate, parameters v1, R1I , and W .
For total effects see Figure 29, the influence of W and v1 is also detected with a few 100 realisa-tions. In this example the parameter v2 produces large negative values.The total effects as computed by Facilia for the parameters v1, R1
I , v2 and W can be found inFigure 30. As for the Sobol’ method, we encounter problems with the IHS method. The totaleffects from the parameter v1 show a large negative outlier for sample size 300, and from theanalysis of the parameters v2 and W we encounter large negative values. The results of EFASTlook more promising: Their variation is small compared to the IHS methods and they seem toconverge for V1,v2 and W , while the results for R1
I show sudden changes between sample sizes300 and 1000.
6.2 Time of occurrence of the peakFor the determination of the time of occurrence of the maximal dose rate we are confronted withthe same problem as above, the data are not directly available in the output. For an analysis wehave chosen the time step where the maximal total dose rate is attained. Essentially, this makesthe peak time a discrete random variable which picks one out of the 25 specified time-steps.Figure 31 shows the Sobol’ indices of the peak rate depending on the sample size. They lookutterly uninformative. Maybe there is a cluster of slightly important variables v1, γ1
C , γ2C which
makes some sense as the velocity and the retardation multipliers should influence the occurrenceof the peak. The results of the cheap methods highlight this impression. Figure 32 shows theresults for the above-mentioned parameters, indeed showing a subtle, but noticeable influence.Facilia’s results are displayed in Figure 33. It can be seen that S7 (i.e., the sensitivity of γ1
C)is positive. The influence of the other two parameters is not so clearly visible for the maximalsample size 1000 as the IHS method has a large variation compared to the other methods whichhinders the decision if the Sensitivity Index is non-zero.
33
102
103
104
−0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
Sample Size
Fra
ctio
n of
Var
ianc
ePeak Dose Rate − First Order Sensitivity Indices − Sobol’ Method
Tk
I
kC
v1
l1
RI1
γC1
v2
l2
RI2
γC2
W
Figure 28: Sobol’ method for different sampling sizes, main effects.
The peak time covers orders of magnitude, so that one can suggest a logarithmic transformationof the time scale. The results obtained in this way differ substantially from the untransformedresults. For this, let us first comment on the histogram for the peak time, see Figure 34. We seetwo local maxima, one at 50,000 years and the other one at 500,000 years. While the first one isdue to the Iodine decay, the second one is due to the Neptunium decay chain.Without a logarithmic transformation of the time data only the “late” outliers feel the strengthof the Sensitivity Analysis, hence the SA qualifies the Neptunium decay chain retardation mul-tipliers which lead to late maxima as influential. With a logarithmic transformation the latemaxima move much more closely to the other maxima and more strength in the SA is given tothe “early” maxima. This is also visible in the results with respect to log-transformed time-dataas the following sensitivities are reported for the layer-1-velocity v1 ≈ 31%, the Iodine retarda-tion R1
I ≈ 7%, and the Np retardation multipliers γ1C ≈ 5% and γ2
C ≈ 2% (without illustration).Hence the analysis puts more emphasis on the “early” Iodine maxima.
6.3 Time-dependent total dose rateFor the peak dose rate we already identified v1 and W as the most influential parameters. Let usnow analyse how these influences change over time.Figure 35 shows a SA performed with EASI over all of the available 75,000 realisations. For therest of the section we will concentrate on the four parameters W , v1, v2 and γ1
C which are most
34
102
103
104
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
1.2
Sample Size
Fra
ctio
n of
Var
ianc
e
Peak Dose Rate − Total Sensitivity Indices − Sobol’ Method
Tk
I
kC
v1
l1
RI1
γC1
v2
l2
RI2
γC2
W
Figure 29: Sobol’ method for different sampling sizes, total effects.
100 300 1.000
−1.5
−1
−0.5
0
0.5
1
Fra
ctio
n of
Var
ianc
e
Sample Sizes
LevelE ST4
Facilia Results
IHSEFAST
100 300 1.000
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
Fra
ctio
n of
Var
ianc
e
Sample Sizes
LevelE ST6
Facilia Results
IHSEFAST
100 300 1.000−1.2
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
Fra
ctio
n of
Var
ianc
e
Sample Sizes
LevelE ST8
Facilia Results
IHSEFAST
100 300 1.000
−1.5
−1
−0.5
0
0.5
Fra
ctio
n of
Var
ianc
e
Sample Sizes
LevelE ST12
Facilia Results
IHSEFAST
Figure 30: Total effects for the peak dose rate, Facilia’s results.
35
102
103
104
−0.1
−0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
Sample Size
Fra
ctio
n of
Var
ianc
e
Peak Time − First Order Sensitivity Indices − Sobol’ Method
Tk
I
kC
v1
l1
RI1
γC1
v2
l2
RI2
γC2
W
Figure 31: Sobol’ method for different sampling sizes, main effects.
100 300 1k 3k
−0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
Fra
ctio
n of
Var
ianc
e
Sample Sizes
peak time S4 − Parameter v1
EASICLMVCEECV
100 300 1k 3k
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
Fra
ctio
n of
Var
ianc
e
Sample Sizes
peak time S7 − Parameter γ
C1
EASICLMVCEECV
100 300 1k 3k−0.05
0
0.05
0.1
0.15
0.2
0.25
Fra
ctio
n of
Var
ianc
e
Sample Sizes
peak time S11
− Parameter γC2
EASICLMVCEECV
Figure 32: Cheap Sensitivity Analysis of the time of the peak dose rate.
36
100 300 1.000
−0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Fra
ctio
n of
Var
ianc
e
Sample Sizes
LevelE S7 Facilia Results
EASIRBDEFASTIHS
100 300 1.000
0
0.2
0.4
0.6
0.8
Fra
ctio
n of
Var
ianc
e
Sample Sizes
LevelE S4 Facilia Results
EASIRBDEFASTIHS
100 300 1.000
−0.1
−0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
Fra
ctio
n of
Var
ianc
e
Sample Sizes
LevelE S11
Facilia Results
EASIRBDEFASTIHS
Figure 33: Sensitivity Analysis of the time of the peak dose rate, parameters v1, γ1C , and γ2
C .
4 4.5 5 5.5 6 6.5 70
1000
2000
3000
4000
5000
6000
7000Histogram for the time of occurrence of the peak dose rate
log10
(yr)
Figure 34: Histogram for the time of the peak dose rate.
37
104
105
106
107
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
time(yr)
Fra
ctio
n of
Var
ianc
e
First Order Sensitivity Indices − Total Dose Rate
Tk
I
kC
v1
l1
RI1
γC1
v2
l2
RI2
γC2
W
Figure 35: Time-dependent SA of the total dose rate based upon 75,000 realisations.
influential. Note that sum of the Sensitivity Indices is smaller than 1, hence there exist parameterinteractions which are not be captured by first order effects.The results from the Sobol’ algorithm with a basic sample size of 4096 can be found in Figure 36.Although the total amount of model runs is 4096 · (k + 2) = 57, 344, the results still show somenegative values, hence the precision of the sensitivity estimates is much worse than for thoseobtained via the EASI analysis of Figure 35 which uses a just 30% larger sample.Let us now consider an analysis based upon the 3000 realisations for a cheap method. Figure 37reports the time-dependent results, showing min and max (dotted lines), median (dashed lines)and mean (solid lines) from the 25 available runs for the four parameters of interest using a ECVcorrelation ratio method with 55 subsamples per partition. The means and the medians are nearlyindistinguishable, and the whole analysis looks sound.Last, but not least we have a look at the results from Facilia. Results for sample sizes of up to1000 realisations are available. We only show the statistics for the Sensitivity Index of parameterv1 based upon the 25 available runs of samples of 1000 realisations. Figures 38 and 39 showminimum, maximum, mean and median of the estimates obtained with four different methods.The Fourier-based methods more or less deliver the same results and parameter ranges are com-parable with those reported in Fig. 37, only IHS performs much worse (remember that IHS usesa basic sample size of 1000, hence the estimate is based upon 14,000 model evaluations).For the total dose rate, we do not discuss the results of the estimation of total effects.As this benchmark is mainly a test for the Sensitivity Analysis benchmark we do not try tointerpret the obtained Sensitivity Indices, and to enlighten the roles of the parameters involved inthe model, which would be the next step in a real world analysis. Such an interpretation wouldallow us to find answers to questions of the type we mentioned in the introduction.
38
104
105
106
107
−0.04
−0.02
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
Time(yr)
Fra
ctio
n of
Var
ianc
eLevel E Total Dose Rate − Sobol’ Method
W
v1
v2
γC1
Figure 36: Time-dependent main effects of the total dose rate with the Sobol’ method.
7 ConclusionsA lot of insight into the internals of variance-based Sensitivity Analysis has been gained duringthe course of this benchmark exercise. We collect and present the lessons learnt in a condensedform.First of all, we noted that for the standard algorithms the different implementations seem to bevery stable and produce results with only subtle differences. Moreover, results obtained withcheap methods are very much comparable to those obtained with more sophisticated methods.However there are some pitfalls which should be kept in mind when performing a variance-basedSA.
• Sobol’/IHS without special Monte-Carlo-integration sequence performs worse than a cheapmethod.
• Sobol’ LPτ without a sample size which is a power of 2 is sub-optimal for small samplesizes.
• For large number of parameters, Sobol’ LPτ needs a large number of realisations.
39
104
105
106
107
−0.02
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
Time(yr)
Fra
ctio
n of
Var
ianc
eMain effects of the total dose rate
W
v1
v2
γC1
Figure 37: Min, Max, Mean and Median of the main effects for the total dose rate.
104
105
106
107
0
0.05
0.1
0.15
0.2
0.25
Time(yr)
Fra
ctio
n of
Var
ianc
e
LevelE Total Dose Rate S4 ,v1, EASI method, 25 runs
104
105
106
107
0
0.05
0.1
0.15
0.2
0.25
Time(yr)
Fra
ctio
n of
Var
ianc
e
LevelE Total Dose Rate S4 ,v1, RBD method, 25 runs
minmaxmeanmedian
minmaxmeanmedian
Figure 38: Statistics of the main effects for the total dose rate, EASI and RBD methods.
• Algorithms with fixed maximal harmonic/numbers of subsamples do not capture disconti-nuities.
• Fourier-based methods and models with periodic output may have unwanted resonancesin the frequencies which render results useless. This may happen for EFAST and smallsample sizes, i.e., if a simple frequency selection scheme is in use.
• For CR methods, if jump discontinuities are not resolved by the choice of the partitionthen the results are sub-optimal. Moreover, the influence of the subsample size is not
40
104
105
106
107
0
0.05
0.1
0.15
0.2
0.25
Time(yr)
Fra
ctio
n of
Var
ianc
e
LevelE Total Dose Rate S4 ,v1, EFAST method, 25 runs
104
105
106
107
0
0.05
0.1
0.15
0.2
0.25
Time(yr)
Fra
ctio
n of
Var
ianc
e
LevelE Total Dose Rate S4 ,v1, IHS method, 25 runs
minmaxmeanmedian
minmaxmeanmedian
Figure 39: Statistics of the main effects for the total dose rate, EFAST and IHS methods.
neglectable.
• Random Balance Design shows no advantages when compared with cheap methods.
• For small Sensitivity Indices nearly all methods show bad convergence properties.
• For EFAST, one has the added value of computing total effects. But if a simulation runis already available then a cheap method will provide first order effects with no additionalsimulation costs.
There are still open problems related to SA and this benchmark exercise.
• Cheap methods can also deal with the estimation of total effects. However, one has to keepthe curse of dimensionality in mind when choosing subsample sizes.
• Cheap methods provide consistent results in situations with dependent input data. It isunclear how to interpret these results.
• The good performance of the ECV correlation ratio method (in combination with a rank-based partition) is currently not well understood.
• The effect of log -transforming the output data on the Sensitivity Indices is not studied indetail. It is clear that when taking the logarithm of a product there are parts of the variancewhich are transferred from higher order effects to main effects.
• These empirically distilled advices are currently not always backed up by theoretical re-sults.
41
AcknowledgementsThe authors thank all the participants of the PAMINA task 2.1.D for providing such a stimulatingworking environment, for their willingness to run the benchmark tests and to provide the data,and for their ideas and constructive remarks.
References[1] A. Badea and R. Bolado. Milestone M.2.1.D.4: Review of sensitivity analysis methods and
experience. Technical report, PAMINA Project, Sixth Framework Programme, EuropeanCommission, 2008. http://www.ip-pamina.eu/downloads/pamina.m2.1.d.4.pdf.
[2] R. Bolado and A. Badea. JRC’s contribution to the benchmark based on synthetic PA cases.Technical report, PAMINA Project, Sixth Framework Programme, European Commission,2008.
[3] R. Cukier, C. Fortuin, K. Shuler, A. Petschek, and J. Schaibly. Study of the sensitivity ofcoupled reaction systems to uncertainties in rate coefficients. I. Theory. J. Chem. Phys.,59:3873–3878, 1973.
[4] R. Cukier, J. Schaibly, and K. Shuler. Study of the sensitivity of coupled reaction systemsto uncertainties in rate cofficients. III. Analysis of the approximations. J. Chem. Phys.,63:1140–1149, 1975.
[5] P.-A. Ekstrom. Eikos – A Simulation Toolbox for Sensitivity Analysis.http://www.luthagen.org/ekstrom/docs/Eikos A Simulation toolbox for Sensitivity Ana-lysis.pdf, 2005.
[6] B. Krzykacz. SAMOS: A Computer Program for the Derivation of Empirical SensitivityMeasures of Results from Large Computer Models. Garching, Germany, 1990. ReportGRS-A-1700, Contract No. 73 708, 31 050.
[7] D. Levandowski, R. M. Cooke, and R. J. Duintjer Tebbens. Sample-based estimation of cor-relation ratio with polynomial approximation. ACM Transactions on Modeling and Com-puter Simulation, 18(1):3:1–3:16, 2007.
[8] Nuclear Energy Agency. PSACOIN level E intercomparison. Technical report, OECD,Paris, 1989.
[9] K. Pearson. On the General Theory of Skew Correlation and Non-linear Regession, vol-ume XIV of Mathematical Contributions to the Theory of Evolution. Drapers’ CompanyResearch Memoirs, Cambridge University Press, Cambridge, UK, 1905.
[10] E. Plischke. An effective algorithm for computing global sensitivity indices (EASI). Reli-ability Engineering&System Safety, 2009. Submitted Manuscript.
42
[11] E. Plischke and K.-J. Rohlig. Milestone M.2.1.D.3: Plan for benchmark, including speci-fication of synthetic PA cases. Technical report, PAMINA Project, Sixth Framework Pro-gramme, European Commission, 2008.
[12] P. Prado-Herrero. SimLab and GTM1. An External Model Example – The PSACOINLevel E. http://sensitivity-analysis.jrc.it/tutorial/SimLab%2520and%2520GTM1-TT1.pdf,2005.
[13] A. Saltelli, K. Chan, and E. Scott. Sensitivity Analysis. John Wiley&Sons, Chichester,2000.
[14] A. Saltelli, M. Ratto, T. Andres, F. Campolongo, J. Cariboni, D. Gatelli, M. Saisana, andS. Tarantola. Global Sensitivity Analysis – The Primer. John Wiley&Sons, Chichester,2008.
[15] A. Saltelli and S. Tarantola. On the relative importance of input factors in mathematicalmodels: Safety assessment for nuclear waste disposal. J. Am. Stat. Assoc., 97(459):702–709, 2002.
[16] A. Saltelli, S. Tarantola, F. Campolongo, and M. Ratto. Sensitivity Analysis in Practise – AGuide to Assessing Scientific Models. John Wiley&Sons, Chichester, 2004.
[17] J. Schaibly and K. Shuler. Study of the sensitivity of coupled reaction systems to uncer-tainties in rate coefficients. II. Applications. J. Chem. Phys., 59:3879–3888, 1973.
[18] I. Sobol´, S. Tarantola, D. Gatelli, S. Kucherenko, and W. Mauntz. Estimating the ap-proximation error when fixing unessential factors in global sensitivity analysis. ReliabilityEngineering&System Safety, 92:957–960, 2007.
[19] A. Stuart, K. Ord, and S. Arnold. Kendall’s advanced theory of statistics: Classical infer-ence and the linear model, volume 2A. John Wiley&Sons, Hoboken, NJ, 2009.
[20] S. Tarantola, D. Gatelli, and T. Mara. Random balance designs for the estimation of firstorder global sensitivity indices. Reliability Engineering&System Safety, 91:717–727, 2006.
43
A A short UA/SA implementationIn the course of the benchmark exercise, TU Clausthal developed a set of MATLAB scripts forperforming UA/SA. This section gives an overview of the available implementations of SA meth-ods. Some design decisions were made to keep the code simple, e.g., Fourier coefficients arecomputed via fft() and not by a direct calculation, partitions are accessed using a find(),and code snippets were re-used between the different methods.
Name Syntax DescriptionSUSI Si=susi(x,y) Main effects from given data (Correlation Ratio)EASI Si=easi(x,y) Main effects from given data (Fourier-based)FITSI Si=fitsi(x,y) Main effects from given data (polynomial fit)XFITSI STi=xfitsi(x,y) Total effects from given data (polynomial fit)CLMSI [Si,STi]=clmsi(x,y) Main and total effects from given data (local fit)IHSSI [Si,STi]=ihssi(k,n,m,t) Main and total effects (Ishigami-Homma-Saltelli)SOBOL [Si,STi]=sobol(k,n,m,t) Main and total effects (Sobol’ LPτ )JANSEN [Si,STi]=jansen(k,n,m,t) Main and total effects (Jansen Winding Stairs)XFAST [Si,STi]=xfast(k,n,m,t) Main and total effects (FAST)EFAST [Si,STi]=xfast(k,n,m,t) Main and total effects (EFAST)RBD Si=rbd(k,n,m,t) Main effects (RBD)
For methods using given data the standard syntax is Si=method(x,y) where x is a matrixof inputs and y is an output vector. For methods using a special sampling-scheme the standardsyntax is [Si,STi]=method(k,n,model,trafo) where model is the function underinspection, trafo is the transformation from the unit cube to the input distribution, k is thenumber of the model parameters and n is the number of the requested basic sample size. Theinternal program flow is given by u=quasirand(n,k); x=trafo(u); y=model(x);.The transformation offers no additional parameters to model different marginal distributions incase of dependent data.Some of the scripts offer further options which are documented in the online help.
B More benchmark resultsIn a first series of PA benchmarking we asked the participants of PAMINA 2.1.D for their resultson a number of benchmarking examples, see the Milestone report [11]. A diverse range of avail-able algorithms was in use, starting from linear regression over variance-based global SensitivityAnalysis to screening methods and statistical tests for performing Monte-Carlo Filtering.This appendix gathers the individual contributions of the partners. The following contributionshave been received.
• ANDRA (France): L. Loth, G. Pepin
The results of the analytical and threshold cases have been performed by Andra with theAlliances computing platform. Sensitivity Analysis indicators based on linear regressionwere calculated.
44
Table 2: Computational coverage of the first round of benchmark examplesNo. Name k Reference ANDRA FACILIA JRC TUC1 Linear model 3 [13, §2.9.1: 1] X X X X2 Linear model with interactions 2 [13, §2.9.1: 2] X X3 Linear Sobol’ function 22 [13, §2.9.1: 3] X X X4a Monotonic model 2 [13, §2.9.2: 4(a)] X X X X4b Monotonic model 2 [13, §2.9.2: 4(b)] X X4c Monotonic model 2 [13, §2.9.2: 4(c)] X X X X5a Exponential Sobol’ function 6 [13, §2.9.2: 5(a)] X X X5b Exponential Sobol’ function 20 [13, §2.9.2: 5(b)] X X X6a Quotient model 2 [13, §2.9.2: 6(a)] X X X X6b Quotient model 2 [13, §2.9.2: 6(b)] X X X X7 Sobol’ g function 8 [13, §2.9.3: 7] X X X X8 ** missing **9 Ishigami function 3 [13, §2.9.3: 9] X X X10 Morris function 20 [13, §2.9.3: 10] X X X11 Bungee jumping man 3 [16, §3.1] X X X X12a Distance of two spheres 6 [16, §3.5] X X X12b Distance of two spheres 6 [16, §3.5] X X X13a Smooth switch 2 [11] X X X13b Smooth switch 2 [11] X X X
• Facilia (Sweden): P.-A. Ekstrom
All computational work has been performed with Eikos[5], a simulation toolbox for Sen-sitivity Analysis written in MATLAB.
• JRC Petten (The Netherlands): A. Badea
The software in use was R (see http://cran.r-project.org/), a free software environment forstatistical computing and graphics. The functions needed for SA where provided by theadditional package “sensitivity”.
• TUC (Germany): E. Plischke
Algorithms for UA/SA were developed using MATLAB. As an alternative option, theSimLab 3.0 software (http://sensitivity-analysis.jrc.it/) was to be tested. Unfortunately,major problems were encountered which prevented its use for the benchmarking.
Table 2 lists the examples and their coverage by the participants. The analysis of some of themodels has been marked as optional for the participants, hence not necessarily all examples arecovered. Table 3 shows the applied UA/SA methods per participant. Some of the methods areonly applied to certain models. Furthermore, note that in this first round the cheap methods arenot covered.The benchmarks were used to gain knowledge of operating the UA/SA frameworks and to buildconfidence in the obtained results.
45
Table 3: UA/SA methods used by the participants for the first roundMethod ANDRA FACILIA JRC TUCMean X X X XVariance X X X XSkewness, Kurtosis XQuartiles, Min/Max XR2 X X X XR2∗ X X XPearson Correlation Coeff. X X X XSpearman Correlation Coeff. X X XPartial Correlation Coeff. X X X XPartial Rank Correlation Coeff. X X XStandard Regression Coeff. X X XStandard Rank Regression Coeff. X XSmirnov XSensitivity Indices (first order)FAST X XIHS X XEASI XSensitivity Indices (total order)FAST/EFAST X XIHS X XMorris OAT X X
The descriptions of the individual benchmark results are available in electronic form as sub-reports which form part of this milestone.
46
Benchmark Exercise Facilia’s Results
In the following, each benchmark model is discussed in a separate chapter. The
discussion is accompanied by diagrams that are automatically created during the
sensitivity analysis.
Model 1
Y = X1 + X2 + X3,
where Xi ~ U(3i-1
/2; 3i/2), i = 1, 2, 3.
The model is defined at page 34 in section 2.9.1 Linear Test Problems, Sensitivity
Analysis (Saltelli-Chan-Scott). All computation work has been performed using Eikos.
All sample sets have been drawn using a seed value of 0. The model is a straightforward
linear model for which all methods except PCC/PRCC handles very well. Rank-
transformation makes results worse.
Table 1 Summary statistics of the output Y for Model 1. Number of simulations=10000.
Mean Variance R2 R
2* Min Median Max
12.99 7.459 1 0.9959 6.872 13 19.34
6 8 10 12 14 16 18 200
10
20
30
40
50
60
70
80
90
100
Figure 1 Histogram of model output.
Table 2 Summary of standard sensitivity coefficients for Model 1. Number of simulations=10000. Smirnov
value has been computed using MCF on 95th percentile of output.
Factor CC RCC SRC SRRC PCC PRCC SMIR
X1 0.0897 0.0844 0.1059 0.1007 1 0.8431 0.1153
X2 0.3264 0.3109 0.3168 0.3012 1 0.9780 0.5422
X3 0.9422 0.9458 0.9415 0.9450 1 0.9977 0.8512
Table 3 Summary of variance based sensitivity coefficients, first order indices (Si). Sobol=5000
simulations (LpTau sampling). EFAST=1947 simulations.
Factor Sobol EFAST Analytic
X1 0.00777 0.01292 0.0110
X2 0.09628 0.09225 0.0989
X3 0.8830 0.8878 0.8901
Table 4 Summary of variance based sensitivity coefficients, total order indices (TSi). Sobol=5000
simulations (LpTau sampling). EFAST=1947 simulations.
Factor Sobol EFAST Analytic
X1 0.00542 0.01517 0.0110
X2 0.09391 0.09439 0.0989
X3 0.8954 0.8899 0.8901
Table 5 Summary of Morris indices, number of simulations=40, levels=4.
Factor Std Mean
X1 1.132e-15 1
X2 1.184e-15 3
X3 8.374e-16 9
0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.56
8
10
12
14
16
18
20
Y
X1
Figure 2 Scatter plot of 1000 data points of model output Y versus X1 with added regression line.
1.5 2 2.5 3 3.5 4 4.56
8
10
12
14
16
18
20
Y
X2
Figure 3 Scatter plot of 1000 data points of model output Y versus X2 with added regression line.
4 5 6 7 8 9 10 11 12 13 146
8
10
12
14
16
18
20
Y
X3
Figure 4 Scatter plot of 1000 data points of model output Y versus X3 with added regression line.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X1
Figure 5 Scatter plot of 1000 data points of model output Y versus X1 with added regression line, using
ranked data.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X2
Figure 6 Scatter plot of 1000 data points of model output Y versus X2 with added regression line, using ranked data.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X3
Figure 7 Scatter plot of 1000 data points of model output Y versus X1 with added regression line, using ranked data.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X1
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 8 Two-sample divisions of data according to the 95th percentile of the model output.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X2
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 9 Two-sample divisions of data according to the 95th percentile of the model output.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X3
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 10 Two-sample divisions of data according to the 95th percentile of the model output.
0 2 4 6 8 100
1
2
3
4
5
6
Estimated means ( )
Sta
ndard
Devia
tions (
)
X_1
X_2
X_3
Figure 11 Estimated means versus standard deviations of elementary effects.
Model 2
Y = X1 + X2,
with a correlation structure between X1 and X2. The joint probability distribution function
is
p(x1,x2) = 2 if 0<=x1,x2<=0.5 or 0.5<=x1,x2<=1, 0 elsewhere.
The model is defined at page 34-35 in section 2.9.1 Linear Test Problems, Sensitivity
Analysis (Saltelli-Chan-Scott). Computations have been performed using Eikos. All
samples have been drawn using a seed value of 0. The dependency between the two
factors has been induced onto the sample matrix. Therefore, only the standard sensitivity
measures have been computed except for the Sobol method which is probably erroneous
due to nonorthogonality. The first order effects compare well with the analytical values
but the total effects are totally erroneous. Rank-transformation makes results worse.
Table 1 Summary statistics of the output Y for Model 2. Number of simulations=10000.
Mean Variance R2 R
2* CC(X1,X2) Min Median Max
1.0028 0.2919 1.0000 0.9788 0.7524 0.0159 1.0698 1.9899
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 20
50
100
150
200
250
300
350
400
450
500
Figure 1 Histogram of model output.
Table 2 Summary of standard sensitivity coefficients for Model 2. Number of simulations=10000. Smirnov
value has been computed using MCF on 95th percentile of output.
Parameter CC RCC SRC SRRC PCC PRCC SMIR
X1 0.9364 0.9275 0.5353 0.5322 1.0000 0.9231 0.8397
X2 0.9358 0.9255 0.5330 0.5243 1.0000 0.9210 0.8405
Table 3 Summary of variance based sensitivity coefficients computed with Sobol method using 5000 runs
in total (MC sampling). Analytical results are the same for both Si and TSi due to an additive model.
Factor Si TSi Analytic
X1 0.9473 0.05562 0.9286
X2 0.9184 0.08174 0.9286
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
Y
X1
Figure 2 Scatter plot of 1000 data points of model output Y versus X1 with added regression
line.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2Y
X2
Figure 3 Scatter plot of 1000 data points of model output Y versus X2 with added regression line.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X2
X1
Figure 4 Scatter plot of 1000 data points of factor X1 versus X2 with added regression line.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X1
Figure 5 Scatter plot of 1000 data points of model output Y versus X1 with added regression line, using ranked data.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X2
Figure 6 Scatter plot of 1000 data points of model output Y versus X2 with added regression line, using ranked data.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X1
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 7 Two-sample divisions of data according to the 95th percentile of the model output.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X2
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 8 Two-sample divisions of data according to the 95th percentile of the model output.
Model 3
Y = sum(cj*(Xj-0.5)),
where k=22, Xj~U(0,1), and cj=(j-11)2.
The model is defined at page 35 in section 2.9.1 Linear Test Problems, Sensitivity
Analysis (Saltelli-Chan-Scott). Computations have been performed using Eikos. All
samples have been drawn using a seed value of 0. The main difficulty of this linear model
is the amount of uncertain parameters. All methods except PCC/PRCC handle the model
very well. Rank-transformation makes results worse. Eikos has limitation on the most 20
factors for using a LpTau sampling, therefore standard MC has been used with increased
number of samples, a total of 120000 runs.
Table 1 Summary statistics of the output Y for Model 3. Number of simulations=10000
Mean Variance R2 R
2* Min Median Max
0.2584 5.3949e+3 1.0000 0.9606 -270.3019 0.3533 294.8286
Figure 1 Histogram of model output.
Table 2 Summary of Sensitivity coefficients for Model 3. Number of simulations=10000. Smirnov value
has been computed using MCF on 95th percentile of output.
Factor CC RCC SRC SRRC PCC PRCC SMIR
X1 0.3896 0.3823 0.3927 0.3853 1 0.8890 0.3670
X2 0.3172 0.3103 0.3185 0.3115 1 0.8434 0.3040
X3 0.2450 0.2377 0.2510 0.2436 1 0.7753 0.2334
X4 0.1969 0.1923 0.1932 0.1886 1 0.6889 0.1878
X5 0.1386 0.1335 0.1412 0.1361 1 0.5657 0.1370
X6 0.0978 0.0944 0.0981 0.0947 1 0.4307 0.0907
X7 0.0596 0.0583 0.0630 0.0615 1 0.2962 0.0481
X8 0.0331 0.0331 0.0353 0.0353 1 0.1749 0.0414
X9 0.0181 0.0176 0.0157 0.0153 1 0.0766 0.0177
X10 5.842e-3 4.952e-3 3.925e-3 3.083e-3 1 0.0155 0.0201
X11 -1.928e-3 -1.668e-3 3.306e-17 2.294e-4 1.41e-10 1.156e-3 0.0106
X12 6.16e-4 5.026e-4 3.928e-3 3.716e-3 1 0.0187 0.0087
X13 0.0130 0.0130 0.0157 0.0157 1 0.0786 0.0136
X14 0.0373 0.0372 0.0354 0.0354 1 0.1754 0.0363
X15 0.0659 0.0639 0.0628 0.0609 1 0.2935 0.0700
X16 0.0921 0.0893 0.0982 0.0952 1 0.4325 0.0959
X17 0.1382 0.1343 0.1413 0.1373 1 0.5689 0.1391
X18 0.1884 0.1830 0.1925 0.1871 1 0.6859 0.1845
X19 0.2502 0.2432 0.2513 0.2442 1 0.7761 0.2457
X20 0.3160 0.3085 0.3182 0.3106 1 0.8427 0.2998
X21 0.3907 0.3847 0.3926 0.3866 1 0.8896 0.3709
X22 0.4732 0.4681 0.4748 0.4696 1 0.9211 0.4448
Table 3 Summary of variance based sensitivity coefficients, first order indices (Si). Sobol=120000
simulations (MC). EFAST=14278 simulations.
Factor Sobol EFAST Analytic
X1 0.13210 0.1222 0.1531
X2 0.07196 0.1024 0.1005
X3 0.04778 0.07893 0.0627
X4 0.01566 0.02378 0.0368
X5 -0.00485 0.01937 0.0198
X6 -0.01414 0.01883 0.0096
X7 -0.01956 0.003554 0.0039
X8 -0.02463 0.001253 0.0012
X9 -0.02475 0.000149 0.0002
X10 -0.02530 1.309e-005 0.0000
X11 -0.02520 1.208e-005 0
X12 -0.02535 1.647e-006 0.0000
X13 -0.02544 0.0001434 0.0002
X14 -0.02411 0.002538 0.0012
X15 -0.02053 0.002864 0.0039
X16 -0.01767 0.01212 0.0096
X17 -0.00432 0.01267 0.0198
X18 0.00940 0.08437 0.0368
X19 0.04192 0.04831 0.0627
X20 0.07879 0.09154 0.1005
X21 0.13000 0.1641 0.1531
X22 0.21280 0.2833 0.2242
Table 3 Summary of variance based sensitivity coefficients, total order indices (TSi). Sobol=120000
simulations (MC). EFAST=14278 simulations.
Factor TSi Sobol TSi EFAST Analytic
X1 0.1596 0.1228 0.1531
X2 0.115 0.1031 0.1005
X3 0.08521 0.0797 0.0627
X4 0.05512 0.02411 0.0368
X5 0.03517 0.01984 0.0198
X6 0.01906 0.01963 0.0096
X7 0.01839 0.004045 0.0039
X8 0.01542 0.001733 0.0012
X9 0.0146 0.000534 0.0002
X10 0.01369 0.0003538 0.0000
X11 0.0138 0.0005953 0
X12 0.01382 0.0004924 0.0000
X13 0.01369 0.0005059 0.0002
X14 0.0151 0.003349 0.0012
X15 0.0171 0.003212 0.0039
X16 0.0256 0.01268 0.0096
X17 0.03624 0.013 0.0198
X18 0.05622 0.08556 0.0368
X19 0.07577 0.04866 0.0627
X20 0.1071 0.09195 0.1005
X21 0.1768 0.165 0.1531
X22 0.2324 0.2846 0.2242
Table 4 Summary of Morris indices, number of simulations=2300, levels=12.
Factor Std Mean
X1 3.279e-014 100
X2 3.009e-014 81
X3 3.052e-014 64
X4 2.78e-014 49
X5 2.759e-014 36
X6 2.068e-014 25
X7 2.286e-014 16
X8 1.887e-014 9
X9 1.434e-014 4
X10 1.926e-014 1
X11 0 0
X12 1.795e-014 1
X13 1.332e-014 4
X14 1.352e-014 9
X15 1.813e-014 16
X16 1.938e-014 25
X17 1.7e-014 36
X18 1.816e-014 49
X19 2.138e-014 64
X20 1.979e-014 81
X21 1.922e-014 100
X22 2e-014 121
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1-250
-200
-150
-100
-50
0
50
100
150
200
250
Y
X1
Figure 2 Scatter plot of 1000 data points of model output Y versus X1 with added regression line.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X1
Figure 3 Scatter plot of 1000 data points of model output Y versus X1 with added regression line, using ranked data.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X1
Prior
Fn(x
i|Bhat) 5000
Fm
(xi|B) 95000
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X2
Prior
Fn(x
i|Bhat) 5000
Fm
(xi|B) 95000
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X3
Prior
Fn(x
i|Bhat) 5000
Fm
(xi|B) 95000
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X4
Prior
Fn(x
i|Bhat) 5000
Fm
(xi|B) 95000
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X5
Prior
Fn(x
i|Bhat) 5000
Fm
(xi|B) 95000
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X6
Prior
Fn(x
i|Bhat) 5000
Fm
(xi|B) 95000
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X7
Prior
Fn(x
i|Bhat) 5000
Fm
(xi|B) 95000
Figure 4 Two-sample divisions of data according to the 95th percentile of the model output.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
0
0.05
0.1
0.15
0.2
0.25
Analytic
Si EFAST
TSi EFAST
Si Sobol
TSi Sobol
Figure 5 Results from Variance-based methods together with analytical answer. Note that model is
additive, thus Si and TSi should be the same.
0 20 40 60 80 100 120 1400
5
10
15
20
25
30
Estimated means ( )
Sta
ndard
Devia
tions (
)
X[1,1]
X[2,1]
X[3,1]
X[4,1]
X[5,1]
X[6,1]
X[7,1]
X[8,1]
X[9,1]
X[10,1]
X[11,1]
X[12,1]
X[13,1]
X[14,1]
X[15,1]
X[16,1]
X[17,1]
X[18,1]
X[19,1]
X[20,1]
X[21,1]
X[22,1]
Figure 11 Estimated means versus standard deviations of elementary effects.
Model 4 configuration (a)
Y = X1 + X24,
where Xj~U(0,1)
The model is defined at page 36 in section 2.9.2 Monotonic Test Problems, Sensitivity
Analysis (Saltelli-Chan-Scott). Computations have been performed using Eikos. All
samples have been drawn using a seed value of 0. This model is slightly monotonic so
rank-transformation makes results little better. Smirnov two-sample test gives
erroneously parameter X2 more importance than parameter X1. Morris method tells us
that X1 has slightly greater mu* than X2 but X2 has a lot of interactions (with itself) that
X1 doesn’t have.
Table 1 Summary statistics of the output Y for Model 4 configuration a. Number of simulations=10000.
Mean Variance R2 R
2* Min Median Max
0.6996 0.1541 0.8861 0.89 3.98e-5 0.6894 1.972
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 20
100
200
300
400
500
600
700
800
Figure 1 Histogram of model output.
Table 2 Summary of sensitivity coefficients for Model 4. Number of simulations=10000. Smirnov value
has been computed using MCF on 95th percentile of output.
Parameter CC RCC SRC SRRC PCC PRCC SMIR
X1 0.7384 0.7631 0.733 0.758 0.9083 0.9161 0.5537
X2 0.5906 0.5616 0.5839 0.5547 0.8658 0.8582 0.8577
Table 3 Summary of variance based sensitivity coefficients, first order indices (Si). Sobol=4000
simulations (LpTau sampling). EFAST=1298 simulations.
Factor Sobol EFAST Analytic
X1 0.5350 0.5385 0.540
X2 0.4596 0.4437 0.460
Table 4 Summary of variance based sensitivity coefficients, total order indices (TSi). Sobol=4000
simulations (LpTau sampling). EFAST=1298 simulations.
Factor Sobol EFAST Analytic
X1 0.5458 0.5468 0.540
X2 0.4603 0.4503 0.460
Table 5 Summary of Morris indices, number of simulations=30, levels=4.
Factor Std Mean
X1 1.2820e-16 1
X2 0.6246 0.8889
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
Y
X1
Figure 2 Scatter plot of 1000 data points of model output Y versus X1 with added regression line.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
Y
X2
Figure 3 Scatter plot of 1000 data points of model output Y versus X2 with added regression line.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X1
Figure 4 Scatter plot of 1000 data points of model output Y versus X1 with added regression line, using
ranked data.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X2
Figure 5 Scatter plot of 1000 data points of model output Y versus X2 with added regression line, using
ranked data.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X1
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 6 Two-sample divisions of X1 data according to the 95th percentile of the model output.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X2
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 7 Two-sample divisions of X2 data according to the 95th percentile of the model output.
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Estimated means ( )
Sta
ndard
Devia
tions (
)
X_1
X_2
Figure 8 Estimated means versus standard deviations of elementary effects.
Model 4 configuration (b)
Y = X1 + X24,
where Xj~U(0,3)
The model is defined at page 36 in section 2.9.2 Monotonic Test Problems, Sensitivity
Analysis (Saltelli-Chan-Scott). Computations have been performed using Eikos. All
samples have been drawn using a seed value of 0. This model is very monotonic so rank-
transformation makes results very much better. Scatter plots and plots of MCF gives good
visual information of this model.
Table 1 Summary statistics of the output Y for Model 4 configuration b. Number of simulations=10000.
Mean Variance R2 R
2* Min Median Max
17.524 460.57 0.7500 0.9528 0.0002 6.5863 83.62
Figure 1 Histogram of model output.
Table 2 Summary of sensitivity coefficients for Model 2. Number of simulations=10000. Smirnov value
has been computed using MCF on 95th percentile of output.
Parameter CC RCC SRC SRRC PCC PRCC SMIR
X1 0.04269 0.1785 0.03477 0.1697 0.06936 0.6157 0.02611
X2 0.8653 0.9613 0.865 0.9597 0.8658 0.9753 0.99663
Table 3 Summary of variance based sensitivity coefficients, first order indices (Si). Sobol=4000
simulations (LpTau sampling). EFAST=1298 simulations.
Factor Sobol EFAST Analytic
X1 -0.0024 0.0016 0.0016
X2 1.0028 0.9808 0.9984
Table 4 Summary of variance based sensitivity coefficients, total order indices (TSi). Sobol=4000
simulations (LpTau sampling). EFAST=1298 simulations.
Factor Sobol EFAST Analytic
X1 0.0061 0.0175 0.0016
X2 1.0022 0.9955 0.9984
Table 5 Summary of Morris indices, number of simulations=30, levels=4.
Factor Std Mean
X1 0 3
X2 50.5964 72
0 0.5 1 1.5 2 2.5 30
10
20
30
40
50
60
70
80
90
Y
X1
Figure 2 Scatter plot of 1000 data points of model output Y versus X1 with added regression line.
0 0.5 1 1.5 2 2.5 3-20
0
20
40
60
80
100
Y
X2
Figure 3 Scatter plot of 1000 data points of model output Y versus X2 with added regression line.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X1
Figure 4 Scatter plot of 1000 data points of model output Y versus X1 with added regression line, using
ranked data.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X2
Figure 5 Scatter plot of 1000 data points of model output Y versus X2 with added regression line, using
ranked data.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X1
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 6 Two-sample divisions of X1 data according to the 95th percentile of the model output.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X2
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 7 Two-sample divisions of X2 data according to the 95th percentile of the model output.
0 10 20 30 40 50 60 70 800
10
20
30
40
50
60
Estimated means ( )
Sta
ndard
Devia
tions (
)
X_1
X_2
Figure 8 Estimated means versus standard deviations of elementary effects.
Model 4 configuration (c)
Y = X1 + X24,
where Xj~U(0,5)
The model is defined at page 36 in section 2.9.2 Monotonic Test Problems, Sensitivity
Analysis (Saltelli-Chan-Scott). Computations have been performed using Eikos. All
samples have been drawn using a seed value of 0. This model is very monotonic so rank-
transformation makes results very much better. Scatter plots and plots of MCF gives good
visual information of this model.
Table 1 Summary statistics of the output Y for Model 4 configuration c. Number of simulations=10000.
Mean Variance R2 R
2* Min Median Max
126.105 2.7374e4 0.7496 0.9819 6.020e-4 41.404 629.35
Figure 1 Histogram of model output.
Table 2 Summary of sensitivity coefficients for Model 4. Number of simulations=10000. Smirnov value
has been computed using MCF on 95th percentile of output.
Factor CC RCC SRC SRRC PCC PRCC SMIR
X1 0.011 0.08857 0.003072 0.07957 0.006139 0.5092 0.0416
X2 0.8658 0.9877 0.8657 0.987 0.8658 0.9908 0.9993
Table 3 Summary of variance based sensitivity coefficients, first order indices (Si). Sobol=4000
simulations (LpTau sampling). EFAST=1298 simulations.
Factor Sobol EFAST Analytic
X1 -0.0037 7.6458e-5 0.0001
X2 1.0043 0.9836 0.9999
Table 4 Summary of variance based sensitivity coefficients, total order indices (TSi). Sobol=4000 simulations (LpTau sampling). EFAST=1298 simulations.
Factor Sobol EFAST Analytic
X1 0.0045 0.0160 0.0001
X2 1.0037 0.9984 0.9999
Table 5 Summary of Morris indices, number of simulations=30, levels=4.
Factor Std Mean
X1 2.6204e-14 5.0000
X2 390.40 555.56
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50
100
200
300
400
500
600
700
Y
X1
Figure 2 Scatter plot of 1000 data points of model output Y versus X1 with added regression line.
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5-200
-100
0
100
200
300
400
500
600
700
Y
X2
Figure 3 Scatter plot of 1000 data points of model output Y versus X2 with added regression line.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X1
Figure 4 Scatter plot of 1000 data points of model output Y versus X1 with added regression line, using
ranked data.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X2
Figure 5 Scatter plot of 1000 data points of model output Y versus X2 with added regression line, using
ranked data.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X1
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 6 Two-sample divisions of X1 data according to the 95th percentile of the model output.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X2
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 7 Two-sample divisions of X2 data according to the 95th percentile of the model output.
0 100 200 300 400 500 6000
50
100
150
200
250
300
350
400
Estimated means ( )
Sta
ndard
Devia
tions (
)
X_1
X_2
Figure 8 Estimated means versus standard deviations of elementary effects.
Model 5 configuration (a)
Y = exp(sum(bj*Xj))-Ik,
where Ik=prod((ebj-1)/bj), Xj~U(0,5), k=6 and b1=1,b2=...b6=0.9
The model is defined at page 37 in section 2.9.2 Monotonic Test Problems, Sensitivity
Analysis (Saltelli-Chan-Scott). Computations have been performed using Eikos. All
samples have been drawn using a seed value of 0. In this configuration we have 6 factors
with the same importance to all factors except for the first one which has about three
times more importance. Model is monotonic. Most methods handle this model quite well.
The variance-based methods have problems with apportioning the correct amount of
variance to the factors. More model runs would increase the correctness. Scatter plots are
not so informative since the differences are not that big. Morris method nicely groups the
factors so you quickly identify graphically the most important factor X1.
Table 1 Summary statistics of the output Y for Model 5 configuration a. Number of simulations=10000.
Mean Variance R2 R
2* Min Median Max
0.1097 424.6 0.8018 0.9668 -24.18 -5.881 184.8
Figure 1 Histogram of model output.
Table 2 Summary of sensitivity coefficients for Model 5. Number of simulations=10000. Smirnov value
has been computed using MCF on 95th percentile of output.
Factor CC RCC SRC SRRC PCC PRCC SMIR
X1 0.5232 0.5916 0.5285 0.5973 0.7647 0.9564 0.5147
X2 0.3357 0.3600 0.3266 0.3497 0.5914 0.8866 0.3793
X3 0.3160 0.3432 0.3170 0.3446 0.5799 0.8838 0.3583
X4 0.3189 0.3466 0.3198 0.3474 0.5834 0.8854 0.3414
X5 0.3330 0.3678 0.3178 0.3514 0.5809 0.8876 0.3690
X6 0.3276 0.3452 0.3288 0.3464 0.5941 0.8849 0.3736
Table 3 Summary of variance based sensitivity coefficients, first order indices (Si). Sobol=8000
simulations (LpTau sampling). EFAST=3894 simulations.
Factor Sobol EFAST Analytic
X1 0.2790 0.3020 0.2870
X2 0.1024 0.0641 0.1057
X3 0.0911 0.1226 0.1057
X4 0.0899 0.0878 0.1057
X5 0.0954 0.1302 0.1057
X6 0.1111 0.0725 0.1057
Table 4 Summary of variance based sensitivity coefficients, total order indices (TSi). Sobol=8000
simulations (LpTau sampling). EFAST=3894 simulations.
Factor Sobol EFAST Analytic
X1 0.4143 0.4106 0.2870
X2 0.1659 0.1259 0.1057
X3 0.2176 0.1780 0.1057
X4 0.1911 0.1398 0.1057
X5 0.1869 0.1858 0.1057
X6 0.1803 0.1330 0.1057
Table 4 Summary of Morris indices, number of simulations=70.
Factor Std Mean
X1 20.99 46.27
X2 12.67 17.14
X3 15.92 23.01
X4 15.24 29.8
X5 12.68 25.33
X6 8.573 15.8
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1-40
-20
0
20
40
60
80
100
120
Y
X1
Figure 2 Scatter plot of 1000 data points of model output Y versus X1 with added regression line.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1-40
-20
0
20
40
60
80
100
120
Y
X2
Figure 3 Scatter plot of 1000 data points of model output Y versus X2 with added regression line.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X1
Figure 4 Scatter plot of 1000 data points of model output Y versus X1 with added regression line, using
ranked data.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X2
Figure 5 Scatter plot of 1000 data points of model output Y versus X2 with added regression line, using
ranked data.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X1
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 6 Two-sample divisions of X1 data according to the 95th percentile of the model output.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X2
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 7 Two-sample divisions of X2 data according to the 95th percentile of the model output.
0 10 20 30 40 500
5
10
15
20
25
30
35
Estimated means ( )
Sta
ndard
Devia
tions (
)
X_1
X_2
X_3
X_4
X_5
X_6
Figure 8 Estimated means versus standard deviations of elementary effects.
Model 5 configuration (b)
Y = exp(sum(bj*Xj))-Ik,
where Ik=prod((ebj-1)/bj), Xj~U(0,5), k=20 and b1…10=0.6,b11…20=...b6=0.4
The model is defined at page 37 in section 2.9.2 Monotonic Test Problems, Sensitivity
Analysis (Saltelli-Chan-Scott). Computations have been performed using Eikos. All
samples have been drawn using a seed value of 0. In this configuration we have 20
factors with the same importance to the first 10 factors and about half of this to the last 10
factors. Model is monotonic. The variance-based methods have big problems with this
model, the effects are so small. More model runs would increase the correctness. Scatter
plots are not so informative since the differences are not that big. Morris method groups
the factors so you quickly identify graphically the factors.
Table 1 Summary statistics of the output Y for Model 5 configuration a. Number of simulations=10000.
Mean Variance R2 R
2* Min Median Max
0.7361 1.836e+4 0.8065 0.9565 -170.5 -34.38 1536
Figure 1 Histogram of model output.
Table 2 Summary of sensitivity coefficients for Model 5. Number of simulations=10000. Smirnov value
has been computed using MCF on 95th percentile of output.
Factor CC RCC SRC SRRC PCC PRCC SMIR
X1 0.2315 0.2576 0.2336 0.2599 0.4687 0.7797 0.2223
X2 0.252 0.2727 0.2406 0.2601 0.4795 0.7798 0.2592
X3 0.221 0.2491 0.2259 0.2543 0.4564 0.7729 0.2408
X4 0.2335 0.2576 0.2339 0.258 0.4692 0.7774 0.2629
X5 0.2405 0.2615 0.2365 0.2569 0.4733 0.7761 0.2805
X6 0.2301 0.2514 0.2363 0.2576 0.4728 0.7768 0.2694
X7 0.2337 0.2484 0.2378 0.2527 0.4752 0.771 0.2689
X8 0.2278 0.2529 0.2345 0.2599 0.47 0.7796 0.2617
X9 0.2401 0.2622 0.2371 0.259 0.4743 0.7787 0.2687
X10 0.2381 0.257 0.2413 0.2605 0.4806 0.7803 0.2753
X11 0.1636 0.173 0.1626 0.172 0.3466 0.6359 0.1998
X12 0.1495 0.1642 0.1554 0.1706 0.3329 0.6329 0.1569
X13 0.1454 0.1569 0.1561 0.1684 0.3341 0.6278 0.1672
X14 0.1567 0.1711 0.1552 0.1698 0.3326 0.631 0.2061
X15 0.1679 0.1832 0.154 0.1683 0.3302 0.6276 0.1942
X16 0.1636 0.1742 0.1616 0.1721 0.3447 0.6362 0.1762
X17 0.1519 0.1583 0.1626 0.1703 0.3465 0.6321 0.1709
X18 0.1752 0.1868 0.1615 0.172 0.3445 0.6359 0.1981
X19 0.1664 0.1887 0.1504 0.1714 0.3232 0.6343 0.1783
X20 0.1573 0.1647 0.1621 0.17 0.3455 0.6314 0.2075
Table 3 Summary of variance based sensitivity coefficients, first order indices (Si). Sobol=22000
simulations (LpTau sampling). EFAST=12980 simulations.
Factor Sobol EFAST Analytic
X1 0.07785 0.04389 0.0562
X2 0.09271 0.1021 0.0562
X3 0.069 0.05247 0.0562
X4 0.09586 0.1225 0.0562
X5 0.09821 0.1014 0.0562
X6 0.0924 0.09013 0.0562
X7 0.08714 0.121 0.0562
X8 0.07325 0.07894 0.0562
X9 0.0822 0.04256 0.0562
X10 0.0716 0.06481 0.0562
X11 0.06041 0.03071 0.0250
X12 0.04223 0.03885 0.0250
X13 0.04669 0.03344 0.0250
X14 0.04966 0.03155 0.0250
X15 0.06909 0.02237 0.0250
X16 0.0352 0.02399 0.0250
X17 0.05236 0.007878 0.0250
X18 0.06527 0.02679 0.0250
X19 0.04434 0.03669 0.0250
X20 0.04449 0.03386 0.0250
Table 4 Summary of variance based sensitivity coefficients, total order indices (TSi). Sobol=22000
simulations (LpTau sampling). EFAST=12980 simulations.
Factor Sobol EFAST Analytic
X1 -0.04114 0.07493 0.0562
X2 0.005637 0.13 0.0562
X3 -0.008542 0.08298 0.0562
X4 0.01095 0.1495 0.0562
X5 0.004646 0.1272 0.0562
X6 -0.05076 0.1214 0.0562
X7 -0.03042 0.1462 0.0562
X8 -0.01061 0.1063 0.0562
X9 0.05229 0.07342 0.0562
X10 -0.009261 0.09293 0.0562
X11 -0.1512 0.045 0.0250
X12 -0.1224 0.05253 0.0250
X13 -0.13 0.04699 0.0250
X14 -0.03503 0.04575 0.0250
X15 -0.04956 0.03675 0.0250
X16 -0.1092 0.03829 0.0250
X17 -0.1287 0.02037 0.0250
X18 -0.08219 0.03845 0.0250
X19 -0.06665 0.04986 0.0250
X20 -0.05193 0.04867 0.0250
Table 4 Summary of Morris indices, number of simulations=840, levels=6.
Factor Std Mean
X1 78.75 106.7
X2 123.1 132.6
X3 80.91 104.3
X4 122.1 127.5
X5 101.9 113
X6 71.54 99.72
X7 96.5 117.8
X8 71.02 87.08
X9 70.58 106.7
X10 85.25 93.93
X11 77.98 75.37
X12 60.89 81.51
X13 71.12 74.3
X14 76.71 76.12
X15 67.07 77.14
X16 43.54 68.5
X17 62.17 71.75
X18 58.26 70.99
X19 47.33 60.31
X20 60.26 69.09
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1-200
0
200
400
600
800
1000
Y
X1
Figure 2 Scatter plot of 1000 data points of model output Y versus X1 with added regression line.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1-200
0
200
400
600
800
1000
Y
X2
Figure 3 Scatter plot of 1000 data points of model output Y versus X2 with added regression line.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X1
Figure 4 Scatter plot of 1000 data points of model output Y versus X1 with added regression line, using
ranked data.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X2
Figure 5 Scatter plot of 1000 data points of model output Y versus X2 with added regression line, using
ranked data.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X1
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 6 Two-sample divisions of X1 data according to the 95th percentile of the model output.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X2
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 7 Two-sample divisions of X2 data according to the 95th percentile of the model output.
0 20 40 60 80 100 120 1400
20
40
60
80
100
120
140
Estimated means ( )
Sta
ndard
Devia
tions (
)
X_1
X_2
X_3
X_4
X_5
X_6
X_7
X_8
X_9
X_10
X_11
X_12
X_13
X_14
X_15
X_16
X_17
X_18
X_19
X_20
Figure 8 Estimated means versus standard deviations of elementary effects.
Model 6 configuration (a)
Y = X24/X1
2,
where Xj~U(0.9,1.1).
The model is defined at pages 37-38 in section 2.9.2 Monotonic Test Problems,
Sensitivity Analysis (Saltelli-Chan-Scott). Computations have been performed using
Eikos. All samples have been drawn using a seed value of 0. Model is non-additive but
has very high R2 and R
2*. All methods deal easily with this model configuration.
Table 1 Summary statistics of the output Y for Model 6 configuration a. Number of simulations=10000.
Mean Variance R2 R
2* Min Median Max
1.027 0.06978 0.9836 0.99 0.5456 0.9991 1.791
0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 20
100
200
300
400
500
600
700
800
900
1000
Figure 1 Histogram of model output.
Table 2 Summary of sensitivity coefficients for Model 6. Number of simulations=10000. Smirnov value
has been computed using MCF on 95th percentile of output.
Parameter CC RCC SRC SRRC PCC PRCC SMIR
X1 -0.4436 -0.4196 -0.4517 -0.4278 -0.9622 -0.9738 0.664
X2 0.883 0.8983 0.8871 0.9022 0.9898 0.9939 0.8274
Table 3 Summary of variance based sensitivity coefficients, first order indices (Si). Sobol=4000
simulations (LpTau sampling). EFAST=1298 simulations.
Factor Sobol EFAST Analytic
X1 0.2007 0.2024 0.2023
X2 0.794 0.7884 0.7690
Table 4 Summary of variance based sensitivity coefficients, total order indices (TSi). Sobol=4000 simulations (LpTau sampling). EFAST=1298 simulations.
Factor Sobol EFAST Analytic
X1 0.2197 0.2155 -
X2 0.8038 0.7976 -
Table 5 Summary of Morris indices, number of simulations=70.
Factor Std Mean
X1 0.1098 -0.4488
X2 0.147 0.8726
0.92 0.94 0.96 0.98 1 1.02 1.04 1.06 1.08
0.6
0.8
1
1.2
1.4
1.6
Y
X1
Figure 2 Scatter plot of 1000 data points of model output Y versus X1 with added regression line.
0.92 0.94 0.96 0.98 1 1.02 1.04 1.06 1.08
0.6
0.8
1
1.2
1.4
1.6Y
X2
Figure 3 Scatter plot of 1000 data points of model output Y versus X2 with added regression line.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X1
Figure 4 Scatter plot of 1000 data points of model output Y versus X1 with added regression line, using
ranked data.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X2
Figure 5 Scatter plot of 1000 data points of model output Y versus X2 with added regression line, using
ranked data.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X1
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 6 Two-sample divisions of X1 data according to the 95th percentile of the model output.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X2
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 7 Two-sample divisions of X2 data according to the 95th percentile of the model output.
-0.5 0 0.5 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Estimated means ( )
Sta
ndard
Devia
tions (
)
X_1
X_2
Figure 8 Estimated means versus standard deviations of elementary effects.
Model 6 configuration (b)
Y = X24/X1
2,
where Xj~U(0.5,1.5).
The model is defined at pages 37-38 in section 2.9.2 Monotonic Test Problems,
Sensitivity Analysis (Saltelli-Chan-Scott). Computations have been performed using
Eikos. All samples have been drawn using a seed value of 0. Model is non-additive but
has very high R2*
, that is it is very monotonic. All methods deal with this model
configuration.
Table 1 Summary statistics of the output Y for Model 6 configuration b. Number of simulations=10000.
Mean Variance R2 R
2* Min Median Max
1.987 6.757 0.6742 0.9794 0.029 1.032 19.29
0 2 4 6 8 10 12 14 16 18 200
200
400
600
800
1000
1200
1400
1600
1800
2000
Figure 1 Histogram of model output.
Table 2 Summary of sensitivity coefficients for Model 6. Number of simulations=10000. Smirnov value
has been computed using MCF on 95th percentile of output.
Parameter CC RCC SRC SRRC PCC PRCC SMIR
X1 -0.4631 -0.4238 -0.4693 -0.432 -0.6351 -0.9491 0.7521
X2 0.6738 0.8904 0.6781 0.8944 0.765 0.9874 0.7442
Table 3 Summary of variance based sensitivity coefficients, first order indices (Si). Sobol=4000
simulations (LpTau sampling). EFAST=1298 simulations.
Factor Sobol EFAST Analytic
X1 0.2434 0.2562 0.2619
X2 0.523 0.5204 0.5110
Table 4 Summary of variance based sensitivity coefficients, total order indices (TSi). Sobol=4000
simulations (LpTau sampling). EFAST=1298 simulations.
Factor Sobol EFAST Analytic
X1 0.4969 0.492 -
X2 0.7398 0.7335 -
Table 5 Summary of Morris indices, number of simulations=70.
Factor Std Mean
X1 7.245 -6.068
X2 9.643 10.21
0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4
2
4
6
8
10
12
14
16
18
Y
X1
Figure 2 Scatter plot of 1000 data points of model output Y versus X1 with added regression line.
0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4
0
2
4
6
8
10
12
14
16
18Y
X2
Figure 3 Scatter plot of 1000 data points of model output Y versus X2 with added regression line.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X1
Figure 4 Scatter plot of 1000 data points of model output Y versus X1 with added regression line, using ranked data.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X2
Figure 5 Scatter plot of 1000 data points of model output Y versus X2 with added regression line, using ranked data.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X1
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 6 Two-sample divisions of X1 data according to the 95th percentile of the model output.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X2
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 7 Two-sample divisions of X2 data according to the 95th percentile of the model output.
-10 -5 0 5 10 150
1
2
3
4
5
6
7
8
9
10
Estimated means ( )
Sta
ndard
Devia
tions (
)
X_1
X_2
Figure 8 Estimated means versus standard deviations of elementary effects.
Model 7
Y = prod(gj*(Xj),
where gj(Xj)=(|4Xj-2|+aj)/(1+aj), a={0, 1, 4.5, 9, 99, 99, 99, 99}, Xj~U(0,1).
The model is defined at pages 39-40 in section 2.9.3 Non-Monotonic Test Problems,
Sensitivity Analysis (Saltelli-Chan-Scott). Computations have been performed using
Eikos. All samples have been drawn using a seed value of 0. This model is very non-
linear, both R2 and R
2* are extremely small. Results built upon regression are therefore
not to be interpreted. Variance-based methods deal with this type of model very good.
Smirnov Two-sample test also seems to rank the factors correctly as well as the Morris
method.
Table 1 Summary statistics of the output Y for Model 7. Number of simulations=10000.
Mean Variance R2 R
2* Min Median Max
0.9958 0.4669 0.000511 0.0004755 0.0001416 0.8911 3.536
0 0.5 1 1.5 2 2.5 3 3.5 40
100
200
300
400
500
600
Figure 1 Histogram of model output.
Table 2 Summary of sensitivity coefficients for Model 7. Number of simulations=10000. Smirnov value
has been computed using MCF on 95th percentile of output.
Factor CC RCC SRC SRRC PCC PRCC SMIR
X1 -0.00218 0.000752 -0.00257 0.000498 -0.00257 0.000498 0.3810
X2 0.000254 -0.00699 -5.67e-5 -0.00727 -5.66e-5 -0.00727 0.3367
X3 -0.00838 -0.00713 -0.00872 -0.00734 -0.00872 -0.00734 0.1578
X4 -0.000275 -2.61e-5 -0.000311 -0.000196 -0.000311 -0.000196 0.1072
X5 0.0132 0.0129 0.0132 0.0130 0.01320 0.0130 0.0358
X6 -0.000551 -0.00334 -0.0008 -0.00362 -0.000806 -0.00362 0.0492
X7 -0.0158 -0.0138 -0.0158 -0.0138 -0.0158 -0.0138 0.0299
X8 0.00298 -0.000929 0.00298 -0.000799 0.00298 -0.000799 0.0765
Table 3 Summary of variance based sensitivity coefficients, first order indices (Si). Sobol=10000
simulations. EFAST=5192 simulations.
Factor Sobol EFAST Analytic
X1 0.7144 0.7136 0.7165
X2 0.1683 0.1756 0.1791
X3 0.0102 0.01736 0.0237
X4 -0.008239 0.01048 0.0072
X5 -0.01474 0.0001053 0.0001
X6 -0.01437 9.857e-5 0.0001
X7 -0.0143 6.553e-5 0.0001
X8 -0.01421 9.606e-5 0.0001
Table 4 Summary of variance based sensitivity coefficients, total order indices (TSi). Sobol=10000
simulations. EFAST=5192 simulations.
Factor Sobol EFAST Analytic
X1 0.7963 0.7911 0.7871
X2 0.2432 0.2363 0.2420
X3 0.03468 0.0281 0.0340
X4 0.008831 0.01414 0.0105
X5 -0.002044 0.0002749 0.0001
X6 -0.002461 0.0002355 0.0001
X7 -0.002552 0.0001569 0.0001
X8 -0.00292 0.0002191 0.0001
Table 5 Summary of Morris indices, number of simulations=90, levels=4.
Factor Std Mean
X1 2.74 -1.535
X2 1.322 -0.09961
X3 0.8585 -0.04627
X4 0.4232 0.2206
X5 0.03717 -0.02193
X6 0.03636 -0.02019
X7 0.05075 0.002361
X8 0.04421 0.01367
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
0.5
1
1.5
2
2.5
3
3.5Y
X1
Figure 2 Scatter plot of 1000 data points of model output Y versus X1 with added regression line.
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
0.5
1
1.5
2
2.5
3
3.5
Y
X2
Figure 3 Scatter plot of 1000 data points of model output Y versus X2 with added regression line.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X1
Figure 4 Scatter plot of 1000 data points of model output Y versus X1 with added regression line, using
ranked data.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X2
Figure 5 Scatter plot of 1000 data points of model output Y versus X2 with added regression line, using
ranked data.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X1
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 6 Two-sample divisions of X1 data according to the 95th percentile of the model output.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X2
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 7 Two-sample divisions of X2 data according to the 95th percentile of the model output.
-2 -1.5 -1 -0.5 0 0.50
0.5
1
1.5
2
2.5
3
Estimated means ( )
Sta
ndard
Devia
tions (
)
X_1
X_2
X_3
X_4
X_5
X_6
X_7
X_8
Figure 8 Estimated means versus standard deviations of elementary effects.
Model 9
Y = sin(X1)+A*sin(X2)2+B*X3
4*sin(X1),
where Xj~U(-pi,pi), A=7 and B=0.1.
The model is defined at pages 31-43 in section 2.9.3 Non-Monotonic Test Problems,
Sensitivity Analysis (Saltelli-Chan-Scott). Computations have been performed using
Eikos. All samples have been drawn using a seed value of 0. Non-linear model with small
R2’s gives us nonsense for regression based methods. Variance-based methods work well.
Smirnov test identifies important factors, but the ranking is not correct. Morris method
gives us a very good interpretation of model behavior. Factor X3 has no first order
effects, that is, only higher order effects/interaction effects. This can be seen since it has
almost 0 mu* but a quite high sigma.
Table 1 Summary statistics of the output Y for Model 9. Number of simulations=10000.
Mean Variance R2 R
2* Min Median Max
3.482 13.62 0.1916 0.1918 -10.12 3.524 16.84
-15 -10 -5 0 5 10 15 200
20
40
60
80
100
120
140
160
180
Figure 1 Histogram of model output.
Table 2 Summary of sensitivity coefficients for Model 9. Number of simulations=10000. Smirnov value
has been computed using MCF on 95th percentile of output.
Parameter CC RCC SRC SRRC PCC PRCC SMIR
X1 0.4366 0.4373 0.4361 0.4369 0.4363 0.437 0.5782
X2 -0.0053 -0.0035 -0.0090 -0.0072 -0.010 -0.0081 0.1202
X3 -0.0393 -0.0308 -0.0304 -0.0219 -0.0338 -0.0244 0.3868
Table 3 Summary of variance based sensitivity coefficients, first order indices (Si). Sobol=5000
simulations (LpTau sampling). EFAST=1947 simulations.
Factor Sobol EFAST Analytic
X1 0.3123 0.3072 0.3139
X2 0.4262 0.4409 0.4424
X3 -0.007478 0.02863 0
Table 4 Summary of variance based sensitivity coefficients, total order indices (TSi). Sobol=5000
simulations (LpTau sampling). EFAST=1947 simulations.
Factor Sobol EFAST Analytic
X1 0.5906 0.5534 0.5574
X2 0.4356 0.4613 0.4442
X3 0.2404 0.2380 0.2410
Table 5 Summary of Morris indices, number of simulations=40, levels=4.
Factor Std Mean
X1 6.037 5.205
X2 8.133 -1.575
X3 8.332 2.12e-005
-3 -2 -1 0 1 2 3
-5
0
5
10
15
Y
X1
Figure 2 Scatter plot of 1000 data points of model output Y versus X1 with added regression line.
-3 -2 -1 0 1 2 3
-5
0
5
10
15
Y
X2
Figure 3 Scatter plot of 1000 data points of model output Y versus X2 with added regression line.
-3 -2 -1 0 1 2 3
-5
0
5
10
15
Y
X3
Figure 3 Scatter plot of 1000 data points of model output Y versus X3 with added regression line.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X1
Figure 4 Scatter plot of 1000 data points of model output Y versus X1 with added regression line, using
ranked data.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X2
Figure 5 Scatter plot of 1000 data points of model output Y versus X2 with added regression line, using
ranked data.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X3
Figure 5 Scatter plot of 1000 data points of model output Y versus X3 with added regression line, using
ranked data.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X1
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 6 Two-sample divisions of X1 data according to the 95th percentile of the model output.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X2
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 7 Two-sample divisions of X2 data according to the 95th percentile of the model output.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X3
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 7 Two-sample divisions of X3 data according to the 95th percentile of the model output.
-2 -1 0 1 2 3 4 5 60
1
2
3
4
5
6
7
8
9
Estimated means ( )
Sta
ndard
Devia
tions (
)
X_1
X_2
X_3
Figure 8 Estimated means versus standard deviations of elementary effects.
Model 10
Y = MorrisFunction(X),
where Xj~U(0,1), k=20. Function implementation is defined in the end of this document.
The model is defined at pages 33-44 in section 2.9.3 Non-Monotonic Test Problems,
Sensitivity Analysis (Saltelli-Chan-Scott). Computations have been performed using
Eikos. All samples have been drawn using a seed value of 0. This non-linear model has
20 uncertain factors. Some of them are important due to interactions (1…7), some are
important due to its high mean (8…10) and the rest is considered non-influential. The
interaction effects can be seen comparing total effects with first order effects with the
variance based methods. The high mean effects are noted in most methods. Morris
method captures all these categorizations within its graph.
Table 1 Summary statistics of the output Y for Model 10. Number of simulations=10000.
Mean Variance R2 R
2* Min Median Max
35.44 1077 0.4517 0.5046 -150.7 38.26 125.8
Figure 1 Histogram of model output.
Table 2 Summary of sensitivity coefficients for Model 10. Number of simulations=10000. Smirnov value
has been computed using MCF on 95th percentile of output.
Factor CC RCC SRC SRRC PCC PRCC SMIR
X1 -0.0838 -0.0719 -0.0803 -0.0683 -0.1077 -0.0965 0.1184
X2 -0.0971 -0.0845 -0.1120 -0.1003 -0.1493 -0.1409 0.1519
X3 0.1135 0.1229 0.1148 0.1237 0.1530 0.1730 0.172
X4 -0.0993 -0.0930 -0.0949 -0.0882 -0.1271 -0.1243 0.1431
X5 0.1080 0.1155 0.1111 0.1178 0.1483 0.1650 0.1763
X6 -0.0260 -0.0204 -0.0171 -0.0106 -0.0230 -0.0150 0.07042
X7 0.2293 0.2327 0.2297 0.2335 0.2960 0.3146 0.2096
X8 0.3196 0.3362 0.3226 0.3392 0.3991 0.4337 0.3814
X9 0.3838 0.4156 0.3881 0.4203 0.4640 0.5125 0.4703
X10 0.3036 0.3293 0.3024 0.3280 0.3777 0.4220 0.3464
X11 0.0073 0.0100 0.0169 0.0199 0.0228 0.0282 0.05695
X12 -0.0055 -0.0097 -0.0099 -0.0141 -0.0134 -0.0200 0.02021
X13 0.0006 0.0019 0.0137 0.0161 0.0185 0.0229 0.03579
X14 -0.0204 -0.0214 -0.0191 -0.0199 -0.0257 -0.0283 0.04874
X15 0.0182 0.0169 0.0157 0.0141 0.0212 0.0200 0.08084
X16 -0.0076 -0.0120 -0.0079 -0.0122 -0.0106 -0.0174 0.05011
X17 0.0188 0.0229 0.0143 0.0182 0.0193 0.0258 0.06579
X18 -0.0199 -0.0187 -0.0252 -0.0248 -0.0340 -0.0352 0.05147
X19 0.0210 0.0228 0.0206 0.0218 0.0278 0.0309 0.04937
X20 -0.0229 -0.0211 -0.0232 -0.0211 -0.0313 -0.0300 0.04105
Table 3 Summary of variance based sensitivity coefficients, first order indices (Si). Sobol=22000 simulations (LpTau sampling). EFAST=12980 simulations.
Factor Sobol EFAST
X1 0.03267 0.00345
X2 -0.01601 0.01344
X3 0.00518 0.01353
X4 -0.02023 0.00935
X5 0.03589 0.02672
X6 0.00230 0.00042
X7 0.07257 0.05939
X8 0.10660 0.07835
X9 0.16780 0.11290
X10 0.09804 0.10430
X11 -0.00488 0.00048
X12 0.00118 4.212e-5
X13 0.00167 0.00054
X14 0.00141 0.00014
X15 -0.00068 9.625e-5
X16 -0.00249 0.00093
X17 0.00132 0.00031
X18 0.00189 0.00012
X19 0.00169 0.00022
X20 -0.00203 0.00066
Table 4 Summary of variance based sensitivity coefficients, total order indices (TSi). Sobol=22000
simulations (LpTau sampling). EFAST=12980 simulations.
Factor Sobol EFAST
X1 0.22520 0.22710
X2 0.22010 0.30340
X3 0.03755 0.10170
X4 0.27040 0.23130
X5 0.07229 0.11950
X6 0.04702 0.08512
X7 0.00781 0.06487
X8 0.05718 0.08510
X9 0.09894 0.11990
X10 0.05547 0.11600
X11 -0.04042 0.00826
X12 -0.05569 0.01647
X13 -0.05089 0.00940
X14 -0.05313 0.00747
X15 -0.04901 0.01121
X16 -0.04432 0.01155
X17 -0.05576 0.00912
X18 -0.05298 0.01508
X19 -0.04870 0.00855
X20 -0.04843 0.00955
Table 5 Summary of Morris indices, number of simulations=2100, levels=12.
Factor Std Mean
X1 66.13 -7.465
X2 59.64 -0.5323
X3 48.65 12.34
X4 67.97 -14.48
X5 54.06 9.616
X6 43.23 0.2427
X7 25.92 30.16
X8 5.626 37.46
X9 5.46 42.24
X10 5.946 37.16
X11 5.142 0.2886
X12 4.943 -0.9773
X13 4.709 0.2822
X14 5.323 -0.1897
X15 5.854 0.8348
X16 5.521 -0.202
X17 5.597 0.5149
X18 4.932 -0.2251
X19 5.634 0.522
X20 5.807 -0.5002
-20 -10 0 10 20 30 40 500
10
20
30
40
50
60
70
Estimated means ( )
Sta
ndard
Devia
tions (
)
X_1
X_2
X_3
X_4
X_5
X_6
X_7
X_8
X_9
X_10
X_11
X_12
X_13
X_14
X_15
X_16
X_17
X_18
X_19
X_20
Figure 2 Estimated means versus standard deviations of elementary effects.
function Y = MorrisFunction(X) wX = w(X,1:20); % B0 Y = (-1)^0; % Sum Bi*wi Y = Y + 20*sum(wX(1:10)) + sum((-1).^(11:20).*wX(11:20)); % Sum Bij*wi*wj for i=1:20 Y = Y - 15*wX(i)*sum(wX(i+1:6)) + sum((-
1).^(i+(max(i+1,7):20)).*wX(i).*wX(max(i+1,7):20)); end % Sum Bijl*wi*wj*wl for i=1:3 for j=i+1:4 for l=j+1:5 Y = Y - 10*wX(i)*wX(j)*wX(l); end end end % Sum Bijls*wi*wj*wl*ws Y = Y + 5*wX(1)*wX(2)*wX(3)*wX(4);
function wsol = w(X,i) wsol = 1:length(i); for ii=1:length(i) switch i(ii) case {3,5,7} wsol(i(ii)) = 2*(1.1*X(i(ii))/(X(i(ii))+0.1)-0.5); otherwise wsol(i(ii)) = 2*(X(i(ii))-0.5); end end
Model 11
hmin = H - 2*M*g/(kel*sigma),
where H~U(40,60), M~U(67,74), g=9.8066, kel=1.5, sigma~U(20,40).
The model is defined at pages 63-66 in section 3.1 The jumping man. Applying variance-
based methods, Sensitivity Analysis in practice (Saltelli-Tarantola-Campolongo-Ratto).
Computations have been performed using Eikos. All samples have been drawn using a
seed value of 0. A very linear model handled by all methods. H is important due to its
high mean. Sigma is important due to its high mean plus some interaction effects. M is
not important.
Table 1 Summary statistics of the output hmin for Model 11. Number of simulations=10000.
Mean Variance R2 R
2* Min Median Max
18.12 73.49 0.982 0.9641 -7.431 18.34 37.71
-10 -5 0 5 10 15 20 25 30 35 400
10
20
30
40
50
60
Figure 1 Histogram of model output.
Table 2 Summary of sensitivity coefficients for Model 2. Number of simulations=10000. Smirnov value
has been computed using MCF on 95th percentile of output.
Factor CC RCC SRC SRRC PCC PRCC SMIR
H 0.662 0.6582 0.6778 0.6736 0.981 0.9626 0.786
M -0.0941 -0.0832 -0.107 -0.0961 -0.624 -0.4522 0.1437
sigma 0.716 0.7088 0.7308 0.7233 0.9836 0.9673 0.6828
Table 3 Summary of variance based sensitivity coefficients, first order indices (Si). Sobol=5000
simulations (LpTau sampling). EFAST=1947 simulations.
Factor Sobol EFAST Analytic
H 0.4388 0.4459 0.44
M 0.005239 0.0136 0.01
sigma 0.5451 0.5402 0.55
Table 4 Summary of variance based sensitivity coefficients, total order indices (TSi). Sobol=5000
simulations (LpTau sampling). EFAST=1947 simulations.
Factor Sobol EFAST
H 0.4412 0.449
M 0.01206 0.01707
sigma 0.5541 0.543
Table 5 Summary of Morris indices, number of simulations=30, levels=4.
Factor Std Mean
H 3.745e-015 20
M 0.9391 -3.02
sigma 4.581 19.99
40 42 44 46 48 50 52 54 56 58 60-10
-5
0
5
10
15
20
25
30
35
40
hm
in
H
Figure 2 Scatter plot of 1000 data points of model output hmin versus H with added regression line.
67 68 69 70 71 72 73 74-10
-5
0
5
10
15
20
25
30
35
40
hm
in
M
Figure 3 Scatter plot of 1000 data points of model output hmin versus M with added regression line.
67 68 69 70 71 72 73 74-10
-5
0
5
10
15
20
25
30
35
40h
min
M
Figure 3 Scatter plot of 1000 data points of model output hmin versus M with added regression line.
20 22 24 26 28 30 32 34 36 38 40-10
-5
0
5
10
15
20
25
30
35
40
hm
in
sigma
Figure 4 Scatter plot of 1000 data points of model output hmin versus sigma with added regression line.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
hm
in
H
Figure 5 Scatter plot of 1000 data points of model output hmin versus M with added regression line, using
ranked data.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000h
min
M
Figure 6 Scatter plot of 1000 data points of model output hmin versus M with added regression line, using
ranked data.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
hm
in
sigma
Figure 7 Scatter plot of 1000 data points of model output hmin versus sigma with added regression line, using ranked data.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
H
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 8 Two-sample divisions of H data according to the 95th percentile of the model output.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
M
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 9 Two-sample divisions of M data according to the 95th percentile of the model output.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
sigma
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 10 Two-sample divisions of sigma data according to the 95th percentile of the model output.
-5 0 5 10 15 200
2
4
6
8
10
12
14
Estimated means ( )
Sta
ndard
Devia
tions (
)
H
M
sigma
Figure 8 Estimated means versus standard deviations of elementary effects.
Model 12 configuration (a)
Y = (sqrt(X12+X2
2+X3
2)-R1)
2/A1+(sqrt(X4
2+X5
2+X6
2)-R2)
2/A2,
where Xj~N(0,0.35,-1,1), R1=R2=0.9 and A1=A2=0.001.
The model is defined at pages 83-85 in section 3.5 Two spheres. Applying variance based
methods in estimation/calibration problems, Sensitvity Analysis in practice (Saltelli-
Tarantola-Campolongo-Ratto). Computations have been performed using Eikos. All
samples have been drawn using a seed value of 0. In this configuration we have 6 factors.
Table 1 Summary statistics of the output Y for Model 12 configuration a. Number of simulations=10000.
Mean Variance R2 R
2* Min Median Max
348.4 4.79e+4 0.0002713 0.0003465 0.006724 320.6 1305
Figure 1 Histogram of model output.
Table 2 Summary of sensitivity coefficients for Model 12. Number of simulations=10000. Smirnov value
has been computed using MCF on 95th percentile of output.
Factor CC RCC SRC SRRC PCC PRCC SMIR
X1 0.001578 0.0029 0.001556 0.002973 0.001556 0.00297 0.2048
X2 0.0004473 0.001365 0.0005964 0.001528 0.000596 0.00153 0.2009
X3 0.0008035 0.005941 0.001028 0.006169 0.001027 0.00617 0.1873
X4 -0.008742 -0.00769 -0.008674 -0.00753 -0.00867 -0.00753 0.2046
X5 -0.01333 -0.01547 -0.01333 -0.01546 -0.01333 -0.01546 0.2105
X6 0.003757 0.001446 0.003752 0.001507 0.003752 0.00151 0.2077
Table 3 Summary of variance based sensitivity coefficients, first order indices (Si). Sobol=8000
simulations (LpTau sampling). EFAST=3894 simulations.
Factor Sobol EFAST
X1 0.1495 0.1755
X2 0.1567 0.1263
X3 0.1565 0.1471
X4 0.147 0.1704
X5 0.1431 0.1341
X6 0.1598 0.1137
Table 4 Summary of variance based sensitivity coefficients, total order indices (TSi). Sobol=8000
simulations (LpTau sampling). EFAST=3894 simulations.
Factor Sobol EFAST
X1 0.2104 0.2796
X2 0.207 0.1991
X3 0.1955 0.2369
X4 0.2101 0.275
X5 0.2186 0.216
X6 0.189 0.1781
Table 4 Summary of Morris indices, number of simulations=70, levels=4.
Factor Std Mean
X1 179.4 63.35
X2 120.3 45.38
X3 130.6 35.43
X4 175 78.95
X5 187.6 79.02
X6 132.5 62.15
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
200
400
600
800
1000
1200
Y
X1
Figure 2 Scatter plot of 1000 data points of model output Y versus X1 with added regression line.
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
200
400
600
800
1000
1200
Y
X2
Figure 3 Scatter plot of 1000 data points of model output Y versus X2 with added regression line.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000Y
X1
Figure 4 Scatter plot of 1000 data points of model output Y versus X1 with added regression line, using
ranked data.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X2
Figure 5 Scatter plot of 1000 data points of model output Y versus X2 with added regression line, using ranked data.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X1
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 6 Two-sample divisions of X1 data according to the 95th percentile of the model output.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X2
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 7 Two-sample divisions of X2 data according to the 95th percentile of the model output.
0 50 100 150 2000
20
40
60
80
100
120
140
Estimated means ( )
Sta
ndard
Devia
tions (
)
X_1
X_2
X_3
X_4
X_5
X_6
Figure 8 Estimated means versus standard deviations of elementary effects.
Model 12 configuration (b)
Y = (sqrt(X12+X2
2+X3
2)-R1)
2/A1+(sqrt(X4
2+X5
2+X6
2)-R2)
2/A2,
where Xj~U(-1,1), R1=R2=0.9 and A1=A2=0.001.
The model is defined at pages 83-85 in section 3.5 Two spheres. Applying variance based
methods in estimation/calibration problems, Sensitvity Analysis in practice (Saltelli-
Tarantola-Campolongo-Ratto). Computations have been performed using Eikos. All
samples have been drawn using a seed value of 0. In this configuration we have 6 factors.
Table 1 Summary statistics of the output Y for Model 12 configuration b. Number of simulations=10000.
Mean Variance R2 R
2* Min Median Max
161 1.995e+4 0.0003655 0.0005372 0.005969 122.2 1041
Figure 1 Histogram of model output.
Table 2 Summary of sensitivity coefficients for Model 12. Number of simulations=10000. Smirnov value
has been computed using MCF on 95th percentile of output.
Factor CC RCC SRC SRRC PCC PRCC SMIR
X1 -0.004908 -0.00266 -0.004934 -0.00277 -0.00493 -0.00277 0.06095
X2 0.0002512 -0.00243 0.000248 -0.00245 0.000248 -0.00245 0.09442
X3 0.002598 -0.00199 0.002657 -0.00183 0.002656 -0.00183 0.07884
X4 -0.01576 -0.01992 -0.01571 -0.01982 -0.01571 -0.01982 0.09674
X5 -0.002756 -0.00251 -0.002667 -0.00230 -0.00267 -0.00230 0.09042
X6 0.008958 0.01102 0.008845 0.01087 0.008846 0.01087 0.09632
Table 3 Summary of variance based sensitivity coefficients, first order indices (Si). Sobol=8000
simulations (LpTau sampling). EFAST=3894 simulations.
Factor Sobol EFAST
X1 0.1172 0.04211
X2 0.1455 0.03328
X3 0.1355 0.03093
X4 0.09679 0.04668
X5 0.09471 0.03223
X6 0.1058 0.05204
Table 4 Summary of variance based sensitivity coefficients, total order indices (TSi). Sobol=8000
simulations (LpTau sampling). EFAST=3894 simulations.
Factor Sobol EFAST
X1 0.2708 0.3745
X2 0.2751 0.3366
X3 0.2493 0.2717
X4 0.294 0.4369
X5 0.2759 0.2801
X6 0.2668 0.4757
Table 4 Summary of Morris indices, number of simulations=70, levels=4.
Factor Std Mean
X1 235.3 -255.6
X2 447.7 18.45
X3 357.5 -146.3
X4 462.4 -79.06
X5 483.1 -146.3
X6 379.9 -6.758
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
100
200
300
400
500
600
700
800
900
Y
X1
Figure 2 Scatter plot of 1000 data points of model output Y versus X1 with added regression line.
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
100
200
300
400
500
600
700
800
900
Y
X2
Figure 3 Scatter plot of 1000 data points of model output Y versus X2 with added regression line.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000Y
X1
Figure 4 Scatter plot of 1000 data points of model output Y versus X1 with added regression line, using
ranked data.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X2
Figure 5 Scatter plot of 1000 data points of model output Y versus X2 with added regression line, using ranked data.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X1
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 6 Two-sample divisions of X1 data according to the 95th percentile of the model output.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X2
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 7 Two-sample divisions of X2 data according to the 95th percentile of the model output.
-300 -250 -200 -150 -100 -50 0 500
50
100
150
200
250
300
350
400
450
500
Estimated means ( )
Sta
ndard
Devia
tions (
)
X_1
X_2
X_3
X_4
X_5
X_6
Figure 8 Estimated means versus standard deviations of elementary effects.
Model 13 configuration (a)
Y = (tanh(k*(X1-0.5))+s)*X2,
where Xj~U(0,1), k=50 and s = 1;
Computations have been performed using Eikos. All samples have been drawn using a
seed value of 0.
Table 1 Summary statistics of the output Y for Model 13 configuration a. Number of simulations=10000.
Mean Variance R2 R
2* Min Median Max
0.504 0.4018 0.6711 0.8347 0 0.09818 2
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 20
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
Figure 1 Histogram of model output.
Table 2 Summary of sensitivity coefficients for Model 13. Number of simulations=10000. Smirnov value
has been computed using MCF on 95th percentile of output.
Factor CC RCC SRC SRRC PCC PRCC SMIR
X1 0.6819 0.8711 0.6777 0.8686 0.7633 0.9057 0.5527
X2 0.4603 0.2834 0.4541 0.2755 0.6208 0.561 0.9395
Table 3 Summary of variance based sensitivity coefficients, first order indices (Si). Sobol=4000
simulations (LpTau sampling). EFAST=1298 simulations.
Factor Sobol EFAST Analytic
X1 0.5915 0.5556 0.59505
X2 0.2019 0.2101 0.20661
Table 4 Summary of variance based sensitivity coefficients, total order indices (TSi). Sobol=4000 simulations (LpTau sampling). EFAST=1298 simulations.
Factor Sobol EFAST Analytic
X1 0.7889 0.794 -
X2 0.412 0.4129 -
Table 5 Summary of Morris indices, number of simulations=30, levels=4.
Factor Std Mean
X1 0.9944 1.9
X2 0.8433 0.4
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1-0.5
0
0.5
1
1.5
2
Y
X1
Figure 2 Scatter plot of 1000 data points of model output Y versus X1 with added regression line.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1-0.5
0
0.5
1
1.5
2
Y
X2
Figure 3 Scatter plot of 1000 data points of model output Y versus X2 with added regression line.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X1
Figure 4 Scatter plot of 1000 data points of model output Y versus X1 with added regression line, using
ranked data.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X2
Figure 5 Scatter plot of 1000 data points of model output Y versus X2 with added regression line, using
ranked data.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X1
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 6 Two-sample divisions of X1 data according to the 95th percentile of the model output.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X2
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 7 Two-sample divisions of X2 data according to the 95th percentile of the model output.
0 0.5 1 1.5 20
0.2
0.4
0.6
0.8
1
1.2
1.4
Estimated means ( )
Sta
ndard
Devia
tions (
)
X_1
X_2
Figure 8 Estimated means versus standard deviations of elementary effects.
Model 13 configuration (b)
Y = (tanh(k*(X1-0.5))+s)*X2,
where Xj~U(0,1), k=50 and s = 0;
Computations have been performed using Eikos. All samples have been drawn using a
seed value of 0.
Table 1 Summary statistics of the output Y for Model 13 configuration b. Number of simulations=10000.
Mean Variance R2 R
2* Min Median Max
0.006249 0.3167 0.5827 0.5943 -0.9999 0.00747 1
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
50
100
150
200
250
300
Figure 1 Histogram of model output.
Table 2 Summary of sensitivity coefficients for Model 13. Number of simulations=10000. Smirnov value
has been computed using MCF on 95th percentile of output.
Factor CC RCC SRC SRRC PCC PRCC SMIR
X1 0.7634 0.7709 0.7634 0.7709 0.7633 0.7709 0.5587
X2 0.00603 0.01271 -0.0009592 0.005682 -0.001485 0.008921 0.938
Table 3 Summary of variance based sensitivity coefficients, first order indices (Si). Sobol=4000
simulations (LpTau sampling). EFAST=1298 simulations.
Factor Sobol EFAST Analytic
X1 0.7488 0.6995 0.75
X2 -0.004739 0.0002224 0
Table 4 Summary of variance based sensitivity coefficients, total order indices (TSi). Sobol=4000 simulations (LpTau sampling). EFAST=1298 simulations.
Factor Sobol EFAST Analytic
X1 0.9914 0.9998 -
X2 0.2525 0.25 -
Table 5 Summary of Morris indices, number of simulations=30, levels=4.
Factor Std Mean
X1 0.9944 1.9
X2 0.8433 -0.6
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Y
X1
Figure 2 Scatter plot of 1000 data points of model output Y versus X1 with added regression line.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Y
X2
Figure 3 Scatter plot of 1000 data points of model output Y versus X2 with added regression line.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X1
Figure 4 Scatter plot of 1000 data points of model output Y versus X1 with added regression line, using
ranked data.
0 100 200 300 400 500 600 700 800 900 10000
100
200
300
400
500
600
700
800
900
1000
Y
X2
Figure 5 Scatter plot of 1000 data points of model output Y versus X2 with added regression line, using
ranked data.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X1
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 6 Two-sample divisions of X1 data according to the 95th percentile of the model output.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
X2
Prior
Fn(x
i|Bhat) 500
Fm
(xi|B) 9500
Figure 7 Two-sample divisions of X2 data according to the 95th percentile of the model output.
-1 -0.5 0 0.5 1 1.5 20
0.2
0.4
0.6
0.8
1
1.2
1.4
Estimated means ( )
Sta
ndard
Devia
tions (
)
X_1
X_2
Figure 8 Estimated means versus standard deviations of elementary effects.
PSACOIN Level E
Far-field model for the migration of radionuclides in the geosphere. We are looking at the
total radiological dose over time as end-point.
Defined at pages 77-82 section 3.4, Sensitivity Analysis in Practice (Saltelli-Taranola-
Compolongo-Ratto).
All computation work except model evaluation has been performed using Eikos. All
sample sets have been drawn using a seed value of 0.
Evaluation of model has been done using a pre-compiled executable GTM_LE.exe. This
program reads an input file (LE-Fast-6000.sam) in Simlab sample-input format and
produces an output file (out_le_fast-6000.dat) in Simlab result-format. No modification
has been done to this executable. It seems to be restricted to 10000 runs. Therefore no
analyses herein have more runs. Model produces several outputs; we have chosen to
analyze the variable Dose_Total over the ten different time-points offered. There are 12
input factors in the model. Model seems to be robust. Model is non-linear and we would
like to be able to do more runs for the variance-based methods.
103
104
105
106
107
0
0.5
1
1.5
2
2.5
3
3.5
4
x 10-5
Time (yr)
Dose (
Sv/y
r)
Total Dose
Np-237
U-233
Th-229
I-129
103
104
105
106
107
0
1
2
3
4
5
6
7
x 10-7
Time (yr)
Dose (
Sv/y
r)
Total Dose
Np-237
U-233
Th-229
I-129
103
104
105
106
107
0
1
2
x 10-8
Time (yr)
Dose (
Sv/y
r)
I-129
103
104
105
106
107
0
2
4
6
x 10-11
Time (yr)
Dose (
Sv/y
r)
Np-237
U-233
Th-229
Figure 1 Dose over time for Fixed RUN 1, Fixed RUN 2, Fixed RUN 3 I-129 + Chain.
Table 1 Summary statistics of Total Dose for Level E. Number of simulations=10000.
Time Mean Variance R2 R
2* Min Median Max
1e+4 3.718e-8 8.081e-14 0.09556 0.4299 0 0 6.605e-6
2e+4 8.464e-8 1.178e-13 0.1413 0.5271 0 0 4.411e-6
5e+4 4.936e-8 2.371e-14 0.06331 0.1364 0 0 1.753e-6
1e+5 2.293e-8 5.534e-15 0.0607 0.06257 0 0 8.809e-7
2e+5 1.378e-8 2.428e-14 0.009287 0.1833 0 0 1.015e-5
5e+5 3.66e-8 1.517e-13 0.05012 0.117 0 0 1.511e-5
1e+6 5.545e-8 1.769e-13 0.09212 0.3065 0 0 1.554e-5
2e+6 7.315e-8 3.663e-13 0.07931 0.6369 0 0 2.449e-5
5e+6 6.574e-8 1.072e-13 0.1588 0.6746 0 0 6.639e-6
1e+7 1.848e-8 4.326e-15 0.1572 0.5227 0 0 8.144e-7
104
105
106
107
0
1
2
3
4
x 10-7
Time (yr)
Tota
l ra
dio
logic
al dose (
Sv/m
ol)
95%
mean
5%
Figure 2 Time-series plot of total radiological dose for probabilistic data.
104
105
106
107
00.010.020.030.040.05
x 10-5
Tota
l ra
dio
logic
al dose (
Sv/m
ol)
Time (yr)
Figure 3 Time-series plot of total radiological dose for probabilistic data.
104
105
106
107
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
Time (yr)
Corr
ela
tion c
oeff
icie
nt
Contim
ki
kc
v1
l1
Rl1
GamaCL1
v2
l2
Rl2
GamaCL2
W
Figure 4 Pearson Correlation coefficients over time.
104
105
106
107
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
Time (yr)
Ranked c
orr
ela
tion c
oeff
icie
nt
Contim
ki
kc
v1
l1
Rl1
GamaCL1
v2
l2
Rl2
GamaCL2
W
Figure 5 Spearman correlation coefficients over time.
104
105
106
107
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
Time (yr)
Regre
ssio
n c
oeff
icie
nt
Contim
ki
kc
v1
l1
Rl1
GamaCL1
v2
l2
Rl2
GamaCL2
W
Figure 6 Standardized regression coefficients over time.
104
105
106
107
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
Time (yr)
Ranked r
egre
ssio
n c
oeff
icie
nt
Contim
ki
kc
v1
l1
Rl1
GamaCL1
v2
l2
Rl2
GamaCL2
W
Figure 7 Ranked standardized regression coefficients over time.
104
105
106
107
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
Time (yr)
Part
ial corr
ela
tion c
oeff
icie
nt
Contim
ki
kc
v1
l1
Rl1
GamaCL1
v2
l2
Rl2
GamaCL2
W
Figure 8 Partial correlation coefficients over time.
104
105
106
107
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
Time (yr)
Ranked p
art
ial corr
ela
tion c
oeff
icie
nt
Contim
ki
kc
v1
l1
Rl1
GamaCL1
v2
l2
Rl2
GamaCL2
W
Figure 9 Ranked partial correlation coefficients over time.
104
105
106
107
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
Time (yr)
First
ord
er
index
Contim
ki
kc
v1
l1
Rl1
GamaCL1
v2
l2
Rl2
GamaCL2
W
Figure 10 First order effects computed with Extended FAST.
104
105
106
107
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Time (yr)
Tota
l ord
er
index
Contim
ki
kc
v1
l1
Rl1
GamaCL1
v2
l2
Rl2
GamaCL2
W
Figure 11 Total effects computed using Extended FAST.
104
105
106
107
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Time (yr)
First
ord
er
index
W
v1
GamaCL1
v2
l2
l1
Rl1
GamaCL2
Rl2
Contim
kc
ki
Figure 13 First order effects computed using Extended FAST.
104
105
106
107
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Time (yr)
Tota
l ord
er
index
GamaCL1
v1
l2
l1
GamaCL2
W
v2
Rl1
Contim
Rl2
kc
ki
Figure 14 Total effects computed using Extended FAST.
104
105
106
107
-0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
Time (yr)
First
ord
er
index
Contim
ki
kc
v1
l1
Rl1
GamaCL1
v2
l2
Rl2
GamaCL2
W
Figure 15 First order effects computed with Sobol method.
104
105
106
107
-3.5
-3
-2.5
-2
-1.5
-1
-0.5
0
0.5
1
Time (yr)
Tota
l ord
er
index
Contim
ki
kc
v1
l1
Rl1
GamaCL1
v2
l2
Rl2
GamaCL2
W
Figure 16 Total effects computed using Sobol method.
104
105
106
107
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Time (yr)
First
ord
er
index
l1
v1
W
Rl1
GamaCL1
v2
l2
GamaCL2
Rl2
Contim
ki
kc
Figure 17 First order effects computed using Sobol method.
104
105
106
107
0
1
2
3
4
5
6
Time (yr)
Tota
l ord
er
index
v1
l1
l2
W
kc
Rl1
v2
GamaCL1
GamaCL2
Rl2
Contim
ki
Figure 18 Total effects computed using Sobol method.
Proposal/Contract no.: FP6-036404
Project acronym: PAMINA
Project title: PERFORMANCE ASSESSMENT METHODOLOGIES
IN APPLICATION TO GUIDE THE DEVELOPMENT
OF THE SAFETY CASE
Instrument: Integrated Project
Thematic Priority: Management of Radioactive Waste and Radiation
Protection and other activities in the field of
Nuclear Technologies and Safety
Milestone M 2.1.D.2
Benchmark exercise - “Analytical and threshold cases” part
Andra’s results
Due date of deliverable: 15.09.08
Actual submission date: 05.09.08
Start date of project: 01.10.2006
Duration: 36 months
Agence Nationale pour la gestion des Déchets RAdioactifs (ANDRA)
Revision: 1
Project co-funded by the European Commission within the Sixth Framework Programme (2002-2006)
Dissemination level
PU Public
PP Restricted to other programme participants (including the Commission Services) x
RE Restricted to a group specified by the consortium (including the Commission Services)
CO Confidential, only for members of the consortium (including the Commission Services)
PAMINA Sixth Framework programme, 10.09.2008 2
Benchmark exercise - “Analytical and threshold cases” part
Andra’s results
ANDRA : L. LOTH, G. PEPIN
PAMINA Sixth Framework programme, 10.09.2008 3
CONTENTS
1 CONTEXT/OBJECTIVES ........................................................................................................... 4
2 FRAMEWORK ........................................................................................................................... 5
2.1 CASES PERFORMED BY ANDRA .................................................................................. 5
2.2 SHORT DESCRIPTION OF SENSIITIVTY MODULE OF ALLIANCES PLATFORM ..................... 5
2.2.1 Generation ....................................................................................................... 6
2.2.2 Launch ............................................................................................................ 7
2.2.3 Analysis ........................................................................................................... 7
2.2.4 Post-processing ............................................................................................... 7
3 RESULTS ................................................................................................................................... 9
3.1 TEST-CASE 1: MODEL 1 ............................................................................................. 9
3.2 TEST-CASE 2: MODEL 4 CONFIGURATION A) .............................................................. 10
3.3 TEST-CASE 3: MODEL 4 CONFIGURATION C) .............................................................. 11
3.4 TEST-CASE 4: MODEL 6 CONFIGURATION A) .............................................................. 11
3.5 TEST-CASE 5: MODEL 7 ........................................................................................... 12
3.6 TEST-CASE 6: MODEL 9 ........................................................................................... 13
3.7 TEST-CASE 7: MODEL 10 ......................................................................................... 14
3.8 TEST-CASE 8: MODEL TARANTOLA – COMPOLONGO – RATTO SECTION 3.1
JUMPING MAN APPLIED TO VARIANCE-BASED METHODS ............................................. 14
3.9 TEST-CASE 10: MODEL 6 CONFIGURATION B) ............................................................ 16
PAMINA Sixth Framework programme, 10.09.2008 4
1 Context/Objectives
This note deals with PAMINA project, RTDC-2, WP2.1D, milestone M2.1D.2. It gives results
of analytical and threshold cases, defined at JRC Petten’s metting January 15-16th and which
have been performed by Andra with Alliances comuting platform.
Andra is also involved in RTDC4, WP4.3 whose main aim is to compare methods and tools
to treat uncertainties and carry out sensitivity analysis, applied to “realistic” test-cases from
french clay site in the context of performance assessment.
In the scope of WP21D, focusing on techniques for sensitivity and uncertainty analysis, these
calculations fulfill two mains objectives inthe field of Andra :
- to be involved in an international benchmark of method and tools, whose issues are
code debugging, code comparison, efficiency in use, who would allow to improve use
of methods and tools in practical cases,
- to complete and improve the level of qualification and confidence in the use of
Alliances platform, comparing results from analytical test-cases to those provided by
the tool.
The report is divided into two main sections:
- a first part gives in a few pages some words about Alliances platform (sensitivity
module) and the way calculations were performed (choixe of sampling scheme, …)
- a second part gives results and interpretations of the performed test-cases.
PAMINA Sixth Framework programme, 10.09.2008 5
2 Framework
2.1 Cases performed by Andra
Test-cases finally performed by Andra are the following:
“Mandatory” cases Done
1 Saltelli – Chan – Scott section 2.9 model 1 X
2 Saltelli – Chan – Scott section 2.9 model 4 configuration (a) X
3 Saltelli – Chan – Scott section 2.9 model 4 configuration (c) X
4 Saltelli – Chan – Scott section 2.9 model 6 configuration (a) X
5 Saltelli – Chan – Scott section 2.9 model 7 X
6 Saltelli – Chan – Scott section 2.9 model 9 X
7 Saltelli – Chan – Scott section 2.9 model 10
8 Saltelli – Tarantola – Compolongo – Ratto section 3.1 X
Voluntary cases
Saltelli – Chan – Scott section 2.9 model 2
Saltelli – Chan – Scott section 2.9 model 3
Saltelli – Chan – Scott section 2.9 model 4 configuration (b)
Saltelli – Chan – Scott section 2.9 model 5
9 Saltelli – Chan – Scott section 2.9 model 6 configuration (b) X
Saltelli – Chan – Scott section 2.9 model 8
Saltelli – Tarantola – Compolongo – Ratto section 3.5
2.2 Short description of sensitivity module of Alliances platform
Test-cases were performed with sensitivity module of Alliances platform.
Each study is divided into three steps (see Figure 2-1) : generation, launch, analysis
PAMINA Sixth Framework programme, 10.09.2008 6
Figure 2-1: Phases of a sensitivity study
2.2.1 Generation
Generation consists of the creation of a collection of data sets corresponding to the
application’s area of analysis. The generator covers two steps: sampling and evaluation of
the data sets.
2.2.1.1 Sampling
This stage allows the sampling of variables defined by a statistical law.
Inputs include:
- One or more stochastic variables defined by a name, a statistical law and the associated parameters, and a certain number of possible correlations between these variables,
- One or more samplers and the associated parameters, - A library containing the sampling routine and the different statistical laws.
It outputs all stochastic variables sampled.
2.2.1.2 Evaluation
The evaluation stage comprises the calculation of variables determined by a function of other
variables (hereafter called static correlation).
Its inputs include:
- A static variable, defined by a name, a function, and a list of parameter variables, - A sample. It outputs the functional variables sampled.
LanceurLanceur
Post-traitementPost-traitement
GénérateurGénérateur
Échantillonnage
Évaluation
AnalyseurAnalyseur
Calcul
Extraction
PAMINA Sixth Framework programme, 10.09.2008 7
2.2.2 Launch
The launcher starts the application on all data sets generated by the generator.
Its inputs include:
- Occurrences of the variables sampled, - The application in the form of a deterministic reference script, - The launch method, - The links between the stochastic variables and their deterministic equivalent in the
application script, - The output list.
It returns the values of all outputs requested for each occurrence of the data set.
Calculations may be performed in a number of ways:
- Sequentially, locally on a machine, - In batches on a network of machines using PBS (Portable Batch System) management
software, - Distributed over an installed base.
2.2.3 Analysis
The analyser corresponds to the post-processing phase of the calculation and uses the
results obtained by the launch phase. It comprises an extraction stage followed by a
calculation stage.
2.2.3.1 Extraction
The aim of the extraction stage is to extract from the launcher outputs the input data directly
usable during the analysis stage.
Its inputs include:
- Occurrences of the variables sampled, - The output obtained for each occurrence.
It outputs the collection of data that will be processed during the statistical analysis stage.
2.2.3.2 Calculation stage Starting from the data provided by the extraction stage, the calculation phase evaluates the
statistical indicators requested for the sensitivity and uncertainty analysis.
Its inputs include:
- Occurrences of the variables sampled, - Occurrences of the corresponding outputs.
It outputs the collection of statistical indicators requested.
2.2.4 Post-processing
The processing of indicators resulting from the statistical analysis is generally carried out in a
post-processing phase, used to present results as numbers or curves.
PAMINA Sixth Framework programme, 10.09.2008 8
Its inputs include the statistical indicators calculated in the analysis phase and returns them
in the form of scalars, tables, or graphical representations.
Graphical are generally:
- 2D curves showing the development of indicators over time, - Scatter plots comparing results as a function of an input parameter.
Stage Action
SA
MP
LIN
G
1. Probabilistic data Declaration of uncertain variables Declaration of correlations (dependence between variables)
2. Sampler data Declaration of properties of sampling(s)
3. Sampling module call Code call to carry out sampling(s) Code call to perform evaluation of functions with stochastic
parameters
LA
UN
CH
ER
1. Application Python script (or link to file containing script) performing the
calculation
2. Definition of application links - Analysis Link between stochastic variables and application parameters Output list
3. Launcher Coherence checking of probabilistic data and samples Creation of n Python scripts Launching of calculations Retrieval of results
AN
AL
YS
IS 1. Extraction of values for analysis
2. Uncertainty and sensitivity analysis Determination of statistical indicators considered as relevant Calculation of indicators
Structure of a statistical analysis script
According to the parts indicated in the file, Alliances will carry out all or part of the study.
PAMINA Sixth Framework programme, 10.09.2008 9
3 Results
3.1 Test-case 1: model 1
Description of the model:
Y = X1 + X2 + X3
Where X1 uniform on [0.5, 1.5]
X2 uniform on [1.5, 4.5]
X3 uniform on [4.5, 13.5]
Results
Results of the test-case are given using both Latin Hypercube Sampling (LHS) and Simple
Random Sampling (SRS). At this step, we also increase the number of simulations in order
both to identify the sufficient number of runs to get a good accuracy (relative error
numerical/analytical around 0,1 %) and to compare sampling methods.
Statistics of the output Y
expected result Nb runs
E(Y) V(Y)
13 7,58333
LHS
5 12,5304 8,3436
100 12,9996 7,6536
500 12,9999 7,59874
5000 13 7,55767
9000 13 7,58917
11000 13 7,59648
16000 13 7,59735
SRS
5 10,6547 3,9603
100 12,9623 7,7654
500 12,9702 8,21885
5000 12,9853 7,57471
Sensitivity coefficients of the output Y
expected result
Nb runs
Pearson R2p R2s PCC
0,104 0,314 0,943 1 1 1 1 1
X1 X2 X3 X1 X2 X3
LHS
5 0,07 0,11 0,95 1 1 0,99 0,99 0,99
100 0,15 0,3 0,94 Nan 0,992 Nan Nan Nan
500 0,11 0,31 0,94 Nan 0,993 Nan Nan Nan
5000 0,1 0,31 0,94 1 0,995 1 1 1
9000 0,11 0,31 0,94 Nan 0,995 Nan Nan Nan
11000 0,1 0,32 0,94 Nan 0,9959 Nan Nan Nan
PAMINA Sixth Framework programme, 10.09.2008 10
16000 0,11 0,32 0,94 Nan 0,9958 Nan Nan Nan
SRS
5 0 0,88 0,97 Nan 1 Nan Nan Nan
100 0,06 0,3 0,95 1 0,987 0,99 0,99 0,99
500 0,12 0,33 0,95 1 0,994 1 1 1
5000 0,12 0,3 0,94 Nan 0,995 Nan Nan Nan
Comments about the results are the following:
- at this step, results using LHS are more accurate than those using SRS method, with a
number of simulations less than 5000; from 5000, both methods seem to be quite equivalent;
- results using LHS method seem to “converge” to analytical solution from 5000 simulations;
- on many cases, NaN (Not a Number) appears as a bad and hazardous result including a
division by zero in the calculations. This particularity, never been seen before in our previous
calculations in complex PA applications, is being highlighted by our developer team to solve
the problem.
3.2 Test-case 2: model 4 configuration a)
Description of the model:
4
21 XXY
Where X1 and X2 uniform on [0, 1]
Results
Results of the test-case are given using Latin Hypercube Sampling (LHS) and taking into
account an increase number of runs, from 5 to 16 000.
Statistics of the output Y
expected result
Nb runs
E(Y) V(Y) R2p R2s
0,7 0,15444 0,89 0,89
LHS
5 0,68655 0,24695 0,9741 0,97
500 0,70006 0,1671 0,8966 0,9053
5000 0,700002 0,15395 0,8966 0,9053
16000 0,7 0,15388 0,8845 0,8895
Sensitivity coefficients of the output Y
expected result
Nb runs
Pearson PCC Spearman PRCC
0,73455 0,58764 0,90784 0,86603 0,76 0,55 0,91 0,85
X1 X2 X1 X2 X1 X2 X1 X2
LHS
5 0,9 0,46 0,98 0,93 0,9 0,4 0,98 0,92
500 0,76 0,56 0,92 0,87 0,8 0,5 0,93 0,86
5000 0,73 0,59 0,91 0,87 0,76 0,56 0,92 0,86
16000 0,73 0,59 0,91 0,87 0,76 0,56 0,92 0,86
PAMINA Sixth Framework programme, 10.09.2008 11
Results from Alliances platform and analytical ones are considered to be the same, both for
statistics and sensitivity analysis, from 5000 runs.
3.3 Test-case 3: model 4 configuration c)
Description of the model:
4
21 XXY
Where X1 and X2 uniform on [0, 5]
Results
Results of the test-case are given using Latin Hypercube Sampling (LHS) and taking into
account an increase number of runs.
Statistics of the output Y
expected result
Nb runs
E(Y) V(Y) R2p R2s
127.5 27779.86 0.75 0.98
LHS
5 101.775 22036.5 0.8865 0.9700
500 127.524 27882.5 0.7580 0.9816
5000 127.501 27784.02 0.7500 0.9826
16000 127.4999 27779.76 0.7500 0.9823
Sensitivity coefficients of the output Y
expected result
Nb runs
Pearson PCC Spearman PRCC
0.00866 0.86599 0.01732 0.86603 0.08 0.99 0.50 0.99
X1 X2 X1 X2 X1 X2 X1 X2
LHS
5 0.38 0.88 0.70 0.93 0.40 0.90 0.92 0.98
500 0.09 0.87 0.19 0.87 0.07 0.99 0.49 0.99
5000 0.01 0.87 0.01 0.87 0.08 0.99 0.51 0.99
16000 0.00 0.87 0.01 0.87 0.08 0.99 0.52 0.99
3.4 Test-case 4: model 6 configuration a)
Description of the model:
2
1
4
2
X
XY
Where X1 and X2 uniform on [0.9, 1.1]
PAMINA Sixth Framework programme, 10.09.2008 12
Results
Results of the test-case are given using Latin Hypercube Sampling (LHS) and taking into
account an increase number of runs.
Statistics of the output Y
expected result
Nb runs
E(Y) V(Y) R2p R2s
1.030323 0.07 0.98 0.99
LHS
5 0.994571 0.052697 0.9994 1.000
500 1.032024 0.068512 0.9855 0.9908
2000 1.030287 0.070997 0.9833 0.9889
5000 1.030346 0.070660 0.9838 0.9907
16000 1.030292 0.070357 0.9836 0.9903
Sensitivity coefficients of the output Y
expected result
Nb runs
Pearson PCC Spearman PRCC
-0.45 0.89 -0.98 0.99 -0.42 0.90 -0.97 0.99
X1 X2 X1 X2 X1 X2 X1 X2
LHS
5 -0.43 0.88 -1.00 1.00 0.00 1.00 -0.00 1.00
500 -0.45 0.89 -0.97 0.99 -0.44 0.89 -0.98 0.98
2000 -0.45 0.88 -0.96 0.99 -0.42 0.90 -0.97 0.99
5000 -0.45 0.88 -0.96 0.99 -0.43 0.90 -0.98 0.99
16000 -0.45 0.88 -0.96 0.99 -0.43 0.90 -0.97 0.99
From 2000 LHS simulations, results from Alliances platform fit with analytical results. We can
observe a very low difference on PCC between calculated one and analytical one (less tha
2%).
3.5 Test-case 5: model 7
Description of the model:
k
j
jj )X(gY1
Where j
jj
jja
aX)X(g
1
24
k = 8
aj = {0, 1, 4.5, 9, 99, 99, 99, 99, 99} and Xj uniform on [0;1]
PAMINA Sixth Framework programme, 10.09.2008 13
Results
Results of the test-case are given using Latin Hypercube Sampling (LHS) and taking into
account an increase number of runs. Only mean, variancy and R² were calculated. Alliances
does not allow us to calculate non-monotonic Sobol.
Statistics of the output Y
expected result
Nb runs
E(Y) V(Y) R2p R2s
1 0.4652 0
LHS
5 1.060069 0.4266685 Nan Nan
500 1.013458 0.4907343 0.0484 0.0461
2000 0.985436 0.4244567 0.0023 0.0022
5000 1.005341 0.4786680 0.0020 0.0016
16000 0.992562 0.4506981 0.0017 0.0019
Results from Alliances platform fit with analytical results. We can observe a very difference
on PCC between calculated one and analytical one (less tha 2%).
3.6 Test-case 6: model 9
Description of the model:
This is non-monotonic Isghigami function
Y = sin X1 + A sin²X2 + BX34 .sin X1
Where Xi uniform on [- ; ]
A = 7; B = 0,1
expected result
Nb runs
E(Y) V(Y) R2p R2s
13.84458 0.19
LHS
5 2.745193 5.081918 0.6128 0.4776
500 3.522002 0.1511568 0.2292 0.2340
2000 3.519376 0.1415279 0.2064 0.2096
5000 3.547255 0.1350316 0.1947 0.1957
16000 3.536333 0.1384331 0.1914 0.1905
Expected result on mean can’t be calculated by Alliances platform, while value R2p seems to
fit with analytical one. Investigations are being done in Alliances programm to try to identify
the error...hoping that theoritical result given in the paper to be compared is the correct one.
PAMINA Sixth Framework programme, 10.09.2008 14
expected result
Nb runs
SRC SRRC
0.435 0 0 0.436 0 0
X1 X2 X3 X1 X2 X3
LHS
5 0.20 0.44 -0.54 0.24 0.24 -0.55
100 0.48 -0.00 0.04 0.48 0.01 0.01
500 0.45 -0.01 -0.01 0.46 -0.01 -0.01
1000 0.44 0.01 -0.01 0.44 -0.01 0.00
2000 0.44 0.01 -0.01 0.44 -0.01 -0.01
From 1000 LHS simulations, results from Alliances platform fit with analytical results. For
rank dependancy between Y and X2 and X3, a very residual value (less than 0,01) is
generated by the tool. But results from Alliances are globally correct.
3.7 Test-case 7: model 10
Not done
3.8 Test-case 8: model Tarantola – Compolongo – Ratto section 3.1 Jumping man applied to variance-based methods
The following optimisation model has to be considered:
elk
MgHh
2min
where hmin is the minimum distance to the asphalt during the oscillation
H is the distance of the platform to the platform [m], represented by a uniform distribution with a minimum value of 40 m and maximal value of 60
M is our mass [kg] represented by a uniform distribution with a lower bound of 67 kg and an upper boung of 74 kg
is the number of strands in the cord
g = 9,81 N/kg, kel is the elastic constant of one strand = 1.5
Results of the test-case are given using Latin Hypercube Sampling (LHS) and taking into
account an increase number of runs.
Statistics of the output Y
PAMINA Sixth Framework programme, 10.09.2008 15
expected result
Nb runs
E(Y) V(Y) R2p R2s
LHS
5 17.59548 103.3824 0.9997 0.9872
500 18.02011 80.32472 0.9831 0.9612
2000 18.01702 76.14252 0.9825 0.9680
5000 18.02134 76.20434 0.9826 0.9688
16000 18.02129 76.55709 0.9826 0.9694
Sensitivity coefficients of the output Y
First order sensitivity indices for the three variables are:
SH = 0,44
SM = 0,01
S = 0.55
expected result
Nb runs Pearson PCC
H M s H M s
LHS
5 0.71 -0.15 0.68 1.00 0.82 1.00
100 0.68 -0.12 0.74 0.98 -0.62 0.98
500 0.66 -0.11 0.73 0.98 -0.65 0.98
1000 0.67 -0.11 0.73 0.98 -0.60 0.98
2000 0.67 -0.11 0.73 0.98 -0.61 0.98
expected result
Nb runs Spearman PRCC SRC
H M s H M s H M s
LHS
5 0.50 0.00 0.80 0.98 0.61 0.99 0.73 0.03 0.71
100 0.68 -0.12 0.73 0.96 -0.46 0.96 0.65 -0.10 0.71
500 0.66 -0.10 0.73 0.96 -0.51 0.97 0.66 -0.11 0.73
1000 0.67 -0.11 0.72 0.97 -0.47 0.97 0.66 -0.10 0.73
2000 0.67 -0.10 0.73 0.97 -0.48 0.97 0.66 -0.10 0.73
Number of simulations (%) where H < 0 (%)
Nb total runs Nb runs where H<0
(%)
5 0 100 4 500 2,4 1000 2,6 reference case
2000 2,5
PAMINA Sixth Framework programme, 10.09.2008 16
The results from Alliances platform, represented by all indicators (linearity and rank) are as
follows:
High level of correlation (linearity and monotoneous) between result and M, as it is
given in the paper. However, it is also the case for , related to all sensitivity
indicators, elsewhere it is written the opposite in the paper.
Looking at the results shows a good accuracy from 1000 LHS simulations.
Due to linearity and monotony model, linear and monotonous sensitivity indicators
shows the same level of dependacy between results and each input data;
For 1000 simualtions of reference case, number of runs where H<0 is the same than
the analytical one; the percentage is nearly the same from 500 simulations.
3.9 Test-case 9: model 6 configuration b)
Description of the model:
2
1
4
2
X
XY
Where X1 and X2 uniform on [0.5, 1.5]
Results
Results of the test-case are given using Latin Hypercube Sampling (LHS) and taking into
account an increase number of runs.
Statistics of the output Y
expected result
Nb runs
E(Y) V(Y) R2p R2s
2.016666 6.90125 0.675 0.98
LHS
5 1.284741 6.381600 0.9795 1.0000
500 1.960978 5.476351 0.6858 0.9779
2000 2.033654 7.337333 0.6749 0.9786
5000 2.023927 7.010407 0.6753 0.9801
Sensitivity coefficients of the output Y
expected result
Nb runs
Pearson PCC Spearman PRCC
-0.47 0.67 -0.64 0.76 -0.43 0.89 -0.95 0.99
X1 X2 X1 X2 X1 X2 X1 X2
LHS
5 -0.26 0.94 -0.91 0.99 -0.00 1.00 -0.00 1.00
500 -0.48 0.68 -0.64 0.77 -0.47 0.88 -0.95 0.99
2000 -0.47 0.67 -0.64 0.76 -0.42 0.89 -0.95 0.99
PAMINA Sixth Framework programme, 10.09.2008 17
5000 -0.47 0.67 -0.64 0.76 -0.43 0.89 -0.95 0.99
Calculations have been done up to 5000 simulations. Results from Alliances platform are
globally fit with analytical results especially for sensitivity indicators. It is not the case for
Mean and variancy which shows a small difference between calculated one and analytical
one. On this case, an increase of the number of runs (more than 10.000) would have been
more accurate on these two indicators.
JRC
1
Results for the models in the benchmark Pamina task 2.1.D
The software used: R (see http://cran.r-project.org/) and package sensitivity.
Analytic benchmark cases 2, 7, 9, 13b
This part concerns the results to be inserted in the synthesis, i.e. the sensitivity indices (1st order and total)
for the models 2, 7, 9, 13b.
The sample sizes considered are: 100, 300, 1000, 3000, 10000. For each sample size 25 runs have been
performed.
Model 2 Model 7: Sobol g-function with 8 parameters
sample
size
values for
the 25 runs
X1 X2 X3 X4 X5 X6 X7 X8expected
results 0.7165 0.1791 0.0237 0.0078 0.0001 0.0001 0.0001 0.0001
100 mean 0.7895 0.2369 0.0579 0.0241 0.0228 0.0231 0.0221 0.0231
st. dev. 0.3083 0.1384 0.0502 0.0237 0.0037 0.0039 0.0034 0.0040
300 mean 0.7008 0.1697 0.0292 0.0156 0.0082 0.0068 0.0073 0.0073
st. dev. 0.1393 0.0631 0.0307 0.0152 0.0017 0.0016 0.0014 0.0015
1000 mean 0.7214 0.1886 0.0286 0.0119 0.0027 0.0019 0.0021 0.0023
st. dev. 0.0637 0.0516 0.0163 0.0079 0.0008 0.0009 0.0007 0.0010
3000 mean 0.7153 0.1711 0.0235 0.0078 0.0007 0.0009 0.0008 0.0008
st. dev. 0.0553 0.0207 0.0068 0.0036 0.0004 0.0004 0.0005 0.0005
10000 mean 0.7128 0.1830 0.0236 0.0086 0.0003 0.0002 0.0003 0.0003st. dev. 0.0267 0.0131 0.0032 0.0025 0.0002 0.0003 0.0003 0.0002
computed
results, SRS
SI first order
sample
size
values for
the 25 runs
X1 X2 X3 X4 X5 X6 X7 X8expected
results
100 mean 0.7167 0.2223 0.0118 -0.0127 -0.0228 -0.0223 -0.0211 -0.0222
st. dev. 0.2194 0.1724 0.0775 0.0388 0.0049 0.0038 0.0058 0.0043
300 mean 0.7874 0.2572 0.0442 -0.0022 -0.0080 -0.0064 -0.0074 -0.0073
st. dev. 0.1078 0.0697 0.0436 0.0215 0.0023 0.0026 0.0021 0.0024
1000 mean 0.7901 0.2355 0.0263 -0.0001 -0.0025 -0.0015 -0.0022 -0.0021
st. dev. 0.0453 0.0568 0.0215 0.0115 0.0011 0.0015 0.0012 0.0016
3000 mean 0.7956 0.2482 0.0333 0.0113 -0.0005 -0.0006 -0.0009 -0.0005
st. dev. 0.0357 0.0234 0.0116 0.0045 0.0007 0.0008 0.0008 0.0008
10000 mean 0.7889 0.2392 0.0359 0.0087 -0.0001 0.0001 -0.0001 -0.0001st. dev. 0.0192 0.0138 0.0053 0.0042 0.0004 0.0004 0.0004 0.0004
SI total
computed
results, SRS
JRC
2
sample
size
X1 X2 X3 X4 X5 X6 X7 X8expected
results 0.7165 0.1791 0.0237 0.0078 0.0001 0.0001 0.0001 0.0001
100 0.53428813 0.135044 0.0158402 0.004545 0.000895 0.000895 0.000895 0.000895
300 0.73864579 0.204853 0.0368873 0.011946 9.48E-05 9.48E-05 9.48E-05 9.48E-05
1000 0.67752622 0.170407 0.0219343 0.006427 6.42E-05 6.42E-05 6.42E-05 6.42E-05
3000 0.71120113 0.177777 0.0241129 0.007427 7.39E-05 7.39E-05 7.39E-05 7.39E-05
10000 0.7064575 0.176708 0.0239641 0.007402 7.5E-05 7.5E-05 7.5E-05 7.5E-05
SI first order
EFAST
sampling
sample
size
X1 X2 X3 X4 X5 X6 X7 X8expected
results
100 0.67786896 0.214663 0.0306157 0.012708 0.008975 0.008975 0.008975 0.008975
300 0.81052275 0.276832 0.047266 0.015794 0.001081 0.001081 0.001081 0.001081
1000 0.77963196 0.246301 0.0354456 0.010735 0.000192 0.000192 0.000192 0.000192
3000 0.78909078 0.244029 0.0347681 0.010768 0.000161 0.000161 0.000161 0.000161
10000 0.78977478 0.245127 0.0353835 0.010976 0.000175 0.000175 0.000175 0.000175
EFAST
sampling
SI total
Other statistics : R
2, R
2*.
sample
size R2 R2*
100 0.0985 0.1013300 0.0262 0.0260
1000 0.0096 0.0096
3000 0.0031 0.0028
10000 0.0007 0.0007
JRC
3
Model 9: Ishigami function
sample size
values for
the 25 runs
X1 X2 X3 X1 X2 X3expected
results 0.3139 0.4424 0 0.5596 0.4424 0.2437
100 mean 0.3640 0.4493 0.0123 0.5349 0.4405 0.2233st. dev. 0.1812 0.1933 0.0943 0.1594 0.1193 0.0952
300 mean 0.3148 0.4257 -0.0026 0.5651 0.4447 0.2408
st. dev. 0.0725 0.0744 0.0655 0.1015 0.0663 0.0726
1000 mean 0.3281 0.4473 0.0006 0.5466 0.4431 0.2440
st. dev. 0.0470 0.0485 0.0411 0.0481 0.0402 0.0342
3000 mean 0.3156 0.4466 0.0039 0.5608 0.4428 0.2436
st. dev. 0.0245 0.0224 0.0202 0.0242 0.0220 0.0189
10000 mean 0.3133 0.4411 0.0013 0.5613 0.4427 0.2434st. dev. 0.0174 0.0140 0.0108 0.0163 0.0078 0.0128
computed
results, SRS
SI first order (Sobol) SI total (Sobol)
JRC
4
sample size
X1 X2 X3 X1 X2 X3expected
results 0.3139 0.4424 0 0.5596 0.4424 0.2437
100 0.3091 0.6638 0.0000 0.5512 0.6855 0.1689
300 0.3090 0.4428 0.0000 0.5532 0.4676 0.1659
1000 0.3077 0.4420 0.0000 0.5506 0.4698 0.2391
3000 0.3076 0.4423 0.0000 0.5507 0.4629 0.2393
10000 0.3076 0.4439 0.0000 0.5508 0.4877 0.2393
EFAST
sampling
SI first order SI total
Other statistics : R
2, R
2*.
sample
size R2 R2*
100 0.2121 0.2063
300 0.1975 0.1978
1000 0.1930 0.1933
3000 0.1908 0.1907
10000 0.1926 0.1928
JRC
5
Model 13b: y = sign(x1-0.5) * x2
sample size
values for
the 25 runs
X1 X2 X1 X2expected
results 0.75 0 1 0.25
100 mean 0.7455 0.0034 1.0183 0.2640
st.dev. 0.0986 0.0880 0.1148 0.0621
300 mean 0.7482 -0.0070 1.0038 0.2523
st.dev. 0.0707 0.0465 0.0933 0.0307
1000 mean 0.7370 0.0033 0.9936 0.2504
st.dev. 0.0350 0.0231 0.0453 0.0189
3000 mean 0.7495 0.0019 1.0004 0.2523
st.dev. 0.0272 0.0140 0.0305 0.0115
10000 mean 0.7486 -0.0013 0.9995 0.2502
st.dev. 0.0124 0.0077 0.0143 0.0061
computed
results, SRS
SI first order (Sobol) SI total (Sobol)
sample size
X1 X2 X1 X2expected
results 0.75 0 1 0.25
100 0.6828 0.0000 0.9945 0.2817
300 0.6760 0.0005 0.9977 0.2606
1000 0.6754 0.0000 0.9987 0.2539
3000 0.6755 0.0000 0.9996 0.2513
10000 0.6755 0.0000 0.9999 0.2503
SI first order SI total
EFAST
sampling
Other statistics : R2, R
2*.
sample
size R2 R2*
100 0.578847 0.568565
300 0.567882 0.568003
1000 0.564531 0.566074
3000 0.562659 0.563264
10000 0.561538 0.561304
JRC
6
JRC
7
Annex : previous results for the models in the benchmark Pamina task 2.1.D – regression and variance based methods
The following results concern
the obligatory models: 1, 4, 6, 7, 9, 10 and 11 and
the voluntary models: 3, 5, 12, and 13.
The following statistics have been computed:
mean, variance, R2, R2*.
The following indices have been computed for each input variable:
Pearson, Spearman, SRC, PCC, SRRC, PRCC, first order Sobol sensitivity index, total Sobol
sensitivity index.
In general, the Sobol indices are computed using A. Saltelli, 2002, Making best use of model evaluations to
compute sensitivity indices, Computer Physics Communication, 145, 580–297.
In particular cases the use of Saltelli’s method has been replaced by the use of FAST method.
For each model we tried to consider the following sample sizes: 100, 1000, 10000, 100000 and 200000.
However, for some of the models we couldn’t run the computations for sizes 100000 and/or 200000, due to
memory size problems. This happened for Models 3, 5, 7, 10 and 12.
More specifically
for Model 7 (8 input variables) and sample size =200000:
o the use of Saltelli’s method replaced by the use of FAST method (for Sobol indices)
o the computations of SRRC, PRCC and R2* have not been possible.
for Models 3, 5b, 10 (more than 20 input variables) and sample sizes =100000, 200000 no
computation have been performed.
for Models 5a, 12 and sample size = 200000 no computation have been performed. For these models
the number of input variables is 6, less than for Model 7, hence Pearson, Spearman, SRC, PCC, and
Sobol indices using FAST could have been computed even for this sample size.
For each sample size, the procedure has been repeated 10 times.
For each model, the results are presented as tables of the mean and standard deviation for every computed
statistic or index and also as a set of 12 figures, one for each computed quantity. On the abscissa of each
figure is the sample size. The 12 figures are grouped in 3 groups of 4
group 1 : mean, variance, R2, R2*
group 2 : Pearson, Spearman, first order Sobol sensitivity index, total Sobol sensitivity index
group 3 : SRC, PCC, SRRC, PRCC.
This pattern is found for each model.
The numerical results are in .csv files (to be read with Excel), can be provided. The only file which is
slightly different from the others is model7.csv, where there are some “holes” and where the results
obtained by FAST have been reported in some additional lines.
JRC
8
Model 1
sample
size
values for
the 10
runsE(Y) V(Y) R2 R2*
expected
results 13 7.58333
100 mean 12.932343 7.600542 1 0.986808
st.dev. 0.1051102 0.809675 0 0.003795
1000 mean 13.06644 7.663799 1 0.994375
st.dev. 0.074774 0.331199 0 0.000895
10000 mean 13.008654 7.619412 1 0.995428
st.dev. 0.0164204 0.061852 0 0.00013
100000 mean 12.998914 7.593424 1 0.995396
st.dev. 0.0073975 0.023977 0 0.000103
200000 mean 13.00157 7.576896 1 0.99542
st.dev. 0.0045198 0.013763 0 5.19E-05
compute
d results,
SRS
sample size
values for the 10 runs
Pearson Spearman
X1 X2 X3 X1 X2 X3
expected results 0.104830 0.314490 0.943460
computed results, SRS
100 mean 0.087329 0.329939 0.943904 0.07569 0.314493 0.94355
st.dev. 0.088959 0.061339 0.006624 0.092359 0.063438 0.006702
1000 mean 0.103847 0.317113 0.943595 0.097313 0.29991 0.946716
st.dev. 0.037543 0.029624 0.002611 0.038711 0.031532 0.002069
10000 mean 0.103879 0.314499 0.943594 0.097417 0.298581 0.94702
st.dev. 0.009039 0.008591 0.000481 0.009125 0.008809 0.000477
100000 mean 0.105508 0.315078 0.943438 0.098926 0.298609 0.947042
st.dev. 0.002441 0.002145 0.00029 0.002504 0.002364 0.00024
200000 mean 0.104376 0.315126 0.94341 0.097868 0.298675 0.94702
st.dev. 0.002311 0.002157 0.000181 0.002353 0.002302 0.000155
JRC
9
sample size
values for the 10 runs
SRC PCC
X1 X2 X3 X1 X2 X3
expected results
computed results, SRS
100 mean 0.105316 0.313555 0.940396 1 1 1
st.dev. 0.007513 0.016442 0.021612 0 0 0
1000 mean 0.103808 0.314058 0.943098 1 1 1
st.dev. 0.002527 0.009361 0.012316 0 0 0
10000 mean 0.104601 0.313866 0.943654 1 1 1
st.dev. 0.000619 0.001285 0.003295 0 0 0
100000 mean 0.104779 0.314369 0.943247 1 1 1
st.dev. 0.000237 0.000808 0.000865 0 0 0
200000 mean 0.104842 0.314625 0.943292 1 1 1
st.dev. 9.03E-05 0.000511 0.000618 0 0 0
sample size
values for the 10 runs
SRRC PRCC
X1 X2 X3 X1 X2 X3
expected results
computed results, SRS
100 mean 0.094737 0.294838 0.940247 0.631811 0.931169 0.992537
st.dev. 0.018602 0.014289 0.020907 0.106192 0.018389 0.002004
1000 mean 0.096777 0.29759 0.946312 0.79065 0.969624 0.996869
st.dev. 0.003699 0.007634 0.011962 0.026896 0.004686 0.000475
10000 mean 0.098198 0.297968 0.947071 0.823601 0.975202 0.997461
st.dev. 0.00076 0.001306 0.003131 0.005032 0.000706 7.4E-05
100000 mean 0.098233 0.297913 0.946863 0.822804 0.975028 0.997442
st.dev. 0.000212 0.000761 0.000876 0.003089 0.000583 5.55E-05
200000 mean 0.098337 0.29817 0.946908 0.823774 0.975197 0.997456
st.dev. 0.00017 0.000421 0.000619 0.001757 0.000253 2.6E-05
JRC
10
sample size
values for the 10 runs
SI first order (Sobol) SI total (Sobol)
X1 X2 X3 X1 X2 X3
expected results 0.010989 0.098901 0.890109 0.010989 0.098901 0.890109
computed results, SRS
100 mean 0.209351 0.443534 1.239186 -0.19501 -0.24807 0.513037
st.dev. 0.056713 0.251715 0.585745 0.059612 0.237778 0.615047
1000 mean 0.034532 0.133905 0.779787 -0.01367 0.064315 0.994934
st.dev. 0.01649 0.050888 0.195354 0.01585 0.050542 0.203523
10000 mean 0.011384 0.096488 0.855071 0.009281 0.103321 0.922177
st.dev. 0.006549 0.017538 0.055106 0.006941 0.019584 0.060987
100000 mean 0.011594 0.099817 0.895779 0.010351 0.098783 0.878347
st.dev. 0.002581 0.007472 0.023032 0.002476 0.006999 0.019583
200000 mean 0.011109 0.09759 0.893136 0.010665 0.100493 0.888161
st.dev. 0.001203 0.004301 0.010084 0.001457 0.004135 0.012251
JRC
11
JRC
12
Model 3
sample size
values for the 10 runs
E(Y) V(Y) R2 R2*
expected results 0 5442.30000
computed results, SRS
100 mean -1.32185 5561.9617 1 0.963184
st.dev. 7.886669 510.72771 0 0.013809
1000 mean -0.18327 5482.9973 1 0.959112
st.dev. 2.874867 222.73222 0 0.005715
10000 mean 0.005509 5467.7223 1 0.962092
st.dev. 0.660955 74.183209 0 0.001638
Pearson
sample size = 100 sample size = 1000 sample size = 10000
expected results mean sd. dev. mean sd. dev. mean sd. dev.
X1 0.391309 0.415698 0.10866 0.3936755 0.023748 0.390265 0.005372
X2 0.316961 0.372162 0.080462 0.3147805 0.020729 0.315044 0.008526
X3 0.250438 0.214832 0.107394 0.2598459 0.045821 0.253741 0.01161
X4 0.191742 0.12979 0.149155 0.2059098 0.031379 0.195035 0.010399
X5 0.140871 0.115899 0.109134 0.1393274 0.031452 0.14082 0.014432
X6 0.097827 0.110574 0.111441 0.0778596 0.04334 0.099712 0.010711
X7 0.062609 0.070358 0.110318 0.0635861 0.040735 0.061301 0.003899
X8 0.035218 0.008595 0.082546 0.0365169 0.027773 0.033264 0.011485
X9 0.015652 0.001068 0.128038 0.0157699 0.023141 0.02277 0.013203
X10 0.003913 -0.00561 0.10256 0.0012864 0.034635 0.002025 0.006552
X11 0 -0.02203 0.085299 0.007615 0.026534 0.006151 0.010558
X12 0.003913 -0.0353 0.147458 -0.0148333 0.030248 0.00803 0.010659
X13 0.015652 0.041142 0.074669 0.0032045 0.028628 0.012605 0.010286
X14 0.035218 0.027856 0.097436 0.0405083 0.020443 0.038473 0.00913
X15 0.062609 0.058307 0.078885 0.067487 0.041829 0.065406 0.009483
X16 0.097827 0.082663 0.127394 0.0993256 0.021259 0.094604 0.012001
X17 0.140871 0.169556 0.046528 0.1542061 0.034249 0.140093 0.005606
X18 0.191742 0.218165 0.085644 0.1865307 0.029749 0.190059 0.012072
X19 0.250438 0.229653 0.075737 0.2450769 0.028634 0.254094 0.011856
X20 0.316961 0.303199 0.078572 0.3275987 0.029309 0.319978 0.006277
X21 0.391309 0.370369 0.091279 0.3869652 0.009074 0.391835 0.011676
X22 0.473484 0.487689 0.067622 0.4712264 0.021759 0.474575 0.008797
JRC
13
Spearman
sample size = 100 sample size = 1000 sample size = 10000
expected results mean sd. dev. mean sd. dev. mean sd. dev.
X1 0.413731 0.116799 0.3878061 0.02386 0.383907 0.004928
X2 0.361785 0.100718 0.3072043 0.021727 0.307313 0.008141
X3 0.193307 0.112418 0.2515753 0.045193 0.247943 0.01086
X4 0.123928 0.141285 0.2026284 0.035219 0.190347 0.010781
X5 0.102339 0.105981 0.1343017 0.037434 0.135629 0.014181
X6 0.10283 0.101018 0.0776979 0.044683 0.097095 0.009854
X7 0.051534 0.105562 0.0626434 0.04002 0.058383 0.004014
X8 0.011917 0.088574 0.0339709 0.028633 0.032032 0.010836
X9 -0.00114 0.130795 0.019767 0.023625 0.023183 0.013501
X10 0.004143 0.103169 -0.0005378 0.034822 0.00155 0.007092
X11 -0.02611 0.083441 0.0052291 0.02623 0.005726 0.010622
X12 -0.04817 0.149725 -0.0163959 0.028649 0.00654 0.010154
X13 0.037721 0.070837 0.0039 0.034622 0.011404 0.011113
X14 0.032868 0.108352 0.0443777 0.019915 0.03675 0.008658
X15 0.056281 0.070291 0.0688048 0.040134 0.063631 0.008599
X16 0.083585 0.122151 0.0951189 0.0226 0.091881 0.011225
X17 0.163892 0.050738 0.150131 0.034868 0.136161 0.006336
X18 0.222205 0.090364 0.1817602 0.031882 0.184925 0.01218
X19 0.215695 0.077979 0.2365672 0.02798 0.248185 0.011531
X20 0.2982 0.08044 0.3180947 0.027908 0.312599 0.006175
X21 0.364772 0.091627 0.3790623 0.010478 0.385177 0.011909
X22 0.480462 0.073858 0.4655328 0.024673 0.469566 0.009658
JRC
14
SRC
sample size = 100 sample size = 1000 sample size = 10000
expected results mean sd. dev. mean sd. dev. mean sd. dev.
X1 0.393327 0.019646 0.391272 0.009474 0.390891 0.003276
X2 0.318684 0.018705 0.314849 0.00657 0.3154 0.002066
X3 0.243685 0.013457 0.2489032 0.006759 0.250022 0.001685
X4 0.186851 0.00594 0.1909019 0.004828 0.191489 0.001322
X5 0.140068 0.012658 0.1401499 0.003578 0.140615 0.001047
X6 0.097046 0.005663 0.097289 0.002333 0.097692 0.000717
X7 0.062362 0.004292 0.062311 0.001285 0.062458 0.000384
X8 0.034302 0.002028 0.0350444 0.000982 0.03516 0.000238
X9 0.015577 0.000902 0.0155457 0.000359 0.015594 0.000124
X10 0.003855 0.000229 0.0038875 6.66E-05 0.003901 3.01E-05
X11 -1.3E-17 3.58E-17 1.689E-18 4.78E-18 -1E-18 2.51E-18
X12 0.003954 0.000237 0.0039293 0.000117 0.0039 2.34E-05
X13 0.015657 0.000849 0.0157439 0.000282 0.015598 9.81E-05
X14 0.034828 0.001782 0.0349455 0.001184 0.035086 0.000297
X15 0.061542 0.00372 0.0621214 0.001238 0.062427 0.000663
X16 0.096119 0.009039 0.0982399 0.002459 0.097531 0.000481
X17 0.138468 0.005762 0.1404509 0.002908 0.140456 0.000618
X18 0.188151 0.010261 0.1915659 0.006332 0.191021 0.000968
X19 0.250708 0.013595 0.249972 0.005187 0.249943 0.001922
X20 0.320136 0.02043 0.3148071 0.006568 0.315607 0.002887
X21 0.386281 0.036558 0.3907458 0.009964 0.390012 0.002922
X22 0.476737 0.039028 0.4709287 0.006918 0.472444 0.003416
JRC
15
PCC
sample size = 100 sample size = 1000 sample size = 10000
expected results mean sd. dev. mean sd. dev. mean sd. dev.
X1 1 0 1 0 1 0
X2 1 0 1 0 1 0
X3 1 0 1 0 1 0
X4 1 0 1 0 1 0
X5 1 0 1 0 1 0
X6 1 0 1 0 1 0
X7 1 0 1 0 1 0
X8 1 0 1 0 1 0
X9 1 0 1 0 1 0
X10 1 0 1 0 1 0
X11 -0.04331 0.117519 -0.0105547 0.031067 -0.0027 0.009048
X12 1 0 1 0 1 0
X13 1 0 1 0 1 0
X14 1 0 1 0 1 0
X15 1 0 1 0 1 0
X16 1 0 1 0 1 0
X17 1 0 1 0 1 0
X18 1 0 1 0 1 0
X19 1 0 1 0 1 0
X20 1 0 1 0 1 0
X21 1 0 1 0 1 0
X22 1 0 1 0 1 0
JRC
16
SRRC
sample size = 100 sample size = 1000 sample size = 10000
expected results mean sd. dev. mean sd. dev. mean sd. dev.
X1 0.390902 0.035293 0.3851989 0.010664 0.384577 0.003335
X2 0.306394 0.024602 0.3063005 0.012463 0.307727 0.003322
X3 0.223385 0.020966 0.2400146 0.010423 0.244241 0.002213
X4 0.175094 0.022811 0.1881093 0.010598 0.186824 0.001715
X5 0.123499 0.031134 0.1348203 0.009559 0.135411 0.002763
X6 0.087884 0.029625 0.0964624 0.007248 0.09519 0.002582
X7 0.040876 0.03027 0.0613353 0.005672 0.059542 0.000881
X8 0.040736 0.036184 0.0323025 0.007509 0.033886 0.001651
X9 0.011539 0.028858 0.0201379 0.007954 0.016185 0.001976
X10 0.007545 0.026163 0.0023577 0.004765 0.003358 0.002241
X11 -0.00077 0.01576 -0.0021883 0.005803 -0.00031 0.002571
X12 -0.00892 0.021052 0.0025559 0.009655 0.002484 0.001487
X13 0.015264 0.018724 0.0168274 0.006776 0.014352 0.002186
X14 0.036783 0.02486 0.0390798 0.006562 0.033397 0.001417
X15 0.060985 0.031222 0.0632205 0.010064 0.060748 0.002801
X16 0.099523 0.02983 0.0939654 0.005886 0.094834 0.001645
X17 0.138315 0.035307 0.1370736 0.005033 0.136573 0.001352
X18 0.203423 0.025342 0.1863694 0.010134 0.185877 0.001673
X19 0.243072 0.016181 0.2413766 0.006423 0.244099 0.002892
X20 0.312208 0.028552 0.3048392 0.010235 0.308359 0.002537
X21 0.384825 0.03722 0.3821192 0.010297 0.383379 0.003415
X22 0.465282 0.051156 0.4648316 0.008125 0.467611 0.003239
JRC
17
PRCC
sample size = 100 sample size = 1000 sample size = 10000
expected results mean sd. dev. mean sd. dev. mean sd. dev.
X1 0.876519 0.034107 0.8830458 0.015505 0.891987 0.004516
X2 0.817648 0.061599 0.8320719 0.025197 0.844831 0.005917
X3 0.720101 0.083637 0.7615507 0.032862 0.781673 0.007712
X4 0.632931 0.107583 0.677311 0.03128 0.692041 0.008756
X5 0.497197 0.113117 0.5521158 0.046461 0.570594 0.011486
X6 0.375158 0.113068 0.428173 0.04492 0.438916 0.013172
X7 0.183758 0.139073 0.2890566 0.033557 0.29232 0.008049
X8 0.184745 0.141923 0.1582687 0.044285 0.171405 0.009117
X9 0.051252 0.12533 0.0995425 0.04126 0.082816 0.010335
X10 0.037605 0.122814 0.0113783 0.023217 0.017228 0.011474
X11 0.000461 0.079218 -0.0117689 0.02904 -0.00158 0.013089
X12 -0.03691 0.096271 0.013034 0.046367 0.012751 0.007584
X13 0.070917 0.088934 0.0824233 0.032995 0.073588 0.012008
X14 0.166214 0.105289 0.1887211 0.031955 0.168905 0.005793
X15 0.282958 0.143863 0.296297 0.044858 0.297759 0.016133
X16 0.409842 0.07125 0.4196064 0.037401 0.437715 0.011258
X17 0.540702 0.149506 0.5586071 0.036461 0.574033 0.009114
X18 0.695954 0.081803 0.6750285 0.031853 0.690239 0.008884
X19 0.754108 0.057466 0.7636585 0.027164 0.781457 0.007908
X20 0.820272 0.039102 0.8306266 0.024768 0.845306 0.006095
X21 0.874545 0.032763 0.8817368 0.016113 0.891442 0.004323
X22 0.906329 0.03639 0.9153368 0.011382 0.923029 0.003298
JRC
18
SI first order
sample size = 100 sample size = 1000 sample size = 10000
expected results mean sd. dev. mean sd. dev. mean sd. dev.
X1 0.155151 0.086325 0.1475831 0.01997 0.149374 0.005482
X2 0.094485 0.036057 0.0867818 0.012961 0.098221 0.003319
X3 0.055161 0.025469 0.0616202 0.009592 0.061359 0.003192
X4 0.029102 0.016911 0.0395024 0.008884 0.035685 0.003242
X5 0.01457 0.025532 0.0184248 0.005378 0.020396 0.001729
X6 0.005287 0.014464 0.0092952 0.00382 0.010167 0.0014
X7 0.001779 0.009297 0.0040502 0.003337 0.003758 0.000631
X8 -0.00065 0.004161 0.0019821 0.001211 0.001144 0.000353
X9 -0.00015 0.00212 -1.598E-05 0.000541 0.00022 0.000195
X10 -5.5E-05 0.001146 2.83E-05 0.000152 1.95E-05 4.49E-05
X11 -0.00013 0.001179 1.65E-06 3.65E-05 2.51E-07 9.06E-07
X12 -7.1E-05 0.001457 -6.121E-05 0.000162 2.31E-05 6.21E-05
X13 -0.00039 0.002821 0.0004586 0.000766 0.000166 0.000186
X14 0.000741 0.004099 0.0009725 0.001706 0.00129 0.000451
X15 0.004545 0.005174 0.0029527 0.00341 0.004042 0.001158
X16 0.007593 0.021058 0.0094059 0.005031 0.008926 0.000911
X17 0.022606 0.024117 0.0181043 0.004035 0.020119 0.001301
X18 0.034398 0.026152 0.0373841 0.008346 0.036444 0.002458
X19 0.040434 0.039643 0.061872 0.008045 0.061983 0.004056
X20 0.108584 0.048796 0.0984214 0.019145 0.104807 0.006967
X21 0.179483 0.062764 0.1568553 0.025988 0.154997 0.004483
X22 0.23832 0.063637 0.2202711 0.028503 0.222486 0.008072
JRC
19
SI total
sample size = 100 sample size = 1000 sample size = 10000
expected results mean sd. dev. mean sd. dev. mean sd. dev.
X1 0.178713 0.068762 0.154561 0.017335 0.151435 0.003961
X2 0.11625 0.047296 0.0979227 0.011881 0.100337 0.004689
X3 0.052478 0.038198 0.0658647 0.012532 0.063473 0.004263
X4 0.019178 0.034256 0.0393964 0.009279 0.038986 0.003992
X5 0.012813 0.020062 0.0168284 0.005679 0.019989 0.002911
X6 0.011443 0.018023 0.0075333 0.003051 0.009686 0.001665
X7 0.00787 0.006408 0.0039122 0.002335 0.003697 0.000933
X8 -0.00039 0.005041 0.0010237 0.001456 0.00113 0.000575
X9 0.000133 0.001896 0.0002757 0.000573 0.000352 0.000214
X10 1.7E-05 0.000405 4.769E-06 0.000214 2.46E-05 4.76E-05
X11 -0.0001 0.000142 -1.353E-06 1.55E-06 -7.1E-09 8.77E-09
X12 -0.00044 0.00068 -0.000104 0.000222 2.29E-05 6.31E-05
X13 0.001631 0.002221 2.162E-05 0.000637 0.000134 0.000188
X14 0.002226 0.003834 0.0020345 0.001426 0.00123 0.000417
X15 0.003947 0.00757 0.0035181 0.003072 0.003876 0.001148
X16 0.00794 0.012559 0.0109268 0.002074 0.0091 0.001831
X17 0.02435 0.012973 0.0225063 0.006906 0.019551 0.001736
X18 0.043799 0.032787 0.0324427 0.008965 0.036943 0.002626
X19 0.065885 0.027161 0.0605782 0.009651 0.061867 0.004399
X20 0.089899 0.063533 0.1049701 0.010286 0.09942 0.004402
X21 0.153808 0.046415 0.1473568 0.012447 0.152594 0.006452
X22 0.229876 0.060487 0.2281005 0.016075 0.223717 0.007764
JRC
20
JRC
21
JRC
22
Model 4a
sample size
values for the 10 runs
E(Y) V(Y) R2 R2*
expected results 0.7 0.15444 0.89 0.89
computed results, SRS
100 mean 0.67888 0.167322 0.894235 0.889557
st.dev. 0.030643 0.020807 0.013318 0.025038
1000 mean 0.693832 0.151854 0.884936 0.888076
st.dev. 0.013005 0.004318 0.004743 0.008281
10000 mean 0.699687 0.154375 0.885094 0.891272
st.dev. 0.003727 0.001641 0.001291 0.00215
100000 mean 0.699752 0.154248 0.885015 0.889624
st.dev. 0.001111 0.000581 0.00025 0.000843
200000 mean 0.700002 0.154516 0.884832 0.889739
st.dev. 0.000768 0.000448 0.000287 0.000475
sample size
values for the 10 runs
Pearson Spearman
X1 X2 X1 X2
expected results 0.74 0.59 0.76 0.55
computed results, SRS
100 mean 0.751057 0.603215 0.771972 0.570393
st.dev. 0.039021 0.06383 0.052281 0.07103
1000 mean 0.735975 0.589101 0.761445 0.558293
st.dev. 0.011502 0.011189 0.012473 0.011936
10000 mean 0.734791 0.588249 0.760288 0.560415
st.dev. 0.004328 0.007141 0.00455 0.007966
100000 mean 0.734786 0.587148 0.761083 0.556784
st.dev. 0.001142 0.001456 0.001494 0.001999
200000 mean 0.734044 0.588017 0.760128 0.558301
st.dev. 0.000553 0.000975 0.00088 0.001208
JRC
23
sample size
values for the 10 runs
SRC PCC
X1 X2 X1 X2
expected results 0.908 0.866
computed results, SRS
100 mean 0.727408 0.574799 0.91127 0.869438
st.dev. 0.051365 0.039219 0.015896 0.009359
1000 mean 0.73352 0.585881 0.907607 0.86538
st.dev. 0.007608 0.010957 0.00327 0.002679
10000 mean 0.734175 0.587502 0.907886 0.866153
st.dev. 0.005665 0.004724 0.001514 0.001419
100000 mean 0.735031 0.587455 0.908029 0.866071
st.dev. 0.00122 0.001291 0.000369 0.000384
200000 mean 0.734213 0.588228 0.907726 0.866184
st.dev. 0.000704 0.000601 0.000215 0.000314
sample size
values for the 10 runs
SRRC PRCC
X1 X2 X1 X2
expected results 0.91 0.85
computed results, SRS
100 mean 0.749894 0.539443 0.912789 0.85046
st.dev. 0.051756 0.056194 0.021812 0.025442
1000 mean 0.759307 0.555177 0.9151 0.856617
st.dev. 0.006848 0.010349 0.00563 0.005484
10000 mean 0.759706 0.559644 0.917311 0.86155
st.dev. 0.005643 0.005827 0.001773 0.003131
100000 mean 0.761322 0.557111 0.916531 0.858875
st.dev. 0.001471 0.001702 0.000675 0.000942
200000 mean 0.760288 0.558519 0.916409 0.859559
st.dev. 0.000691 0.001234 0.000276 0.000776
JRC
24
sample size
values for the 10 runs
SI first order (Sobol)
SI total (Sobol)
X1 X2 X1 X2
expected results 0.54 0.46
computed results, SRS
100 mean 0.637708 0.358282 0.41452 0.442453
st.dev. 0.233106 0.184284 0.196456 0.132412
1000 mean 0.570971 0.494467 0.512321 0.435776
st.dev. 0.077546 0.058723 0.079777 0.03787
10000 mean 0.556065 0.445866 0.532323 0.472108
st.dev. 0.021992 0.02511 0.02105 0.027949
100000 mean 0.540226 0.462109 0.543596 0.460257
st.dev. 0.006929 0.005086 0.00586 0.004153
200000 mean 0.54258 0.457099 0.535384 0.462043
st.dev. 0.003438 0.004886 0.00194 0.003824
JRC
25
JRC
26
Model 4b
sample size
values for the 10 runs
E(Y) V(Y) R2 R2*
expected results
computed results, SRS
100 mean 18.25801 488.6675 0.759967 0.954603
st.dev. 2.421902 93.75527 0.014666 0.019298
1000 mean 17.92406 479.2074 0.75229 0.953553
st.dev. 0.490808 15.37915 0.004645 0.006206
10000 mean 17.68072 464.589 0.74944 0.953728
st.dev. 0.197856 6.589549 0.002345 0.001991
100000 mean 17.71055 468.3674 0.750481 0.953407
st.dev. 0.070945 2.326598 0.001025 0.000429
200000 mean 17.68915 467.5154 0.750412 0.953434
st.dev. 0.058019 1.943157 0.000408 0.000358
sample size
values for the 10 runs
Pearson Spearman
X1 X2 X1 X2
expected results
computed results, SRS
100 mean 0.024824 0.869776 0.149001 0.963174
st.dev. 0.09756 0.008186 0.093377 0.016931
1000 mean 0.044894 0.86624 0.166798 0.962483
st.dev. 0.034526 0.002638 0.036844 0.005139
10000 mean 0.038175 0.864739 0.164446 0.962189
st.dev. 0.008372 0.001367 0.011061 0.001748
100000 mean 0.039937 0.86541 0.169377 0.961748
st.dev. 0.002834 0.000593 0.002853 0.000426
200000 mean 0.040468 0.865284 0.168133 0.961718
st.dev. 0.001817 0.000236 0.001646 0.000328
JRC
27
sample size
values for the 10 runs
SRC PCC
X1 X2 X1 X2
expected results
computed results, SRS
100 mean 0.030804 0.86923 0.061242 0.870417
st.dev. 0.05227 0.009642 0.105884 0.008834
1000 mean 0.042511 0.866084 0.085042 0.866902
st.dev. 0.010936 0.002687 0.02186 0.002546
10000 mean 0.040504 0.864891 0.08064 0.865481
st.dev. 0.005242 0.001322 0.010387 0.001341
100000 mean 0.039296 0.86538 0.078425 0.866072
st.dev. 0.001528 0.000605 0.003042 0.000594
200000 mean 0.041169 0.865317 0.082127 0.866027
st.dev. 0.001246 0.000249 0.002477 0.000237
sample size
values for the 10 runs
SRRC PRCC
X1 X2 X1 X2
expected results
computed results, SRS
100 mean 0.157661 0.964264 0.597212 0.976124
st.dev. 0.046015 0.022674 0.044063 0.0106
1000 mean 0.164517 0.962069 0.60722 0.975772
st.dev. 0.011618 0.007684 0.00938 0.003345
10000 mean 0.167055 0.962663 0.613385 0.975929
st.dev. 0.004182 0.00201 0.002509 0.001036
100000 mean 0.16866 0.961622 0.615694 0.975721
st.dev. 0.00131 0.00056 0.001942 0.000229
200000 mean 0.168916 0.961855 0.616388 0.975745
st.dev. 0.000854 0.00034 0.000872 0.000189
JRC
28
sample size
values for the 10 runs
SI first order (Sobol)
SI total (Sobol)
X1 X2 X1 X2
expected results
computed results, SRS
100 mean 0.011113 0.935986 -0.00921 1.043591
st.dev. 0.006899 0.250552 0.006385 0.152237
1000 mean 0.002202 0.999566 0.000408 1.002659
st.dev. 0.002551 0.06334 0.002257 0.054267
10000 mean 0.001577 1.011562 0.001546 0.995533
st.dev. 0.000626 0.032052 0.000685 0.011692
100000 mean 0.001563 0.99551 0.001572 1.000132
st.dev. 0.000308 0.006757 0.000314 0.003231
200000 mean 0.001596 0.999087 0.001662 0.998248
st.dev. 0.000125 0.008808 0.000143 0.003518
JRC
29
JRC
30
Model 4c
sample size
values for the 10 runs
E(Y) V(Y) R2 R2*
expected results 127.5 27780 0.75 0.98
computed results, SRS
100 mean 123.6812 26695.679 0.751967 0.970396
st.dev. 16.39837 4803.84148 0.021911 0.017798
1000 mean 127.5418 27650.7826 0.750126 0.982245
st.dev. 6.029571 1654.37615 0.006179 0.003348
10000 mean 126.9651 27554.1658 0.74858 0.982567
st.dev. 1.290108 292.100712 0.002643 0.000592
100000 mean 127.6007 27818.1632 0.750164 0.982302
st.dev. 0.259432 87.6603782 0.000419 0.000171
200000 mean 127.3584 27733.2661 0.749752 0.982316
st.dev. 0.403801 83.6824922 0.000383 0.000263
sample size
values for the 10 runs
Pearson Spearman
X1 X2 X1 X2
expected results 8.66E-03 0.866 0.08 0.99
computed results, SRS
100 mean 0.000557 0.866112 0.081079 0.980036
st.dev. 0.138512 0.013271 0.145654 0.012095
1000 mean 0.008713 0.865973 0.084555 0.987964
st.dev. 0.034077 0.003554 0.038976 0.002372
10000 mean 0.00635 0.865146 0.074892 0.988136
st.dev. 0.008423 0.001513 0.007722 0.000484
100000 mean 0.008444 0.866081 0.080003 0.987922
st.dev. 0.004321 0.000238 0.003699 0.000116
200000 mean 0.009204 0.86583 0.079232 0.987933
st.dev. 0.002638 0.000222 0.002728 0.000185
JRC
31
sample size
values for the 10 runs
SRC PCC
X1 X2 X1 X2
expected results 0.017 0.866
computed results, SRS
100 mean 0.009061 0.86424088 0.016509 0.864636
st.dev. 0.042048 0.01226562 0.082277 0.011691
1000 mean 0.002716 0.86619449 0.005321 0.865928
st.dev. 0.014863 0.00378348 0.029757 0.003618
10000 mean 0.009412 0.86517217 0.01878 0.865189
st.dev. 0.003631 0.00152735 0.007266 0.001522
100000 mean 0.007954 0.86607427 0.015913 0.866107
st.dev. 0.002025 0.00022839 0.004052 0.000238
200000 mean 0.009351 0.8658318 0.01869 0.865869
st.dev. 0.00137 0.00021989 0.002737 0.000221
sample size
values for the 10 runs
SRRC PRCC
X1 X2 X1 X2
expected results 0.50 0.99
computed results, SRS
100 mean 0.095373 0.980239 0.489841 0.984552
st.dev. 0.030736 0.020514 0.041431 0.009613
1000 mean 0.078159 0.987639 0.506521 0.991009
st.dev. 0.008775 0.002634 0.013523 0.001673
10000 mean 0.07842 0.988417 0.510592 0.991195
st.dev. 0.002422 0.00069 0.006217 0.000301
100000 mean 0.079442 0.987877 0.512703 0.991054
st.dev. 0.000388 0.00033 0.000642 8.85E-05
200000 mean 0.079403 0.987946 0.512672 0.991062
st.dev. 0.000673 0.0003 0.001053 0.000135
JRC
32
sample size
values for the 10 runs
SI first order (Sobol)
SI total (Sobol)
X1 X2 X1 X2
expected results 7.50E-05 0.9999
computed results, SRS
100 mean 0.00593 1.08249093 -0.00504 0.962934
st.dev. 0.002404 0.28593974 0.002001 0.098833
1000 mean 0.00076 1.02378202 -0.00046 0.994247
st.dev. 0.000699 0.09280417 0.00045 0.06004
10000 mean 0.000223 1.0099725 -0.0001 0.99324
st.dev. 0.000111 0.02603018 0.000117 0.012278
100000 mean 0.00011 1.00061602 3.96E-05 1.001216
st.dev. 4.56E-05 0.00914604 4.93E-05 0.004024
200000 mean 6.89E-05 1.00218169 8.45E-05 1.000042
st.dev. 3.24E-05 0.00878322 3.2E-05 0.002874
JRC
33
JRC
34
Model 5a
sample size
values for the 10 runs
E(Y) V(Y) R2 R2*
expected results 0 427 0.80 0.97
computed results, SRS
100 mean 0.21938 444.1293 0.822881 0.954358
st.dev. 1.557859 122.2117 0.038491 0.017551
1000 mean -0.03435 423.8771 0.804718 0.964167
st.dev. 0.665854 60.15521 0.027893 0.006239
10000 mean -0.13636 413.0685 0.80248 0.965933
st.dev. 0.156762 9.327934 0.005867 0.001637
100000 mean -0.00328 427.8102 0.797911 0.965608
st.dev. 0.097584 6.060007 0.001751 0.000522
sample size
values for the 10 runs Pearson
X1 X2 X3 X4 X5 X6
expected results 0.51 0.32 0.32 0.32 0.32 0.32
computed results, SRS
100 mean 0.504818 0.34628 0.327165 0.32582 0.354869 0.363798
st. dev. 0.079145 0.082656 0.050708 0.042402 0.073094 0.079838
1000 mean 0.522362 0.331995 0.317091 0.320882 0.325374 0.326214
st. dev. 0.021694 0.029659 0.031353 0.02862 0.019041 0.023912
10000 mean 0.528409 0.319446 0.323085 0.32077 0.324395 0.322545
st. dev. 0.005296 0.010862 0.007252 0.004847 0.007199 0.01075
100000 mean 0.526464 0.323863 0.323773 0.32216 0.324646 0.322253
st. dev. 0.001944 0.002196 0.003796 0.002199 0.002545 0.002053
sample size
values for the 10 runs Spearman
X1 X2 X3 X4 X5 X6
expected results 0.59 0.35 0.35 0.35 0.35 0.35
computed results, SRS
100 mean 0.571094 0.353063 0.347579 0.350209 0.363496 0.395896
st. dev. 0.0941 0.076223 0.063281 0.055575 0.065766 0.0989
1000 mean 0.596897 0.351755 0.337485 0.350492 0.354442 0.341692
st. dev. 0.022331 0.027118 0.029559 0.026202 0.028601 0.031232
10000 mean 0.595842 0.347722 0.346281 0.345019 0.351786 0.348901
st. dev. 0.006631 0.010736 0.007965 0.006473 0.00829 0.01005
100000 mean 0.597712 0.3498 0.349645 0.348108 0.351353 0.348599
st. dev. 0.001976 0.002867 0.003693 0.002917 0.003589 0.002067
JRC
35
sample size
values for the 10 runs SRC
X1 X2 X3 X4 X5 X6
expected results
computed results, SRS
100 mean 0.511777 0.329945 0.31815 0.311239 0.339389 0.335829
st. dev. 0.039351 0.031978 0.061502 0.034897 0.050692 0.044024
1000 mean 0.525693 0.328789 0.326305 0.322163 0.321161 0.333661
st. dev. 0.019623 0.017833 0.016438 0.014104 0.017437 0.008681
10000 mean 0.531093 0.32269 0.325267 0.325174 0.323547 0.323544
st. dev. 0.004276 0.002881 0.002975 0.005182 0.003147 0.003736
100000 mean 0.525276 0.322591 0.322714 0.322604 0.322204 0.322342
st. dev. 0.00251 0.001602 0.001537 0.001166 0.001057 0.001306
sample size
values for the 10 runs PCC
X1 X2 X3 X4 X5 X6
expected results 0.76 0.58 0.58 0.58 0.58 0.58
computed results, SRS
100 mean 0.763947 0.607187 0.585912 0.58618 0.615877 0.612787
st. dev. 0.046047 0.065667 0.095057 0.067569 0.074708 0.08167
1000 mean 0.764541 0.596818 0.594207 0.589528 0.588037 0.603098
st. dev. 0.033545 0.044227 0.042866 0.033628 0.038882 0.029047
10000 mean 0.766853 0.587508 0.590575 0.59044 0.588548 0.588498
st. dev. 0.006797 0.007572 0.008416 0.010659 0.007035 0.008181
100000 mean 0.759739 0.583009 0.583156 0.58303 0.58255 0.582717
st. dev. 0.002647 0.00284 0.00308 0.002567 0.002262 0.002942
sample size
values for the 10 runs SRRC
X1 X2 X3 X4 X5 X6
expected results
computed results, SRS
100 mean 0.578192 0.331412 0.341017 0.338486 0.345626 0.359114
st. dev. 0.040938 0.041763 0.037829 0.040067 0.033384 0.047815
1000 mean 0.599803 0.348333 0.346358 0.35145 0.350031 0.349247
st. dev. 0.013159 0.008063 0.006974 0.005769 0.008008 0.007082
10000 mean 0.598751 0.351149 0.34878 0.349904 0.350932 0.349941
st. dev. 0.003428 0.002391 0.002496 0.002559 0.002356 0.002555
100000 mean 0.596429 0.348409 0.348452 0.348605 0.348655 0.348691
st. dev. 0.001199 0.00146 0.00105 0.001011 0.001008 0.00121
JRC
36
sample size
values for the 10 runs PRCC
X1 X2 X3 X4 X5 X6
expected results 0.96 0.88 0.88 0.88 0.88 0.88
computed results, SRS
100 mean 0.936279 0.83317 0.84019 0.840949 0.846606 0.853229
st. dev. 0.017621 0.057885 0.051862 0.047785 0.042748 0.049849
1000 mean 0.953347 0.878513 0.877268 0.88031 0.879382 0.878926
st. dev. 0.008231 0.01627 0.018475 0.017876 0.018985 0.019037
10000 mean 0.955614 0.885155 0.883836 0.884456 0.885032 0.884464
st. dev. 0.00204 0.004558 0.0053 0.005386 0.005121 0.005311
100000 mean 0.954904 0.882737 0.882763 0.882848 0.882877 0.882898
st. dev. 0.0006 0.00144 0.001319 0.001537 0.001271 0.001263
sample size
values for the 10 runs SI first order (Sobol)
X1 X2 X3 X4 X5 X6
expected results 0.287 0.1057 0.1057 0.1057 0.1057 0.1057
computed results, SRS
100 mean 0.388109 0.116343 0.091452 0.108983 0.063358 0.100637
st. dev. 0.214918 0.088415 0.071903 0.050917 0.078553 0.063889
1000 mean 0.289041 0.098196 0.123295 0.111049 0.114437 0.104815
st. dev. 0.054615 0.013041 0.013582 0.020094 0.033719 0.034054
10000 mean 0.298967 0.110893 0.10565 0.110151 0.107855 0.109122
st. dev. 0.00941 0.00889 0.005683 0.006118 0.007048 0.004415
100000 mean 0.287967 0.105394 0.105227 0.105202 0.106336 0.105929
st. dev. 0.005818 0.002281 0.002198 0.002142 0.002258 0.001838
sample size
values for the 10 runs SI total (Sobol)
X1 X2 X3 X4 X5 X6
expected results
computed results, SRS
100 mean 0.304644 0.142847 0.161266 0.171051 0.206709 0.116007
st. dev. 0.105096 0.108985 0.085326 0.112367 0.120751 0.118895
1000 mean 0.376312 0.175425 0.153289 0.138088 0.153598 0.169082
st. dev. 0.029425 0.032007 0.058212 0.057804 0.044783 0.046716
10000 mean 0.396738 0.15017 0.163815 0.156156 0.162215 0.156516
st. dev. 0.015592 0.011247 0.01065 0.006488 0.012964 0.013629
100000 mean 0.394711 0.162374 0.161773 0.162409 0.163043 0.160719
st. dev. 0.005001 0.002942 0.004065 0.003791 0.004699 0.003608
JRC
37
JRC
38
JRC
39
Model 5b
sample size
values for the 10 runs
E(Y) V(Y) R2 R2*
expected results 0 18022 0.81 0.96
computed results, SRS
100 mean -1.26735 16301.004 0.869759 0.953927
st.dev. 12.24993 6248.8911 0.048923 0.014298
1000 mean -0.20124 17369.062 0.825716 0.957788
st.dev. 4.083186 1577.7781 0.018655 0.006043
10000 mean -0.33692 17667.128 0.811964 0.958077
st.dev. 2.003676 539.56896 0.005788 0.001987
Pearson
sample size = 100 sample size = 1000 sample size = 10000
expected results mean sd. dev. mean sd. dev. mean sd. dev.
X1 0.24 0.244025 0.062191 0.2362397 0.026408 0.232252 0.009793
X2 0.24 0.212403 0.098714 0.2493012 0.021823 0.238387 0.011399
X3 0.24 0.291533 0.08994 0.2431911 0.0177 0.237193 0.006526
X4 0.24 0.256098 0.062294 0.2353651 0.021558 0.234398 0.008681
X5 0.24 0.290879 0.076295 0.2480797 0.024849 0.242779 0.005019
X6 0.24 0.225783 0.080181 0.237292 0.028596 0.23542 0.011654
X7 0.24 0.241959 0.072846 0.2446917 0.032656 0.235906 0.007718
X8 0.24 0.21201 0.057513 0.2350596 0.028231 0.238814 0.005304
X9 0.24 0.205671 0.115657 0.2285628 0.020216 0.238952 0.00986
X10 0.24 0.229274 0.092665 0.2476385 0.024096 0.233835 0.009203
X11 0.16 0.181914 0.135796 0.1521533 0.029555 0.163531 0.009783
X12 0.16 0.087994 0.119959 0.1517772 0.02595 0.157375 0.011851
X13 0.16 0.141748 0.080241 0.1596761 0.026367 0.158999 0.012185
X14 0.16 0.118626 0.089119 0.1634418 0.042887 0.158112 0.0103
X15 0.16 0.163685 0.063361 0.1508833 0.041969 0.153222 0.007546
X16 0.16 0.144521 0.065931 0.1729761 0.023012 0.156521 0.010654
X17 0.16 0.186329 0.122614 0.1497955 0.024187 0.155006 0.007614
X18 0.16 0.108147 0.091348 0.1658309 0.032731 0.158363 0.010559
X19 0.16 0.155026 0.084192 0.1506017 0.039771 0.158321 0.008974
X20 0.16 0.191542 0.106503 0.1399129 0.02641 0.160926 0.012689
JRC
40
Spearman
sample size = 100 sample size = 1000 sample size = 10000
expected results mean sd. dev. mean sd. dev. mean sd. dev.
X1 0.26 0.256497 0.083103 0.2542546 0.025844 0.251288 0.011345
X2 0.26 0.247864 0.081899 0.2638508 0.026593 0.257875 0.012906
X3 0.26 0.295975 0.099212 0.2586995 0.020713 0.261198 0.007134
X4 0.26 0.27429 0.079498 0.2567893 0.028756 0.253491 0.007534
X5 0.26 0.310586 0.063466 0.264981 0.023085 0.260939 0.009653
X6 0.26 0.242616 0.074369 0.2543903 0.033217 0.257532 0.009798
X7 0.26 0.230585 0.079969 0.2643035 0.028659 0.255081 0.007941
X8 0.26 0.237307 0.068306 0.2623466 0.035721 0.258857 0.006448
X9 0.26 0.234975 0.100739 0.249127 0.017112 0.258999 0.009697
X10 0.26 0.237541 0.079314 0.2754266 0.015869 0.25625 0.006825
X11 0.17 0.185463 0.170756 0.1768898 0.025573 0.178002 0.008111
X12 0.17 0.105383 0.146891 0.1598323 0.025096 0.173237 0.01321
X13 0.17 0.153042 0.092159 0.1778369 0.031195 0.173394 0.012563
X14 0.17 0.143976 0.104203 0.1670625 0.03826 0.172669 0.01112
X15 0.17 0.171782 0.081624 0.1532091 0.039377 0.16501 0.009554
X16 0.17 0.145294 0.06156 0.175791 0.026485 0.168093 0.010127
X17 0.17 0.195424 0.140784 0.1654522 0.022046 0.169524 0.007932
X18 0.17 0.10468 0.081509 0.1738327 0.032097 0.172708 0.011662
X19 0.17 0.161127 0.105049 0.1636984 0.026373 0.173006 0.008273
X20 0.17 0.191834 0.083545 0.1527849 0.030777 0.172883 0.011156
JRC
41
SRC
sample size = 100 sample size = 1000 sample size = 10000
expected results mean sd. dev. mean sd. dev. mean sd. dev.
X1 0.257287 0.053377 0.2431484 0.016005 0.238351 0.004051
X2 0.240311 0.072747 0.2447163 0.011356 0.237277 0.002768
X3 0.280252 0.052199 0.2385997 0.012383 0.234467 0.005292
X4 0.249619 0.059458 0.2361284 0.012515 0.238534 0.003144
X5 0.254179 0.026467 0.2389861 0.01546 0.239528 0.005988
X6 0.240199 0.053918 0.2419474 0.01336 0.236155 0.005571
X7 0.267946 0.048437 0.2394795 0.006908 0.237338 0.003456
X8 0.236128 0.050845 0.2314276 0.011372 0.238756 0.002911
X9 0.238148 0.042103 0.2354606 0.013226 0.239019 0.004003
X10 0.255968 0.052818 0.2342957 0.015104 0.235732 0.004402
X11 0.168088 0.041891 0.1510314 0.009889 0.157623 0.004819
X12 0.170272 0.067979 0.160652 0.017492 0.156222 0.003768
X13 0.174479 0.041467 0.1558222 0.014544 0.158223 0.003234
X14 0.140627 0.047643 0.1649502 0.015454 0.156209 0.004276
X15 0.164083 0.07461 0.1655993 0.015257 0.157902 0.004142
X16 0.172344 0.061632 0.168015 0.005701 0.158904 0.00474
X17 0.163406 0.048925 0.158977 0.019122 0.157598 0.004841
X18 0.155529 0.052739 0.1609522 0.011932 0.15744 0.0032
X19 0.164326 0.059054 0.1563896 0.016127 0.157572 0.00422
X20 0.175777 0.052224 0.1574505 0.010179 0.159763 0.002227
JRC
42
PCC
sample size = 100 sample size = 1000 sample size = 10000
expected results mean sd. dev. mean sd. dev. mean sd. dev.
X1 0.47 0.544689 0.117698 0.5001562 0.025699 0.481408 0.010338
X2 0.47 0.513829 0.153246 0.5029752 0.027847 0.479759 0.005148
X3 0.47 0.578935 0.113 0.4932904 0.025875 0.475338 0.012159
X4 0.47 0.534146 0.134782 0.4897253 0.034577 0.481673 0.004298
X5 0.47 0.54941 0.087001 0.4943139 0.032076 0.48314 0.012219
X6 0.47 0.517223 0.128646 0.4978977 0.02985 0.477956 0.013032
X7 0.47 0.560612 0.105156 0.4949412 0.019972 0.479857 0.007967
X8 0.47 0.515092 0.127312 0.4820489 0.027693 0.482037 0.007725
X9 0.47 0.518084 0.103068 0.4885291 0.033138 0.48243 0.010383
X10 0.47 0.545067 0.117128 0.4856481 0.033201 0.477352 0.009777
X11 0.32 0.393195 0.113234 0.3380975 0.028588 0.341404 0.010864
X12 0.32 0.391014 0.154823 0.3573128 0.043517 0.338683 0.007304
X13 0.32 0.40671 0.108991 0.347385 0.032181 0.34255 0.008667
X14 0.32 0.345447 0.13144 0.3653736 0.03921 0.3387 0.011076
X15 0.32 0.38134 0.182721 0.3656846 0.030265 0.341958 0.009896
X16 0.32 0.398861 0.138317 0.3715453 0.0234 0.343832 0.010999
X17 0.32 0.385851 0.123998 0.3530547 0.04089 0.341386 0.010814
X18 0.32 0.368139 0.124127 0.3573196 0.028233 0.341029 0.006822
X19 0.32 0.385223 0.142656 0.3473814 0.023387 0.341288 0.010531
X20 0.32 0.408965 0.119479 0.3507898 0.024305 0.345433 0.004565
JRC
43
SRRC
sample size = 100 sample size = 1000 sample size = 10000
expected results mean sd. dev. mean sd. dev. mean sd. dev.
X1 0.270272 0.049656 0.2608008 0.005365 0.257916 0.001468
X2 0.265052 0.049523 0.2583799 0.006966 0.256739 0.002944
X3 0.272278 0.040718 0.2543627 0.009826 0.258264 0.002863
X4 0.262185 0.034142 0.2573089 0.009308 0.257916 0.003424
X5 0.269592 0.034291 0.2550755 0.006392 0.257464 0.002749
X6 0.251931 0.039744 0.2596767 0.009668 0.258121 0.003062
X7 0.253987 0.041555 0.2585316 0.008851 0.256611 0.003004
X8 0.265444 0.036883 0.2590195 0.012218 0.258766 0.003594
X9 0.253544 0.048013 0.2566683 0.008504 0.259084 0.002589
X10 0.262391 0.038062 0.261145 0.009969 0.25823 0.002789
X11 0.193637 0.022251 0.1748692 0.009918 0.171452 0.002684
X12 0.188355 0.025081 0.1703016 0.005492 0.172022 0.003162
X13 0.160927 0.030182 0.1738072 0.00903 0.172585 0.001555
X14 0.172972 0.018838 0.1684056 0.009547 0.170561 0.00319
X15 0.164198 0.041964 0.1689179 0.008474 0.170047 0.001748
X16 0.172697 0.0241 0.1694516 0.006064 0.170722 0.002707
X17 0.16479 0.021401 0.1753598 0.007215 0.17227 0.001856
X18 0.159052 0.042238 0.169071 0.007226 0.171632 0.002725
X19 0.182757 0.035825 0.169077 0.008603 0.172229 0.003284
X20 0.177705 0.020776 0.17118 0.006049 0.171866 0.002601
JRC
44
PRCC
sample size = 100 sample size = 1000 sample size = 10000
expected results mean sd. dev. mean sd. dev. mean sd. dev.
X1 0.76 0.743749 0.091202 0.7834192 0.019743 0.782994 0.007551
X2 0.76 0.735309 0.092817 0.7806392 0.017583 0.781615 0.007496
X3 0.76 0.75433 0.072834 0.7754034 0.023095 0.783406 0.007088
X4 0.76 0.745016 0.055752 0.7788561 0.026687 0.782929 0.00955
X5 0.76 0.755049 0.060191 0.7771812 0.016499 0.782406 0.007738
X6 0.76 0.723802 0.094924 0.7810255 0.026577 0.783209 0.007648
X7 0.76 0.727486 0.101901 0.7809286 0.011832 0.781479 0.00665
X8 0.76 0.747843 0.057907 0.7807563 0.023992 0.783945 0.008603
X9 0.76 0.724784 0.107425 0.7784622 0.020079 0.784325 0.008197
X10 0.76 0.741878 0.066135 0.7829103 0.020706 0.783391 0.00697
X11 0.61 0.632515 0.083789 0.6450398 0.02585 0.64179 0.00997
X12 0.61 0.62152 0.092447 0.6361715 0.018811 0.642986 0.012272
X13 0.61 0.56192 0.096195 0.6434103 0.017584 0.644264 0.009475
X14 0.61 0.599956 0.082146 0.6311165 0.0285 0.639766 0.0113
X15 0.61 0.566245 0.10793 0.6319853 0.038115 0.638716 0.008845
X16 0.61 0.593084 0.096922 0.6340681 0.025193 0.640155 0.008215
X17 0.61 0.575302 0.089861 0.6463909 0.034371 0.643652 0.009038
X18 0.61 0.549452 0.09559 0.6327258 0.028065 0.642149 0.010764
X19 0.61 0.60954 0.096289 0.6322821 0.034919 0.64341 0.012457
X20 0.61 0.604107 0.086391 0.6378354 0.02711 0.642621 0.010368
JRC
45
SI first order
sample size = 100 sample size = 1000 sample size = 10000
expected results mean sd. dev. mean sd. dev. mean sd. dev.
X1 0.0562 0.069266 0.035108 0.0605084 0.010812 0.054435 0.003979
X2 0.0562 0.069876 0.044044 0.0672876 0.014197 0.058215 0.00434
X3 0.0562 0.096475 0.076133 0.0620062 0.015176 0.055273 0.004337
X4 0.0562 0.108579 0.111176 0.0559661 0.010641 0.05632 0.004871
X5 0.0562 0.106117 0.088003 0.0643538 0.012642 0.056192 0.003938
X6 0.0562 0.075397 0.07769 0.0610131 0.012547 0.058141 0.003744
X7 0.0562 0.05679 0.061086 0.0583436 0.009221 0.055692 0.004501
X8 0.0562 0.091701 0.081279 0.0656898 0.008342 0.056332 0.004714
X9 0.0562 0.086976 0.083313 0.0561658 0.017421 0.057675 0.006157
X10 0.0562 0.081874 0.048874 0.0592313 0.01204 0.055307 0.004367
X11 0.025 0.048964 0.045328 0.0208927 0.010562 0.025299 0.001098
X12 0.025 0.033713 0.036532 0.0254369 0.009342 0.02658 0.0038
X13 0.025 0.028037 0.028841 0.030762 0.006431 0.025996 0.004473
X14 0.025 0.019459 0.040294 0.0285852 0.00677 0.025606 0.001413
X15 0.025 0.019939 0.027176 0.0235418 0.007565 0.025976 0.003072
X16 0.025 0.010628 0.036058 0.0231617 0.008406 0.025439 0.002856
X17 0.025 0.046952 0.05049 0.0221737 0.007565 0.025583 0.003323
X18 0.025 0.032303 0.042291 0.0242037 0.008936 0.025158 0.00237
X19 0.025 0.013693 0.035961 0.0286544 0.012877 0.026726 0.001563
X20 0.025 0.046913 0.032318 0.0259001 0.008861 0.026437 0.002579
JRC
46
SI total
sample size = 100 sample size = 1000 sample size = 10000
expected results mean sd. dev. mean sd. dev. mean sd. dev.
X1 0.060119 0.064444 0.0777746 0.030887 0.082993 0.009294
X2 0.061579 0.054444 0.0923132 0.031344 0.083283 0.01142
X3 0.059185 0.099584 0.096461 0.018211 0.076707 0.005749
X4 0.11974 0.080165 0.0753325 0.020247 0.082 0.012469
X5 0.119264 0.084336 0.1003567 0.036006 0.086695 0.008484
X6 0.077269 0.080366 0.0782087 0.023409 0.079941 0.013852
X7 0.082566 0.067527 0.0731696 0.031421 0.083473 0.01191
X8 0.038938 0.087724 0.0704035 0.032009 0.081979 0.007158
X9 0.064848 0.103993 0.09279 0.014953 0.083927 0.009614
X10 0.083014 0.107755 0.0957531 0.029025 0.078735 0.007971
X11 0.059462 0.070798 0.0273936 0.018716 0.037942 0.010807
X12 0.021688 0.04888 0.0379926 0.020964 0.034203 0.008351
X13 0.025194 0.047975 0.0332864 0.015398 0.036348 0.006918
X14 0.032888 0.04684 0.0448812 0.022059 0.037251 0.006432
X15 0.049138 0.047227 0.0431615 0.014044 0.037278 0.00489
X16 0.028017 0.054554 0.041139 0.021258 0.036961 0.005778
X17 0.052533 0.079783 0.0313351 0.019343 0.035843 0.008022
X18 0.00704 0.082006 0.0324105 0.018637 0.034639 0.008356
X19 0.053332 0.020793 0.0341688 0.034916 0.031904 0.006407
X20 0.054794 0.080322 0.0346937 0.022024 0.04058 0.005818
JRC
47
JRC
48
Model 6a
sample size
values for the 10 runs
E(Y) V(Y) R2 R2*
expected results 1.0303 0.070442 0.98 0.99
computed results, SRS
100 mean 1.026923 0.07197 0.98342 0.978809
st.dev. 0.020063 0.00657 0.002364 0.006908
1000 mean 1.029394 0.070228 0.983331 0.988501
st.dev. 0.011649 0.002951 0.000574 0.001516
10000 mean 1.029776 0.070509 0.983514 0.990123
st.dev. 0.002455 0.000641 0.0002 0.00055
100000 mean 1.030235 0.070461 0.983531 0.990227
st.dev. 0.000871 0.000187 5.51E-05 0.000115
200000 mean 1.030286 0.07045 0.983509 0.990183
st.dev. 0.000705 0.000222 4.91E-05 6.26E-05
sample size
values for the 10 runs
Pearson Spearman
X1 X2 X1 X2
expected results -0.45 0.89 -0.42 0.90
computed results, SRS
100 mean -0.47474 0.884731 -0.44961 0.892878
st.dev. 0.051861 0.012709 0.055134 0.015822
1000 mean -0.44728 0.884601 -0.42144 0.900069
st.dev. 0.031769 0.005921 0.033028 0.006065
10000 mean -0.44945 0.885119 -0.42536 0.900567
st.dev. 0.005419 0.00156 0.006654 0.001508
100000 mean -0.44933 0.884281 -0.42544 0.899745
st.dev. 0.002263 0.0003 0.002419 0.000156
200000 mean -0.44946 0.88418 -0.42557 0.899635
st.dev. 0.001959 0.000339 0.00207 0.000352
JRC
49
sample size
values for the 10 runs
SRC PCC
X1 X2 X1 X2
expected results -0.98 0.99
computed results, SRS
100 mean -0.44812 0.870686 -0.96069 0.989146
st.dev. 0.025488 0.027578 0.006613 0.001835
1000 mean -0.44825 0.884998 -0.9608 0.989496
st.dev. 0.012062 0.016432 0.002693 0.000649
10000 mean -0.4473 0.88403 -0.96118 0.989616
st.dev. 0.002983 0.002752 0.000514 0.000148
100000 mean -0.44897 0.8841 -0.96149 0.989628
st.dev. 0.000584 0.001171 0.000149 5.66E-05
200000 mean -0.44915 0.884024 -0.96147 0.989613
st.dev. 0.000674 0.001013 0.000166 4.9E-05
sample size
values for the 10 runs
SRRC PRCC
X1 X2 X1 X2
expected results -0.97 0.99
computed results, SRS
100 mean -0.42592 0.88145 -0.94625 0.986586
st.dev. 0.030136 0.026276 0.014969 0.004287
1000 mean -0.42243 0.900463 -0.96918 0.992972
st.dev. 0.012686 0.015461 0.004158 0.000942
10000 mean -0.4232 0.899542 -0.97352 0.993952
st.dev. 0.003119 0.003075 0.001445 0.000333
100000 mean -0.42507 0.899572 -0.974 0.994015
st.dev. 0.000276 0.001127 0.000289 6.8E-05
200000 mean -0.42525 0.899487 -0.97392 0.993988
st.dev. 0.000741 0.00096 0.000184 3.35E-05
JRC
50
sample size
values for the 10 runs
SI first order (Sobol)
SI total (Sobol)
X1 X2 X1 X2
expected results 0.2023 0.7690
computed results, SRS
100 mean 0.241215 1.065869 0.201815 0.609147
st.dev. 0.236617 0.511374 0.288289 0.436161
1000 mean 0.201877 0.798315 0.20757 0.787548
st.dev. 0.059392 0.165625 0.071194 0.16851
10000 mean 0.209382 0.791091 0.205412 0.789588
st.dev. 0.034994 0.06107 0.034741 0.050666
100000 mean 0.204775 0.789398 0.211377 0.794392
st.dev. 0.010483 0.014537 0.010841 0.016612
200000 mean 0.203326 0.787725 0.213473 0.797349
st.dev. 0.005759 0.014906 0.006212 0.015486
JRC
51
JRC
52
Model 6b
sample size
values for the 10 runs
E(Y) V(Y) R2 R2*
expected results 2.0167 6.9012 0.675 0.98
computed results, SRS
100 mean 1.842632 6.217633 0.667145 0.964597
st.dev. 0.198897 2.158196 0.063311 0.012112
1000 mean 2.04614 6.907842 0.678315 0.97745
st.dev. 0.079046 0.807258 0.014298 0.002207
10000 mean 2.019818 6.931499 0.675905 0.979893
st.dev. 0.026985 0.230862 0.004854 0.000595
100000 mean 2.014895 6.873053 0.673503 0.979423
st.dev. 0.006124 0.039505 0.001222 0.000242
200000 mean 2.016555 6.904608 0.673637 0.979616
st.dev. 0.007318 0.061798 0.001191 0.000201
sample size
values for the 10 runs
Pearson Spearman
X1 X2 X1 X2
expected results -0.47 0.67 -0.43 0.89
computed results, SRS
100 mean -0.43262 0.657991 -0.37311 0.879666
st.dev. 0.065807 0.05141 0.115052 0.026998
1000 mean -0.47222 0.671902 -0.42928 0.88841
st.dev. 0.013847 0.005113 0.025286 0.006535
10000 mean -0.47036 0.675997 -0.43219 0.892144
st.dev. 0.005678 0.003057 0.005837 0.001122
100000 mean -0.46798 0.673803 -0.43039 0.890834
st.dev. 0.001079 0.000884 0.002278 0.000457
200000 mean -0.46849 0.674005 -0.43097 0.891087
st.dev. 0.001244 0.001011 0.001541 0.000322
JRC
53
sample size
values for the 10 runs
SRC PCC
X1 X2 X1 X2
expected results -0.64 0.76
computed results, SRS
100 mean -0.48537 0.693378 -0.64221 0.762912
st.dev. 0.031483 0.072578 0.044285 0.057708
1000 mean -0.4763 0.674854 -0.64273 0.765204
st.dev. 0.016182 0.016363 0.020556 0.014122
10000 mean -0.4679 0.674286 -0.63494 0.764077
st.dev. 0.003564 0.0034 0.005015 0.003322
100000 mean -0.4685 0.674168 -0.63404 0.762855
st.dev. 0.000845 0.001126 0.001147 0.001056
200000 mean -0.46835 0.673909 -0.634 0.762798
st.dev. 0.000857 0.001131 0.00103 0.001017
sample size
values for the 10 runs
SRRC PRCC
X1 X2 X1 X2
expected results -0.95 0.99
computed results, SRS
100 mean -0.43766 0.908935 -0.91678 0.978756
st.dev. 0.049824 0.054326 0.025575 0.007708
1000 mean -0.43386 0.890711 -0.94505 0.986077
st.dev. 0.011188 0.011707 0.003181 0.001241
10000 mean -0.42893 0.890575 -0.94947 0.98756
st.dev. 0.001789 0.00262 0.001147 0.00033
100000 mean -0.43109 0.891169 -0.94885 0.987291
st.dev. 0.000932 0.001021 0.000611 0.000133
200000 mean -0.43079 0.890997 -0.94923 0.987404
st.dev. 0.000563 0.00078 0.000446 0.000132
JRC
54
sample size
values for the 10 runs
SI first order (Sobol)
SI total (Sobol)
X1 X2 X1 X2
expected results 0.26191 0.51098
computed results, SRS
100 mean 0.474913 0.786988 0.369355 0.80449
st.dev. 0.340499 0.548319 0.23161 0.205707
1000 mean 0.298423 0.504775 0.461239 0.74177
st.dev. 0.07584 0.089479 0.066928 0.055559
10000 mean 0.262736 0.494562 0.485049 0.746427
st.dev. 0.018264 0.031236 0.019991 0.018229
100000 mean 0.263781 0.514685 0.4867 0.739988
st.dev. 0.004344 0.005421 0.003501 0.005099
200000 mean 0.260368 0.510987 0.488963 0.739961
st.dev. 0.003506 0.006157 0.004779 0.005411
JRC
55
JRC
56
Model 7
sample size values for the 10 runs
E(Y) V(Y) R2 R2*
expected results 1 0 0
computed results, SRS
100 mean 0.98936375 0.4525659 0.09776591 0.10017267
st.dev. 0.04816007 0.05785684 0.0522737 0.05262957
1000 mean 0.9966547 0.45506578 0.00932941 0.00833276
st.dev. 0.01591101 0.01344016 0.00396665 0.00325208
10000 mean 1.00058116 0.46482934 0.00094423 0.0009098
st.dev. 0.00822015 0.00843884 0.00041414 0.00041086
100000 mean 1.00013684 0.46543067 8.4003E-05 8.161E-05
st.dev. 0.00165622 0.0017084 3.3244E-05 3.0605E-05
200000 mean 0.99971825 0.46528397
st.dev. 0.00143443 0.00096541
Pearson
sample size = 100 sample size = 1000 sample size = 10000 sample size = 100000 sample size = 200000
mean sd. dev. mean sd. dev. mean sd. dev. mean sd. dev. mean sd. dev.
X1 -0.0527 0.0865 -0.0034 0.0423 -0.0027 0.0126 -0.0006 0.0031 0.0002 0.0017
X2 -0.0425 0.1331 0.0139 0.0355 0.0042 0.0116 -0.0027 0.0023 0.0000 0.0025
X3 -0.0415 0.1303 -0.0066 0.0329 0.0022 0.0122 -0.0002 0.0035 -0.0004 0.0033
X4 -0.0142 0.1245 -0.0030 0.0224 0.0021 0.0121 -0.0019 0.0035 0.0001 0.0017
X5 -0.0370 0.1368 -0.0102 0.0271 -0.0007 0.0077 -0.0018 0.0034 0.0008 0.0033
X6 0.0218 0.0579 0.0072 0.0422 -0.0055 0.0082 0.0007 0.0039 0.0003 0.0030
X7 -0.0108 0.1226 0.0055 0.0423 0.0042 0.0112 0.0001 0.0029 0.0007 0.0026
X8 0.0129 0.0627 -0.0100 0.0309 0.0011 0.0112 0.0004 0.0016 0.0005 0.0025
Spearman
sample size = 100 sample size = 1000 sample size = 10000 sample size = 100000 sample size = 200000
mean sd. dev. mean sd. dev. mean sd. dev. mean sd. dev. mean sd. dev.
X1 -0.0776 0.1386 -0.0030 0.0358 -0.0010 0.0123 -0.0017 0.0030 0.0005 0.0024
X2 -0.0507 0.1167 0.0162 0.0316 0.0027 0.0111 -0.0024 0.0022 0.0000 0.0020
X3 -0.0338 0.1203 -0.0023 0.0308 0.0016 0.0126 0.0000 0.0031 -0.0004 0.0034
X4 -0.0179 0.1210 -0.0050 0.0257 0.0019 0.0128 -0.0018 0.0038 -0.0001 0.0018
X5 -0.0400 0.1221 -0.0127 0.0286 -0.0005 0.0080 -0.0015 0.0036 0.0009 0.0035
X6 0.0211 0.0794 0.0048 0.0414 -0.0064 0.0084 0.0007 0.0036 0.0003 0.0025
X7 -0.0029 0.1133 0.0029 0.0376 0.0038 0.0097 -0.0001 0.0030 0.0008 0.0027
X8 0.0048 0.0700 -0.0060 0.0310 0.0022 0.0105 0.0005 0.0013 0.0006 0.0021
JRC
57
SRC
sample size = 100 sample size = 1000 sample size = 10000 sample size = 100000 sample size = 200000
mean sd. dev. mean sd. dev. mean sd. dev. mean sd. dev. mean sd. dev.
X1 -0.0555 0.0772 -0.0046 0.0394 -0.0027 0.0126 -0.0006 0.0031 0.0002 0.0017
X2 -0.0452 0.1376 0.0116 0.0355 0.0039 0.0117 -0.0027 0.0023 0.0000 0.0025
X3 -0.0422 0.1370 -0.0075 0.0337 0.0023 0.0120 -0.0002 0.0035 -0.0004 0.0033
X4 -0.0386 0.1253 -0.0033 0.0226 0.0022 0.0120 -0.0019 0.0035 0.0001 0.0017
X5 -0.0176 0.1350 -0.0095 0.0276 -0.0006 0.0076 -0.0018 0.0034 0.0008 0.0033
X6 0.0237 0.0776 0.0066 0.0430 -0.0054 0.0081 0.0007 0.0039 0.0003 0.0030
X7 -0.0032 0.1325 0.0062 0.0416 0.0043 0.0110 0.0001 0.0029 0.0007 0.0026
X8 0.0294 0.0605 -0.0104 0.0308 0.0010 0.0111 0.0004 0.0016 0.0005 0.0025
PCC
sample size = 100 sample size = 1000 sample size = 10000 sample size = 100000 sample size = 200000
mean sd. dev. mean sd. dev. mean sd. dev. mean sd. dev. mean sd. dev.
X1 -0.0561 0.0783 -0.0046 0.0393 -0.0027 0.0126 -0.0006 0.0031 0.0002 0.0017
X2 -0.0456 0.1387 0.0116 0.0356 0.0039 0.0117 -0.0027 0.0023 0.0000 0.0025
X3 -0.0435 0.1403 -0.0074 0.0336 0.0023 0.0120 -0.0002 0.0035 -0.0004 0.0033
X4 -0.0397 0.1257 -0.0033 0.0226 0.0022 0.0120 -0.0019 0.0035 0.0001 0.0017
X5 -0.0160 0.1371 -0.0095 0.0277 -0.0006 0.0076 -0.0018 0.0034 0.0008 0.0033
X6 0.0245 0.0782 0.0065 0.0429 -0.0054 0.0081 0.0007 0.0039 0.0003 0.0030
X7 -0.0042 0.1318 0.0062 0.0417 0.0043 0.0110 0.0001 0.0029 0.0007 0.0026
X8 0.0296 0.0608 -0.0104 0.0307 0.0010 0.0111 0.0004 0.0016 0.0005 0.0025
SRRC
sample size = 100 sample size = 1000 sample size = 10000 sample size = 100000
mean sd. dev. mean sd. dev. mean sd. dev. mean sd. dev.
X1 -0.0785 0.1294 -0.0041 0.0334 -0.0010 0.0123 -0.0017 0.0030
X2 -0.0535 0.1141 0.0142 0.0315 0.0025 0.0112 -0.0024 0.0022
X3 -0.0373 0.1221 -0.0024 0.0311 0.0017 0.0125 0.0000 0.0031
X4 -0.0475 0.1179 -0.0058 0.0258 0.0020 0.0126 -0.0018 0.0038
X5 -0.0211 0.1266 -0.0118 0.0287 -0.0004 0.0079 -0.0015 0.0036
X6 0.0222 0.0924 0.0048 0.0424 -0.0063 0.0083 0.0007 0.0036
X7 0.0064 0.1153 0.0037 0.0369 0.0038 0.0095 -0.0002 0.0030
X8 0.0190 0.0638 -0.0069 0.0308 0.0021 0.0104 0.0005 0.0013
JRC
58
PRCC
sample size = 100 sample size = 1000 sample size = 10000 sample size = 100000
mean sd. dev. mean sd. dev. mean sd. dev. mean sd. dev.
X1 -0.0795 0.1300 -0.0042 0.0333 -0.0010 0.0123 -0.0017 0.0030
X2 -0.0541 0.1155 0.0142 0.0315 0.0025 0.0112 -0.0024 0.0022
X3 -0.0393 0.1245 -0.0023 0.0309 0.0017 0.0125 0.0000 0.0031
X4 -0.0472 0.1206 -0.0058 0.0258 0.0020 0.0126 -0.0018 0.0038
X5 -0.0195 0.1284 -0.0118 0.0287 -0.0004 0.0079 -0.0015 0.0036
X6 0.0230 0.0931 0.0047 0.0423 -0.0063 0.0083 0.0007 0.0036
X7 0.0062 0.1151 0.0037 0.0370 0.0038 0.0095 -0.0002 0.0030
X8 0.0190 0.0639 -0.0069 0.0308 0.0021 0.0104 0.0005 0.0013
SI first order (Sobol)
sample size = 100 sample size = 1000 sample size = 10000 sample size = 100000 sample size = 200000
expected results mean sd. dev. mean sd. dev. mean sd. dev. mean sd. dev. mean sd. dev.
X1 0.7165 0.8899 0.2988 0.7552 0.0848 0.7075 0.0288 0.7160 0.0105 0.7144 0.0037
X2 0.1791 0.2340 0.1532 0.2112 0.0351 0.1778 0.0095 0.1792 0.0054 0.1785 0.0010
X3 0.0237 0.0412 0.0403 0.0287 0.0143 0.0224 0.0058 0.0239 0.0016 0.0236 0.0005
X4 0.0072 0.0220 0.0246 0.0063 0.0080 0.0080 0.0023 0.0070 0.0007 0.0072 0.0003
X5 0.0001 0.0237 0.0047 0.0023 0.0007 0.0003 0.0002 0.0001 0.0001 0.0001 0.0000
X6 0.0001 0.0223 0.0043 0.0019 0.0006 0.0003 0.0002 0.0001 0.0001 0.0001 0.0000
X7 0.0001 0.0228 0.0047 0.0024 0.0010 0.0003 0.0003 0.0001 0.0001 0.0001 0.0000
X8 0.0001 0.0230 0.0040 0.0020 0.0009 0.0003 0.0003 0.0001 0.0001 0.0001 0.0000
SI total (Sobol)
sample size = 100 sample size = 1000 sample size = 10000 sample size = 100000 sample size = 200000
mean sd. dev. mean sd. dev. mean sd. dev. mean sd. dev. mean sd. dev.
X1 0.7017 0.1858 0.7856 0.0797 0.7868 0.0231 0.7845 0.0087 0.7874 0.0018
X2 0.1804 0.1642 0.2224 0.0411 0.2375 0.0116 0.2422 0.0059 0.2418 0.0030
X3 0.0272 0.0630 0.0322 0.0209 0.0373 0.0097 0.0342 0.0027 0.0344 0.0006
X4 0.0062 0.0358 0.0121 0.0110 0.0091 0.0029 0.0109 0.0016 0.0106 0.0003
X5 -0.0238 0.0068 -0.0022 0.0008 -0.0002 0.0002 0.0001 0.0001 0.0002 0.0004
X6 -0.0218 0.0058 -0.0015 0.0013 -0.0001 0.0002 0.0001 0.0001 0.0002 0.0004
X7 -0.0214 0.0050 -0.0021 0.0009 -0.0002 0.0003 0.0000 0.0001 0.0002 0.0004
X8 -0.0205 0.0041 -0.0020 0.0012 -0.0001 0.0004 0.0001 0.0001 0.0002 0.0004
JRC
59
JRC
60
JRC
61
Model 9
sample size
values for the 10 runs
E(Y) V(Y) R2 R2*
expected results 3.5 13.845 0.19 0.19
computed results, SRS
100 mean 3.6918 13.2248 0.2301 0.2159
st.dev. 0.26973 1.49374 0.0761 0.0741
1000 mean 3.4664 13.4126 0.1833 0.1812
st.dev. 0.11785 0.57131 0.0185 0.0201
10000 mean 3.49506 13.8395 0.1937 0.1943
st.dev. 0.03967 0.21148 0.0072 0.0089
100000 mean 3.5032 13.8519 0.1906 0.1913
st.dev. 0.01 0.07829 0.0015 0.0016
200000 mean 3.50266 13.8617 0.1916 0.1921
st.dev. 0.00699 0.04682 0.0012 0.0015
sample size values for the 10 runs
Pearson Spearman
X1 X2 X3 X1 X2 X3
expected results
computed results, SRS
100 mean 0.45126 0.050637 -0.0389 0.4406 0.0398 -0.0469
st.dev. 0.07957 0.132341 0.09551 0.08316 0.1141 0.0746
1000 mean 0.42455 -0.0002 0.03476 0.42223 -0.0018 0.0349
st.dev. 0.02077 0.023419 0.04094 0.02344 0.0289 0.0316
10000 mean 0.43974 -0.00115 0.00373 0.44053 -0.0005 0.0033
st.dev. 0.00818 0.01118 0.0128 0.01003 0.0115 0.0102
100000 mean 0.43658 0.000814 -0.0009 0.43733 0.0012 -0.0012
st.dev. 0.00173 0.003069 0.00378 0.00189 0.0028 0.0031
200000 mean 0.4377 0.000402 -0.001 0.4383 0.0005 -0.0009
st.dev. 0.00132 0.00211 0.0029 0.00168 0.0022 0.0025
JRC
62
sample size values for the 10 runs
SRC PCC
X1 X2 X3 X1 X2 X3
expected results 0.435 0 0
computed results, SRS
100 mean 0.44655 0.0278 -0.0218 0.4489 0.0298 -0.0262
st.dev. 0.08697 0.1136 0.08897 0.086 0.1274 0.10031
1000 mean 0.4241 -0.0017 0.02925 0.4246 -0.002 0.03241
st.dev. 0.02044 0.0256 0.03667 0.0205 0.0283 0.04051
10000 mean 0.43973 0.0007 0.00399 0.4398 0.0007 0.00442
st.dev. 0.00814 0.0101 0.01175 0.0081 0.0112 0.01306
100000 mean 0.43658 0.0007 -1E-04 0.4366 0.0007 -0.0001
st.dev. 0.00171 0.0033 0.00347 0.0017 0.0037 0.00386
200000 mean 0.4377 1E-05 -0.0006 0.4377 1E-05 -0.0007
st.dev. 0.00132 0.0016 0.00259 0.0013 0.0017 0.00288
sample size values for the 10 runs
SRRC PRCC
X1 X2 X3 X1 X2 X3
expected results 0.436 0 0
computed results, SRS
100 mean 0.43811 0.015258 -0.0302 0.43857 0.0156 -0.0338
st.dev. 0.09017 0.10611 0.06459 0.08932 0.1179 0.0715
1000 mean 0.42187 -0.00314 0.02938 0.42226 -0.0035 0.0325
st.dev. 0.02323 0.030601 0.02779 0.02323 0.0338 0.0306
10000 mean 0.44052 0.00136 0.00353 0.44054 0.0015 0.0039
st.dev. 0.01 0.01051 0.00905 0.00999 0.0117 0.0101
100000 mean 0.43734 0.001027 -0.0004 0.43733 0.0011 -0.0005
st.dev. 0.00188 0.003344 0.0032 0.00188 0.0037 0.0036
200000 mean 0.43829 8.72E-05 -0.0005 0.43829 1E-04 -0.0006
st.dev. 0.00168 0.001562 0.00217 0.00168 0.0017 0.0024
JRC
63
sample size values for the 10 runs
SI first order (Sobol) SI total (Sobol)
X1 X2 X3 X1 X2 X3
expected results 0.3139 0.4424 0 0.5596 0.4424 0.2437
computed results, SRS
100 mean 0.22094 0.4859 0.07864 0.5968 0.509 0.20695
st.dev. 0.11905 0.124 0.13491 0.1227 0.1182 0.08827
1000 mean 0.35667 0.4863 -0.0016 0.5674 0.4452 0.24839
st.dev. 0.05938 0.0527 0.02425 0.0626 0.0254 0.03746
10000 mean 0.31893 0.4386 -0.0002 0.5518 0.4413 0.23473
st.dev. 0.01584 0.0143 0.01455 0.0175 0.0094 0.01413
100000 mean 0.31344 0.4413 -0.0009 0.5613 0.4434 0.24468
st.dev. 0.00611 0.006 0.00237 0.0041 0.0028 0.0033
200000 mean 0.31273 0.4418 -3E-05 0.5576 0.4422 0.24417
st.dev. 0.00407 0.002 0.00211 0.0023 0.0019 0.00184
JRC
64
JRC
65
Model 10
sample size
values for the 10 runs
E(Y) V(Y) R2 R2*
expected results
computed results, SRS
100 mean 30.89064 1189.586 0.511788 0.554804
st.dev. 3.467383 299.4291 0.067566 0.08072
1000 mean 32.36634 1036.52 0.42595 0.481236
st.dev. 1.158452 73.38625 0.026761 0.018854
10000 mean 32.42767 1045.022 0.434386 0.48958
st.dev. 0.309014 15.08042 0.008714 0.008245
Pearson
sample size = 100 sample size = 1000 sample size = 10000
mean sd. dev. mean sd. dev. mean sd. dev.
X1 -0.0554 0.133167 -0.06097 0.026697 -0.0678 0.010875
X2 -0.10324 0.081433 -0.08409 0.027103 -0.08763 0.01349
X3 0.080275 0.127839 0.086016 0.023491 0.097942 0.012169
X4 -0.10641 0.10441 -0.09319 0.044366 -0.08014 0.009728
X5 0.028772 0.065831 0.103038 0.023968 0.111851 0.010688
X6 -0.02299 0.062304 -0.00061 0.036324 -0.00379 0.010748
X7 0.195746 0.110822 0.205281 0.027035 0.214638 0.005649
X8 0.294989 0.070306 0.324673 0.027876 0.337634 0.008603
X9 0.341889 0.086349 0.342033 0.027403 0.356015 0.008164
X10 0.330334 0.052621 0.311962 0.02737 0.315351 0.00898
X11 0.04935 0.096686 0.027786 0.018169 0.007909 0.008641
X12 -0.01409 0.138436 -0.01457 0.030044 -0.0151 0.008458
X13 -0.03365 0.113416 -0.0461 0.028018 -0.04005 0.014087
X14 -0.02194 0.11051 -0.02492 0.031688 -0.0115 0.009322
X15 0.022085 0.09667 0.000119 0.023769 0.009921 0.00824
X16 0.0799 0.066737 0.049507 0.035003 0.046812 0.009496
X17 -0.06152 0.084267 0.017868 0.033346 0.006417 0.009041
X18 -0.01771 0.069163 0.016166 0.035508 0.01277 0.007445
X19 -0.05008 0.096395 -0.0495 0.02578 -0.04642 0.011052
X20 -0.02145 0.072733 -0.01964 0.019934 -0.02246 0.01061
JRC
66
Spearman
sample size = 100 sample size = 1000 sample size = 10000
mean sd. dev. mean sd. dev. mean sd. dev.
X1 -0.02716 0.135888 -0.05003 0.023773 -0.05765 0.010026
X2 -0.09472 0.105698 -0.07947 0.030905 -0.08162 0.012507
X3 0.087423 0.140344 0.097836 0.021885 0.106382 0.009083
X4 -0.08847 0.103369 -0.08585 0.043637 -0.0711 0.01018
X5 0.057443 0.052463 0.108919 0.023038 0.121805 0.010307
X6 -0.03562 0.05058 0.002533 0.037212 0.001042 0.010638
X7 0.195616 0.087762 0.220665 0.020821 0.220587 0.007243
X8 0.319399 0.087251 0.34916 0.028651 0.361494 0.00687
X9 0.371225 0.087714 0.368647 0.023808 0.382985 0.007156
X10 0.350766 0.0688 0.332729 0.031626 0.337853 0.008222
X11 0.070819 0.096887 0.026723 0.019662 0.007002 0.009523
X12 -0.00903 0.142135 -0.01736 0.03022 -0.01716 0.008433
X13 -0.03102 0.106714 -0.04607 0.030735 -0.04332 0.015006
X14 -0.02785 0.105366 -0.02175 0.029078 -0.01277 0.00904
X15 0.025325 0.09392 -0.00111 0.017239 0.012024 0.006168
X16 0.095448 0.054937 0.051877 0.038074 0.050207 0.008754
X17 -0.05409 0.092704 0.015563 0.024225 0.005816 0.009701
X18 0.001435 0.049643 0.017363 0.03454 0.014913 0.007456
X19 -0.06185 0.095608 -0.05685 0.027467 -0.04994 0.011885
X20 -0.03184 0.072324 -0.01891 0.015922 -0.02337 0.009746
SRC
sample size = 100 sample size = 1000 sample size = 10000
mean sd. dev. mean sd. dev. mean sd. dev.
X1 -0.07057 0.128704 -0.06713 0.01966 -0.06691 0.009275
X2 -0.14726 0.052549 -0.09572 0.027778 -0.08366 0.011346
X3 0.087599 0.106154 0.089843 0.013732 0.095967 0.008665
X4 -0.09668 0.076537 -0.09515 0.033578 -0.07993 0.008793
X5 0.033753 0.079453 0.11171 0.01821 0.110676 0.007239
X6 -0.05141 0.082555 -0.0034 0.036044 -0.00513 0.008725
X7 0.17519 0.085199 0.197611 0.03002 0.21226 0.006741
X8 0.304678 0.046098 0.329415 0.026246 0.337417 0.009351
X9 0.348119 0.07109 0.3462 0.033905 0.355891 0.008321
X10 0.355647 0.078198 0.324557 0.02312 0.316295 0.005378
X11 0.048538 0.0963 0.018131 0.025666 0.006511 0.007375
X12 -0.02242 0.109586 -0.01504 0.023068 -0.01616 0.009095
X13 -0.00611 0.121202 -0.04095 0.0205 -0.04219 0.007265
X14 -0.0051 0.084622 -0.02015 0.012121 -0.00975 0.008485
X15 -0.01047 0.098384 0.006059 0.020233 0.005981 0.007099
X16 0.088071 0.090843 0.036834 0.023731 0.044069 0.008987
X17 -0.03738 0.053003 0.009174 0.036246 0.009251 0.005048
X18 -0.00962 0.068857 0.011692 0.029896 0.012131 0.007838
X19 -0.04084 0.08171 -0.05041 0.01554 -0.04803 0.005171
X20 -0.00912 0.07124 -0.01882 0.023628 -0.02585 0.006827
JRC
67
PCC
sample size = 100 sample size = 1000 sample size = 10000
mean sd. dev. mean sd. dev. mean sd. dev.
X1 -0.08313 0.153306 -0.08724 0.024982 -0.0885 0.012197
X2 -0.18624 0.063453 -0.12433 0.035935 -0.11039 0.014569
X3 0.115413 0.136621 0.116927 0.017415 0.126467 0.011523
X4 -0.12354 0.099515 -0.1232 0.043128 -0.10549 0.010823
X5 0.041782 0.103301 0.144706 0.02448 0.145465 0.009539
X6 -0.06345 0.110341 -0.00465 0.046883 -0.00677 0.011618
X7 0.215363 0.10088 0.250388 0.039014 0.271362 0.007694
X8 0.360683 0.040985 0.39514 0.028137 0.408966 0.011485
X9 0.407674 0.07623 0.412319 0.039055 0.427413 0.010442
X10 0.412846 0.077788 0.390851 0.02668 0.387373 0.00665
X11 0.062081 0.1263 0.023954 0.033505 0.008651 0.009756
X12 -0.02914 0.137361 -0.01978 0.030486 -0.02145 0.012077
X13 -0.01353 0.153322 -0.05379 0.027708 -0.05594 0.00957
X14 -0.00461 0.110375 -0.02633 0.015665 -0.01297 0.011321
X15 -0.0166 0.129369 0.008206 0.026114 0.007924 0.009387
X16 0.112676 0.115196 0.048341 0.031472 0.058452 0.011963
X17 -0.04514 0.065047 0.011846 0.047008 0.012297 0.006743
X18 -0.01265 0.090591 0.015205 0.03884 0.016076 0.010361
X19 -0.0528 0.106527 -0.06583 0.020307 -0.06367 0.006932
X20 -0.0079 0.089136 -0.02503 0.03124 -0.03433 0.00915
SRRC
sample size = 100 sample size = 1000 sample size = 10000
mean sd. dev. mean sd. dev. mean sd. dev.
X1 -0.0431 0.123221 -0.05766 0.017913 -0.05667 0.008365
X2 -0.13591 0.051623 -0.09315 0.026012 -0.0775 0.009742
X3 0.105331 0.100402 0.101817 0.011804 0.10409 0.005405
X4 -0.07636 0.085525 -0.08795 0.031818 -0.07098 0.006964
X5 0.067898 0.085437 0.117972 0.017947 0.120517 0.007383
X6 -0.06735 0.06997 0.000469 0.036965 -0.00046 0.007407
X7 0.166439 0.077556 0.212492 0.019278 0.21817 0.008072
X8 0.323015 0.03614 0.35395 0.026569 0.361374 0.006462
X9 0.383158 0.062943 0.372845 0.030169 0.382753 0.006792
X10 0.373635 0.072565 0.345321 0.026555 0.338904 0.005862
X11 0.065236 0.092074 0.016759 0.027229 0.005351 0.008052
X12 -0.00967 0.106418 -0.01836 0.019068 -0.01831 0.00941
X13 -0.01072 0.114539 -0.0405 0.021975 -0.04553 0.007599
X14 -0.01804 0.074433 -0.01611 0.01488 -0.01096 0.006663
X15 -0.00335 0.097997 0.004503 0.018767 0.007763 0.006684
X16 0.109142 0.072854 0.038283 0.02551 0.047335 0.009027
X17 -0.0274 0.073912 0.005861 0.026342 0.008795 0.005263
X18 -0.00087 0.048503 0.012673 0.027583 0.014088 0.006492
X19 -0.0438 0.082148 -0.05743 0.015103 -0.05156 0.006117
X20 -0.02031 0.068414 -0.01804 0.023482 -0.02688 0.005009
JRC
68
PRCC
sample size = 100 sample size = 1000 sample size = 10000
mean sd. dev. mean sd. dev. mean sd. dev.
X1 -0.04947 0.156621 -0.07887 0.023968 -0.07895 0.011451
X2 -0.18193 0.068032 -0.12731 0.035953 -0.10769 0.013197
X3 0.144457 0.14209 0.139013 0.015713 0.14407 0.007973
X4 -0.10043 0.118663 -0.11963 0.042325 -0.09871 0.009176
X5 0.08962 0.11718 0.160207 0.024357 0.166186 0.010082
X6 -0.08918 0.096868 0.000637 0.050328 -0.00061 0.010398
X7 0.217283 0.102258 0.280736 0.025482 0.291783 0.010193
X8 0.396559 0.049166 0.437072 0.027409 0.450982 0.008097
X9 0.459883 0.078341 0.456133 0.033089 0.471904 0.009019
X10 0.448221 0.073248 0.428958 0.029416 0.428269 0.007763
X11 0.087517 0.123885 0.023317 0.037384 0.007492 0.011237
X12 -0.01455 0.142166 -0.02527 0.026448 -0.02557 0.013076
X13 -0.023 0.154595 -0.05589 0.030696 -0.06353 0.010683
X14 -0.0224 0.101309 -0.02208 0.020113 -0.01532 0.009316
X15 -0.00929 0.134198 0.006203 0.025269 0.010827 0.009263
X16 0.146891 0.097535 0.052574 0.035041 0.066048 0.012608
X17 -0.02943 0.098954 0.007768 0.035891 0.012326 0.007422
X18 -0.00243 0.065596 0.017626 0.037595 0.019659 0.009005
X19 -0.05834 0.110451 -0.07876 0.020431 -0.0719 0.008577
X20 -0.02105 0.090931 -0.02495 0.032522 -0.03757 0.007085
SI first order
sample size = 100 sample size = 1000 sample size = 10000
mean sd. dev. mean sd. dev. mean sd. dev.
X1 -0.05354 0.150829 0.005789 0.020929 0.007391 0.015254
X2 0.071777 0.089242 0.008095 0.039602 0.006514 0.009927
X3 0.005104 0.059667 0.006367 0.015589 0.012159 0.004975
X4 0.040915 0.053071 0.008925 0.03038 0.002328 0.011783
X5 0.010738 0.051927 0.023132 0.011674 0.018432 0.004017
X6 0.029321 0.050292 -0.00158 0.02447 0.001221 0.005459
X7 0.062123 0.029338 0.069265 0.017572 0.061975 0.008587
X8 0.127735 0.085238 0.109158 0.02526 0.107409 0.006085
X9 0.146352 0.085749 0.123896 0.012996 0.126801 0.00868
X10 0.090049 0.072506 0.09328 0.016243 0.099445 0.005831
X11 0.009943 0.00889 0.001919 0.002911 -0.00059 0.001435
X12 0.011246 0.01509 0.000531 0.004653 -4.7E-05 0.000939
X13 0.009906 0.011953 0.001844 0.004481 0.001432 0.000958
X14 0.009417 0.015855 0.000504 0.004052 0.000481 0.001373
X15 0.013811 0.009615 0.00167 0.002726 0.000532 0.000638
X16 0.019301 0.013081 0.003735 0.003754 0.002229 0.000919
X17 0.01093 0.017442 0.001748 0.002916 0.000221 0.000547
X18 0.010096 0.005386 0.000631 0.002598 0.000227 0.000643
X19 0.013998 0.013563 0.004367 0.004171 0.002995 0.00139
X20 0.006345 0.006252 0.002905 0.003082 0.00082 0.001412
JRC
69
SI total
sample size = 100 sample size = 1000 sample size = 10000
mean sd. dev. mean sd. dev. mean sd. dev.
X1 0.324691 0.110302 0.262874 0.032037 0.262786 0.012852
X2 0.305119 0.092212 0.266835 0.03378 0.261091 0.012091
X3 0.093652 0.035405 0.119901 0.019271 0.117754 0.008358
X4 0.244043 0.078569 0.268482 0.027251 0.265127 0.013387
X5 0.115459 0.070421 0.109532 0.007632 0.115485 0.007262
X6 0.094669 0.041151 0.105722 0.015446 0.09677 0.007567
X7 0.062889 0.036708 0.059835 0.012382 0.067131 0.005971
X8 0.065541 0.064447 0.118854 0.017987 0.114827 0.006548
X9 0.10336 0.055605 0.118386 0.02129 0.123861 0.008222
X10 0.105736 0.040279 0.107156 0.020104 0.101761 0.006808
X11 -0.00743 0.008389 0.002352 0.002714 0.003235 0.001266
X12 -0.00349 0.013022 0.003058 0.002885 0.002752 0.000617
X13 -0.00414 0.010593 0.002291 0.003474 0.00334 0.00117
X14 -0.00341 0.012093 0.002937 0.004188 0.003837 0.001381
X15 -0.00661 0.011426 0.001111 0.002953 0.001778 0.000782
X16 -0.00415 0.015388 0.001664 0.00598 0.00361 0.001163
X17 -0.01275 0.015332 -0.00054 0.002145 0.002642 0.000937
X18 -0.00885 0.006595 0.001586 0.0027 0.001149 0.000567
X19 -0.00462 0.016361 0.002028 0.003021 0.003427 0.001318
X20 -0.00584 0.006233 2.96E-05 0.002637 0.002449 0.001133
JRC
70
JRC
71
Model 11
sample size
values for the 10 runs
E(Y) V(Y) R2 R2*
expected results 18.237 75.2433
computed results, SRS
100 mean 18.18478 76.8334 0.983523 0.960687
st.dev. 0.362799 7.972788 0.001885 0.006932
1000 mean 18.32489 76.20511 0.982632 0.965601
st.dev. 0.222892 2.31039 0.000836 0.002714
10000 mean 18.19265 75.17888 0.982436 0.966927
st.dev. 0.066944 0.928568 0.000164 0.00146
100000 mean 18.21747 75.43541 0.982426 0.966948
st.dev. 0.020244 0.26461 9.22E-05 0.000301
200000 mean 18.24037 75.30048 0.982423 0.96663
st.dev. 0.012698 0.120834 2.75E-05 0.000277
sample size
values for the 10 runs
Pearson Spearman
X1 X2 X3 X1 X2 X3
expected results
computed results, SRS
100 mean 0.675991 -0.08557 0.729822 0.671378 -0.08675 0.719274
st.dev. 0.034403 0.087938 0.030859 0.040437 0.081544 0.036003
1000 mean 0.672472 -0.11129 0.727982 0.669505 -0.10496 0.720066
st.dev. 0.015558 0.027823 0.010872 0.017441 0.027715 0.01274
10000 mean 0.664892 -0.10618 0.72683 0.660544 -0.09986 0.720999
st.dev. 0.004003 0.011103 0.003121 0.004679 0.01113 0.003506
100000 mean 0.665499 -0.10648 0.727966 0.661566 -0.0997 0.721824
st.dev. 0.001449 0.005211 0.001083 0.001596 0.0049 0.001348
200000 mean 0.665813 -0.10667 0.727135 0.662218 -0.09986 0.72047
st.dev. 0.000783 0.002534 0.00108 0.000885 0.002556 0.001123
JRC
72
sample size
values for the 10 runs
SRC PCC
X1 X2 X3 X1 X2 X3
expected results
computed results, SRS
100 mean 0.660877 -0.10676 0.72337 0.981289 -0.63269 0.984443
st.dev. 0.029313 0.014706 0.025766 0.002586 0.051644 0.001505
1000 mean 0.663781 -0.1064 0.720514 0.980814 -0.62775 0.983657
st.dev. 0.010467 0.003898 0.014014 0.000765 0.012456 0.000508
10000 mean 0.665368 -0.1052 0.727627 0.980731 -0.62168 0.983812
st.dev. 0.003508 0.000846 0.003779 0.000261 0.002443 0.000131
100000 mean 0.664289 -0.1049 0.726918 0.980663 -0.6205 0.983774
st.dev. 0.001291 0.000659 0.001249 0.000124 0.002233 6.36E-05
200000 mean 0.665144 -0.105 0.726635 0.980707 -0.62085 0.983759
st.dev. 0.001098 0.000354 0.000699 5.91E-05 0.001451 3.33E-05
sample size
values for the 10 runs
SRRC PRCC
X1 X2 X3 X1 X2 X3
expected results
computed results, SRS
100 mean 0.65463 -0.1061 0.712085 0.955948 -0.46814 0.962897
st.dev. 0.034282 0.015718 0.032059 0.009835 0.060069 0.005972
1000 mean 0.660678 -0.09998 0.712185 0.962701 -0.47464 0.967663
st.dev. 0.01195 0.003018 0.015483 0.002717 0.015454 0.002474
10000 mean 0.661038 -0.09883 0.721775 0.964173 -0.47763 0.96969
st.dev. 0.003708 0.001117 0.004383 0.001561 0.008757 0.001298
100000 mean 0.660376 -0.09812 0.720788 0.96413 -0.47495 0.969631
st.dev. 0.001477 0.000475 0.001284 0.000312 0.002166 0.000267
200000 mean 0.661566 -0.0982 0.719978 0.963927 -0.4735 0.969287
st.dev. 0.001085 0.000332 0.000646 0.0003 0.001452 0.000215
JRC
73
sample size
values for the 10 runs
SI first order (Sobol) SI total (Sobol)
X1 X2 X3 X1 X2 X3
expected results 0.44 0.01 0.55
computed results, SRS
100 mean 0.458717 0.05859 0.538042 0.458501 -0.04131 0.653696
st.dev. 0.142929 0.033691 0.207527 0.197966 0.026304 0.186208
1000 mean 0.442536 0.005975 0.550524 0.437716 0.016703 0.543226
st.dev. 0.076701 0.010154 0.077128 0.067803 0.007569 0.058752
10000 mean 0.444628 0.012684 0.558567 0.445179 0.009256 0.541098
st.dev. 0.012404 0.002119 0.018489 0.010751 0.002037 0.020489
100000 mean 0.439542 0.011374 0.549809 0.446149 0.011317 0.539418
st.dev. 0.008578 0.000591 0.002963 0.005399 0.000651 0.006932
200000 mean 0.443131 0.010956 0.541274 0.442069 0.011655 0.547516
st.dev. 0.005883 0.000407 0.004989 0.005481 0.000398 0.005451
JRC
74
JRC
75
Model 12a
sample size
values for the 10 runs
E(Y) V(Y) R2 R2*
computed results, SRS
100 mean 347.6652 44850.96 0.069139 0.083874
st.dev. 18.9563 6763.889 0.045306 0.049086
1000 mean 342.4956 46046.2 0.006675 0.007607
st.dev. 7.524587 2324.916 0.00365 0.00369
10000 mean 344.6347 47551.59 0.000327 0.000441
st.dev. 1.5726 760.2262 0.000233 0.000315
100000 mean 344.1855 47439.21 5.34E-05 6.88E-05
st.dev. 0.938724 253.2672 3.26E-05 4.48E-05
sample size
values for the 10 runs Pearson
X1 X2 X3 X4 X5 X6
computed results, SRS
100 mean -0.02444 0.00941 -0.00466 -0.0574 0.031021 0.033084
st. dev. 0.065468 0.095037 0.092449 0.119481 0.140677 0.103071
1000 mean -0.01323 -0.00461 -0.00755 0.003471 0.006475 0.004895
st. dev. 0.032996 0.037282 0.033892 0.044501 0.020036 0.03375
10000 mean 0.001113 -0.00072 -0.00225 -0.00085 -0.00171 -0.00324
st. dev. 0.005672 0.008967 0.008381 0.006152 0.00734 0.008042
100000 mean -0.00143 -0.00017 0.000211 -0.00123 -2.4E-05 0.000131
st. dev. 0.001701 0.003215 0.004358 0.002596 0.002624 0.00306
sample size
values for the 10 runs Spearman
X1 X2 X3 X4 X5 X6
computed results, SRS
100 mean -0.03317 0.028935 -0.01567 -0.0551 0.042005 0.020841
st. dev. 0.080715 0.117746 0.094807 0.132787 0.153814 0.108368
1000 mean -0.01302 -0.01018 -0.00045 0.001404 0.00639 0.011607
st. dev. 0.032898 0.03325 0.038738 0.048342 0.024362 0.036684
10000 mean 0.001194 0.000272 -0.00342 -0.00087 -0.0033 -0.00115
st. dev. 0.006045 0.011556 0.008875 0.005893 0.009409 0.009361
100000 mean -0.00148 -8.8E-05 6.44E-05 -0.0015 -0.0001 0.000141
st. dev. 0.002298 0.003798 0.004748 0.0034 0.00296 0.002982
JRC
76
sample size
values for the 10 runs SRC
X1 X2 X3 X4 X5 X6
computed results, SRS
100 mean -0.01749 0.009758 -0.00527 -0.06613 0.040194 0.033134
st. dev. 0.068058 0.09578 0.09937 0.125628 0.147794 0.116927
1000 mean -0.01333 -0.00481 -0.00622 0.002427 0.006746 0.005141
st. dev. 0.032832 0.036781 0.033153 0.044442 0.01977 0.033102
10000 mean 0.001059 -0.00069 -0.00225 -0.00078 -0.00176 -0.00328
st. dev. 0.005696 0.008985 0.008371 0.00617 0.007398 0.008068
100000 mean -0.00144 -0.00016 0.000216 -0.00123 -2.7E-05 0.000129
st. dev. 0.001703 0.003219 0.004359 0.002595 0.002627 0.003057
sample size
values for the 10 runs PCC
X1 X2 X3 X4 X5 X6
computed results, SRS
100 mean -0.01773 0.00905 -0.00389 -0.06688 0.04108 0.033401
st. dev. 0.068767 0.095499 0.098783 0.12807 0.150024 0.11746
1000 mean -0.01331 -0.00482 -0.00624 0.002486 0.006737 0.005108
st. dev. 0.032866 0.036827 0.033182 0.044483 0.019764 0.033109
10000 mean 0.001058 -0.00069 -0.00225 -0.00078 -0.00176 -0.00328
st. dev. 0.005696 0.008985 0.008369 0.00617 0.007398 0.008068
100000 mean -0.00144 -0.00016 0.000216 -0.00123 -2.7E-05 0.000129
st. dev. 0.001703 0.003219 0.004359 0.002595 0.002627 0.003057
sample size
values for the 10 runs SRRC
X1 X2 X3 X4 X5 X6
computed results, SRS
100 mean -0.02136 0.028935 -0.01577 -0.05987 0.049688 0.018838
st. dev. 0.082683 0.121661 0.107962 0.133497 0.155678 0.124663
1000 mean -0.01315 -0.01027 0.001007 0.000123 0.006484 0.012062
st. dev. 0.033317 0.033161 0.038045 0.048779 0.024343 0.036387
10000 mean 0.001141 0.000288 -0.00344 -0.00084 -0.00341 -0.00122
st. dev. 0.006092 0.011578 0.008823 0.005979 0.009425 0.009406
100000 mean -0.00148 -8.6E-05 6.89E-05 -0.00151 -0.00011 0.000141
st. dev. 0.002303 0.003803 0.00474 0.003401 0.002956 0.002981
JRC
77
sample size
values for the 10 runs PRCC
X1 X2 X3 X4 X5 X6
computed results, SRS
100 mean -0.02227 0.02804 -0.01426 -0.06168 0.050579 0.019036
st. dev. 0.083678 0.119576 0.107093 0.136745 0.157771 0.127638
1000 mean -0.01314 -0.01031 0.001025 0.000144 0.006479 0.012044
st. dev. 0.033347 0.033218 0.038059 0.048814 0.024376 0.036369
10000 mean 0.001141 0.000287 -0.00344 -0.00084 -0.00341 -0.00122
st. dev. 0.006093 0.011577 0.008823 0.005979 0.009425 0.009406
100000 mean -0.00148 -8.6E-05 6.89E-05 -0.00151 -0.00011 0.000141
st. dev. 0.002303 0.003803 0.00474 0.003401 0.002956 0.002981
sample size
values for the 10 runs SI first order (Sobol)
X1 X2 X3 X4 X5 X6
computed results, SRS
100 mean 0.232478 0.155589 0.199743 0.183789 0.154904 0.167004
st. dev. 0.20836 0.078062 0.105184 0.109981 0.159998 0.063125
1000 mean 0.13233 0.12623 0.140096 0.138966 0.139317 0.117129
st. dev. 0.044967 0.030002 0.03827 0.031991 0.024676 0.040507
10000 mean 0.130655 0.122149 0.126188 0.125965 0.126584 0.128275
st. dev. 0.013676 0.012165 0.014416 0.015427 0.012998 0.008268
100000 mean 0.127855 0.130785 0.129994 0.130775 0.132229 0.128284
st. dev. 0.003485 0.002394 0.002799 0.003222 0.00423 0.003283
sample size
values for the 10 runs SI total (Sobol)
X1 X2 X3 X4 X5 X6
computed results, SRS
100 mean 0.124767 0.184463 0.116348 0.157552 0.252764 0.182572
st. dev. 0.226402 0.129149 0.103062 0.123984 0.143707 0.119949
1000 mean 0.207773 0.201576 0.209552 0.185183 0.192994 0.209178
st. dev. 0.05458 0.039444 0.035838 0.017265 0.030737 0.04067
10000 mean 0.205666 0.210134 0.21044 0.211621 0.203746 0.204874
st. dev. 0.012167 0.016074 0.013874 0.022845 0.012413 0.012129
100000 mean 0.20766 0.205909 0.206987 0.204326 0.204458 0.206877
st. dev. 0.004306 0.003207 0.004429 0.003514 0.003824 0.003333
JRC
78
JRC
79
JRC
80
Model 12b
sample size
values for the 10 runs
E(Y) V(Y) R2 R2*
computed results, SRS
100 mean 162.5659 22176.45 0.069331 0.071817
st.dev. 19.42397 6634.087 0.037946 0.04122
1000 mean 163.4213 20250.72 0.007188 0.006644
st.dev. 5.703223 1216.665 0.004033 0.002731
10000 mean 161.9412 20260.51 0.000725 0.000736
st.dev. 1.787221 312.3671 0.000364 0.00044
100000 mean 162.0473 20202.22 4.85E-05 4.85E-05
st.dev. 0.348819 88.85797 2.53E-05 3.35E-05
sample size
values for the 10 runs Pearson
X1 X2 X3 X4 X5 X6
computed results, SRS
100 mean -0.00547 -0.03038 0.02422 0.017138 0.004942 0.062219
st. dev. 0.122535 0.102167 0.120092 0.088944 0.121953 0.049563
1000 mean 0.011427 -9.1E-06 0.000613 -0.0001 0.010487 -0.01643
st. dev. 0.026918 0.054612 0.033322 0.031999 0.028844 0.02939
10000 mean 0.001473 -0.00407 -0.00343 -0.005 0.001643 -0.00459
st. dev. 0.008972 0.012996 0.00775 0.012513 0.013468 0.008769
100000 mean -0.00049 -0.00023 -0.00097 0.000878 6.83E-05 -0.00105
st. dev. 0.003366 0.002485 0.002742 0.002764 0.002179 0.003624
sample size
values for the 10 runs Spearman
X1 X2 X3 X4 X5 X6
computed results, SRS
100 mean -0.01802 -0.02089 0.01735 0.035297 -0.00802 0.070128
st. dev. 0.128646 0.074889 0.104147 0.102863 0.133602 0.080583
1000 mean 0.007391 -0.00088 0.004055 -0.00494 0.008657 -0.01852
st. dev. 0.02232 0.054772 0.030017 0.02536 0.028309 0.032689
10000 mean 0.000871 -0.00409 -0.00325 -0.00221 -0.0013 -0.00561
st. dev. 0.006229 0.012642 0.009588 0.014696 0.011271 0.010507
100000 mean -0.00029 -0.00049 -0.00147 0.000906 0.000406 -0.00054
st. dev. 0.003217 0.002324 0.003092 0.002394 0.001896 0.003889
JRC
81
sample size
values for the 10 runs SRC
X1 X2 X3 X4 X5 X6
computed results, SRS
100 mean -0.00299 -0.03701 0.037717 0.022836 0.028141 0.071125
st. dev. 0.129648 0.109495 0.125822 0.103955 0.139629 0.058704
1000 mean 0.011644 0.00046 -0.00025 0.000744 0.009959 -0.01677
st. dev. 0.026165 0.053008 0.032782 0.032806 0.029637 0.028554
10000 mean 0.00147 -0.00398 -0.00333 -0.00489 0.00153 -0.00461
st. dev. 0.008817 0.012943 0.007644 0.012485 0.0133 0.008814
100000 mean -0.00049 -0.00022 -0.00096 0.000883 7.46E-05 -0.00105
st. dev. 0.003366 0.002476 0.002735 0.00277 0.002179 0.003616
sample size
values for the 10 runs PCC
X1 X2 X3 X4 X5 X6
computed results, SRS
100 mean -0.00358 -0.0372 0.03763 0.023278 0.027566 0.071301
st. dev. 0.130491 0.109175 0.125936 0.103766 0.138606 0.059303
1000 mean 0.011634 0.000389 -0.00021 0.000734 0.009966 -0.01678
st. dev. 0.026188 0.052955 0.032794 0.032791 0.029661 0.028581
10000 mean 0.00147 -0.00398 -0.00333 -0.00489 0.00153 -0.00461
st. dev. 0.008817 0.012943 0.007643 0.012486 0.013299 0.008814
100000 mean -0.00049 -0.00022 -0.00096 0.000883 7.46E-05 -0.00105
st. dev. 0.003366 0.002476 0.002735 0.00277 0.002179 0.003616
sample size
values for the 10 runs SRRC
X1 X2 X3 X4 X5 X6
computed results, SRS
100 mean -0.01479 -0.02624 0.030569 0.040782 0.015879 0.079686
st. dev. 0.135585 0.079192 0.106123 0.116453 0.14438 0.085383
1000 mean 0.007306 -0.00058 0.003609 -0.00365 0.0075 -0.01876
st. dev. 0.021941 0.053728 0.029232 0.026225 0.028821 0.032218
10000 mean 0.000928 -0.00401 -0.00316 -0.00209 -0.00137 -0.00567
st. dev. 0.006213 0.012665 0.009548 0.014721 0.011227 0.010611
100000 mean -0.00029 -0.00049 -0.00146 0.00091 0.000412 -0.00054
st. dev. 0.003225 0.00231 0.00309 0.002403 0.001893 0.003887
JRC
82
sample size
values for the 10 runs PRCC
X1 X2 X3 X4 X5 X6
computed results, SRS
100 mean -0.01568 -0.02648 0.031426 0.041037 0.015258 0.079838
st. dev. 0.136684 0.078875 0.107108 0.115665 0.143704 0.085602
1000 mean 0.007295 -0.00064 0.003639 -0.00365 0.007503 -0.01876
st. dev. 0.021961 0.053642 0.029257 0.026226 0.028838 0.032223
10000 mean 0.000928 -0.00401 -0.00316 -0.00209 -0.00138 -0.00567
st. dev. 0.006214 0.012665 0.009548 0.014721 0.011227 0.010612
100000 mean -0.00029 -0.00049 -0.00146 0.00091 0.000412 -0.00054
st. dev. 0.003225 0.00231 0.00309 0.002403 0.001893 0.003887
sample size
values for the 10 runs SI first order (Sobol)
X1 X2 X3 X4 X5 X6
computed results, SRS
100 mean 0.109172 0.08781 0.038427 0.101936 0.117579 0.096021
st. dev. 0.143001 0.16877 0.151786 0.226251 0.138327 0.167627
1000 mean 0.039169 0.011034 0.039368 0.050159 0.062363 0.068964
st. dev. 0.067377 0.062834 0.041679 0.044494 0.044222 0.04078
10000 mean 0.033933 0.038721 0.039537 0.037194 0.040012 0.03898
st. dev. 0.007862 0.012633 0.01299 0.010566 0.011675 0.008912
100000 mean 0.037757 0.038152 0.039258 0.040102 0.040083 0.038983
st. dev. 0.00348 0.00471 0.003637 0.003636 0.004398 0.003587
sample size
values for the 10 runs SI total (Sobol)
X1 X2 X3 X4 X5 X6
computed results, SRS
100 mean 0.282445 0.301 0.26562 0.242717 0.237996 0.260721
st. dev. 0.126807 0.214686 0.156413 0.184596 0.172175 0.131706
1000 mean 0.320792 0.326057 0.312068 0.298535 0.309083 0.275405
st. dev. 0.080678 0.055479 0.045217 0.050571 0.067778 0.06009
10000 mean 0.313757 0.299861 0.306797 0.302869 0.301742 0.302305
st. dev. 0.012016 0.011859 0.016331 0.011992 0.012618 0.011374
100000 mean 0.306229 0.304379 0.304194 0.306253 0.305735 0.303974
st. dev. 0.004754 0.00385 0.004008 0.005435 0.005145 0.005941
JRC
83
JRC
84
JRC
85
Model 13a
sample size
values for the 10 runs
E(Y) V(Y) R2 R2*
expected results 0.5 0.40333
computed results, SRS
100 mean 0.49295 0.405617 0.665683 0.843021
st.dev. 0.073689 0.055638 0.031394 0.055142
1000 mean 0.496902 0.400391 0.671675 0.842018
st.dev. 0.022992 0.008496 0.013281 0.017633
10000 mean 0.500646 0.403967 0.671174 0.83629
st.dev. 0.003539 0.004003 0.004061 0.004993
100000 mean 0.499763 0.40319 0.671451 0.837767
st.dev. 0.001963 0.001133 0.001226 0.001188
200000 mean 0.500319 0.403289 0.670553 0.836699
st.dev. 0.001303 0.000853 0.000821 0.000755
sample size
values for the 10 runs
Pearson Spearman
X1 X2 X1 X2
expected results
computed results, SRS
100 mean 0.675428 0.477195 0.871334 0.306864
st.dev. 0.047574 0.063293 0.05897 0.092031
1000 mean 0.686411 0.453379 0.877091 0.274382
st.dev. 0.014709 0.021669 0.016549 0.0339
10000 mean 0.680241 0.45537 0.872462 0.271489
st.dev. 0.002997 0.006671 0.004292 0.008625
100000 mean 0.681931 0.455686 0.873961 0.272749
st.dev. 0.001065 0.003657 0.001383 0.004212
200000 mean 0.680409 0.455965 0.872832 0.272982
st.dev. 0.000962 0.002109 0.000925 0.002535
JRC
86
sample size
values for the 10 runs
SRC PCC
X1 X2 X1 X2
expected results
computed results, SRS
100 mean 0.660344 0.453629 0.750127 0.612733
st.dev. 0.045662 0.061547 0.029513 0.055646
1000 mean 0.682616 0.447454 0.765764 0.615137
st.dev. 0.012614 0.017351 0.009236 0.016603
10000 mean 0.681014 0.456548 0.764926 0.622852
st.dev. 0.004269 0.003727 0.003151 0.0046
100000 mean 0.681022 0.454332 0.765075 0.621163
st.dev. 0.002009 0.002313 0.000947 0.002515
200000 mean 0.680181 0.455626 0.764251 0.621731
st.dev. 0.001036 0.001602 0.00047 0.001614
sample size
values for the 10 runs
SRRC PRCC
X1 X2 X1 X2
expected results
computed results, SRS
100 mean 0.861873 0.275 0.906019 0.564112
st.dev. 0.059886 0.082526 0.039192 0.058989
1000 mean 0.874857 0.26902 0.910217 0.560262
st.dev. 0.017666 0.022278 0.011442 0.014636
10000 mean 0.872725 0.274602 0.907231 0.561564
st.dev. 0.004006 0.005563 0.003016 0.0052
100000 mean 0.873147 0.272798 0.908039 0.560761
st.dev. 0.001829 0.002395 0.000883 0.002344
200000 mean 0.87244 0.274417 0.907388 0.56178
st.dev. 0.001082 0.001743 0.000536 0.001696
JRC
87
sample size
values for the 10 runs
SI first order (Sobol)
SI total (Sobol)
X1 X2 X1 X2
expected results 0.5951 0.2066
computed results, SRS
100 mean 0.58667 0.188659 0.832469 0.359938
st.dev. 0.169499 0.141525 0.11431 0.129581
1000 mean 0.641444 0.231868 0.77554 0.379393
st.dev. 0.072578 0.03838 0.043397 0.042948
10000 mean 0.600743 0.204745 0.797207 0.408683
st.dev. 0.012953 0.019382 0.024434 0.006524
100000 mean 0.594122 0.206929 0.791207 0.40475
st.dev. 0.007883 0.004208 0.004156 0.003282
200000 mean 0.593687 0.20632 0.793734 0.404972
st.dev. 0.001816 0.002588 0.002466 0.003443
JRC
88
JRC
89
Model 13b (initial one)
sample size
values for the 10 runs
E(Y) V(Y) R2 R2*
expected results 0 0.32
computed results, SRS
100 mean -0.02965 0.328402 0.57226 0.57503
st.dev. 0.043688 0.031893 0.041932 0.042684
1000 mean 0.002144 0.323688 0.588397 0.59435
st.dev. 0.013188 0.007167 0.011846 0.011387
10000 mean -0.00177 0.319781 0.584563 0.594258
st.dev. 0.006031 0.004212 0.004095 0.003523
100000 mean 0.000998 0.320235 0.58502 0.593648
st.dev. 0.002395 0.000851 0.001946 0.001459
200000 mean -9.1E-05 0.320074 0.584265 0.592745
st.dev. 0.00144 0.000603 0.000683 0.000718
sample size
values for the 10 runs
Pearson Spearman
X1 X2 X1 X2
expected results
computed results, SRS
100 mean 0.753504 -0.05588 0.753943 -0.07911
st.dev. 0.028228 0.111521 0.027869 0.109483
1000 mean 0.766771 -0.00432 0.770452 0.001803
st.dev. 0.00761 0.033596 0.007361 0.033722
10000 mean 0.764535 0.000134 0.770785 -0.00275
st.dev. 0.002689 0.009323 0.002352 0.013059
100000 mean 0.764861 0.001476 0.770476 0.002269
st.dev. 0.001275 0.00468 0.000947 0.005563
200000 mean 0.764369 -0.00082 0.769893 -0.00055
st.dev. 0.000445 0.002661 0.000465 0.003256
JRC
90
sample size
values for the 10 runs
SRC PCC
X1 X2 X1 X2
expected results
computed results, SRS
100 mean 0.751511 -0.00752 0.751656 -0.01286
st.dev. 0.030477 0.064486 0.029895 0.096929
1000 mean 0.767142 0.000363 0.766754 0.000565
st.dev. 0.008142 0.021288 0.007826 0.033308
10000 mean 0.764532 -9.5E-05 0.764541 -0.00015
st.dev. 0.00271 0.006836 0.002693 0.010584
100000 mean 0.764855 0.000795 0.764859 0.001228
st.dev. 0.001279 0.002512 0.001275 0.003895
200000 mean 0.764371 -0.00056 0.76437 -0.00087
st.dev. 0.000449 0.002143 0.000448 0.003325
sample size
values for the 10 runs
SRRC PRCC
X1 X2 X1 X2
expected results
computed results, SRS
100 mean 0.752371 -0.02426 0.752918 -0.03818
st.dev. 0.027908 0.077262 0.028572 0.117052
1000 mean 0.770998 0.00599 0.770638 0.009395
st.dev. 0.007568 0.027281 0.007411 0.042805
10000 mean 0.770794 -0.00296 0.770835 -0.00467
st.dev. 0.002414 0.012254 0.002339 0.019169
100000 mean 0.770468 0.001581 0.770476 0.002476
st.dev. 0.000949 0.003431 0.000947 0.005383
200000 mean 0.769896 -0.00028 0.769897 -0.00044
st.dev. 0.00047 0.003327 0.000468 0.005214
JRC
91
sample size
values for the 10 runs
SI first order (Sobol)
SI total (Sobol)
X1 X2 X1 X2
expected results 0.75 0 1 0.25
computed results, SRS
100 mean 0.83554 0.016758 1.113432 0.246042
st.dev. 0.138906 0.072083 0.20355 0.051963
1000 mean 0.736775 -0.00536 0.983091 0.255463
st.dev. 0.03678 0.021644 0.035113 0.015233
10000 mean 0.751159 -0.00397 0.998202 0.249735
st.dev. 0.017654 0.007352 0.017514 0.008207
100000 mean 0.748375 -0.00047 0.997349 0.249943
st.dev. 0.002565 0.002764 0.004223 0.001488
200000 mean 0.749032 0.001152 1.000553 0.249571
st.dev. 0.003394 0.001518 0.002712 0.001437
JRC
92
JRC
93
Annex : results for the models in the benchmark Pamina task 2.1.D – screening method
We have also performed, for the same models, the OAT – Morris method. Even if the method is interesting
for the models with many input variables, for the sake of the unity of the presentation, we performed the
computations for all the models. The computations for model 12a have not been performed (normal
distributions).
The procedure has been repeated 10 times for each model, and the means and standard deviations of *
and corresponding to each variable are reported in the tables here after.
The numerical results are in .csv files (to be read with Excel).
In order to have a “readable” figure for each case, we only plotted the first line of each .csv file. Model 1
values for the 10 runs mean st. dev.
*X1 0.103815 0.006087
X2 0.310033 0.024364
X3 0.932394 0.037321
X1 1.78E-16 4.51E-17
X2 1.22E-16 3.23E-17
X3 1.3E-16 6.36E-17
JRC
94
Model 3
values for the 10 runs mean st. dev.
*
X1 0.426003 0.083879
X2 0.370721 0.064034
X3 0.293616 0.058333
X4 0.213524 0.033458
X5 0.16648 0.028734
X6 0.110344 0.016343
X7 0.072022 0.011856
X8 0.040196 0.005818
X9 0.018421 0.002899
X10 0.004563 0.000741
X11 0 0
X12 0.004577 0.000739
X13 0.018115 0.002828
X14 0.041305 0.006615
X15 0.071956 0.010992
X16 0.112149 0.016938
X17 0.162973 0.027228
X18 0.217946 0.045305
X19 0.280772 0.048134
X20 0.355967 0.055531
X21 0.423813 0.08498
X22 0.517881 0.09105
X1 9.65E-17 3.15E-17
X2 9.91E-17 4.58E-17
X3 9.4E-17 2.74E-17
X4 9.73E-17 2.7E-17
X5 9.22E-17 3.19E-17
X6 8.97E-17 3.51E-17
X7 9.49E-17 3.22E-17
X8 4.55E-17 1.48E-17
X9 9.14E-17 3.44E-17
X10 8.67E-17 2.55E-17
X11 0 0
X12 8.29E-17 2.55E-17
X13 7.3E-17 2.2E-17
X14 6.58E-17 1.38E-17
X15 7.05E-17 1.89E-17
X16 7.14E-17 2.49E-17
X17 5.86E-17 2.32E-17
X18 7.88E-17 2.72E-17
X19 7.73E-17 2.94E-17
X20 7.73E-17 2.98E-17
X21 7.61E-17 3.01E-17
X22 9.24E-17 5.68E-17
JRC
95
Model 4a
values for the 10 runs mean st. dev.
X1 0.745506 0.07336
X2 0.606987 0.062047
X1 8.95E-17 2.21E-17
X2 0.321863 0.043554
Model 4b
values for the 10 runs mean st. dev.
X1 0.043181 0.008679
X2 0.851126 0.051115
X1 4.71E-17 1.73E-17
X2 0.43778 0.08745
Model 4c
values for the 10 runs mean st. dev.
X1 0.010002 0.002476
X2 0.837197 0.065962
X1 5.61E-17 2.31E-17
X2 0.449439 0.068157
Model 5a
values for the 10 runs mean st. dev.
*
X1 0.518409 0.167647
X2 0.344465 0.100039
X3 0.3847 0.087741
X4 0.38594 0.098791
X5 0.36776 0.092316
X6 0.35273 0.10344
X1 0.350342 0.110894
X2 0.228022 0.080643
X3 0.260444 0.055719
X4 0.293539 0.075511
X5 0.284753 0.079965
X6 0.244745 0.095014
JRC
96
Model 5b
values for the 10 runs mean st. dev.
*
X1 0.237159 0.101658
X2 0.245594 0.090246
X3 0.244189 0.091374
X4 0.268749 0.105937
X5 0.252598 0.088883
X6 0.260106 0.070209
X7 0.278255 0.07942
X8 0.273241 0.074966
X9 0.264513 0.07123
X10 0.270853 0.054253
X11 0.178052 0.033917
X12 0.171715 0.034126
X13 0.165789 0.04495
X14 0.160814 0.047483
X15 0.162717 0.035719
X16 0.164007 0.03892
X17 0.165008 0.035045
X18 0.154765 0.042282
X19 0.156254 0.044617
X20 0.150403 0.047089
X1 0.162869 0.106416
X2 0.172106 0.096486
X3 0.169575 0.106911
X4 0.178131 0.119753
X5 0.163381 0.094096
X6 0.170083 0.074419
X7 0.204018 0.088341
X8 0.211107 0.081995
X9 0.19688 0.075192
X10 0.205465 0.090503
X11 0.144454 0.077136
X12 0.136489 0.064098
X13 0.129261 0.05519
X14 0.122875 0.05287
X15 0.120418 0.051088
X16 0.119336 0.051409
X17 0.126631 0.058593
X18 0.12129 0.058882
X19 0.124324 0.061527
X20 0.1157 0.066342
JRC
97
Model 6a
values for the 10 runs mean st. dev.
X1 0.437329 0.056178
X2 0.89441 0.060524
X1 0.118979 0.025016
X2 0.148582 0.016816
Model 6b
values for the 10 runs mean st. dev.
X1 0.516496 0.110102
X2 0.658527 0.11493
X1 0.558254 0.096949
X2 0.469521 0.156093
Model 7
values for the 10 runs mean st. dev.
*
X1 0.820507 0.186314
X2 0.412057 0.120584
X3 0.184459 0.070009
X4 0.11629 0.04378
X5 0.010437 0.003687
X6 0.011063 0.00478
X7 0.01213 0.003427
X8 0.010988 0.00402
X1 0.909155 0.261537
X2 0.471662 0.13919
X3 0.223323 0.07318
X4 0.14336 0.051228
X5 0.01217 0.004609
X6 0.013635 0.005621
X7 0.014513 0.003337
X8 0.013317 0.004356
JRC
98
Model 9
values for the 10 runs mean st. dev.
*X1 0.584158 0.131754
X2 0.474222 0.064822
X3 0.454989 0.118145
X1 0.563756 0.100497
X2 0.497456 0.072748
X3 0.547502 0.114006
JRC
99
Model 10
values for the 10 runs mean st. dev.
*
X1 0.399074 0.158346
X2 0.437888 0.21091
X3 0.291249 0.173821
X4 0.445532 0.195782
X5 0.231908 0.149363
X6 0.264556 0.093579
X7 0.204984 0.099978
X8 0.295328 0.072909
X9 0.312161 0.071607
X10 0.290369 0.072898
X11 0.042745 0.012824
X12 0.046436 0.033237
X13 0.04287 0.018971
X14 0.070187 0.031827
X15 0.038553 0.015743
X16 0.04621 0.018481
X17 0.047478 0.030033
X18 0.03858 0.01177
X19 0.04236 0.013498
X20 0.032317 0.016914
X1 0.446739 0.212444
X2 0.520046 0.24525
X3 0.308923 0.2467
X4 0.507729 0.255401
X5 0.294187 0.209464
X6 0.340062 0.125354
X7 0.10347 0.070805
X8 0.039184 0.019978
X9 0.036235 0.009082
X10 0.04682 0.021876
X11 0.050137 0.018859
X12 0.044007 0.021413
X13 0.031601 0.015898
X14 0.07956 0.038392
X15 0.039404 0.021021
X16 0.039206 0.013296
X17 0.051025 0.030538
X18 0.043082 0.014231
X19 0.033742 0.013321
X20 0.034382 0.016591
JRC
100
Model 11
values for the 10 runs mean st. dev.
*X1 0.651292 0.068048
X2 0.109617 0.009748
X3 0.743594 0.05923
X1 1E-16 2.73E-17
X2 0.027086 0.003608
X3 0.121807 0.025765
Model 12b
values for the 10 runs mean st. dev.
*
X1 0.377677 0.115049
X2 0.382663 0.096351
X3 0.414962 0.127859
X4 0.378379 0.089148
X5 0.361821 0.08133
X6 0.406796 0.100527
X1 0.449906 0.128965
X2 0.415941 0.123
X3 0.497748 0.134603
X4 0.459647 0.107457
X5 0.421555 0.105973
X6 0.5014 0.090267
Model 13a
values for the 10 runs mean st. dev.
*X1 0.747626 0.155118
X2 0.488854 0.149849
X1 0.515873 0.086488
X2 0.520306 0.044871
Model 13b
values for the 10 runs mean st. dev.
*X1 0.822466 0.091808
X2 0.590247 0.043984
X1 0.653894 0.117367
X2 0.583138 0.072679
JRC
101
JRC
102
JRC
103
1
Notes on the Benchmark Exercise
Elmar Plischke, TU Clausthal
Context/Objective
A set of benchmark examples was presented in the PAMINA milestone M2.1.D.3. This note gathers
the results of the sensitivity analysis benchmark cases performed by TU Clausthal within the PAMINA
project based upon that milestone.
Framework
Test-Cases Considered All test cases agreed on in the Petten meeting were analysed. The following table lists all of these
test functions. Some functions only differ in the use of different input distributions.
Model Name Parameters Source
1 Linear Model 3 SA, §2.9.1: Model 1
2 Linear Model with Interactions 2 SA, §2.9.1: Model 2
3 Linear Sobol’ Function 22 SA, §2.9.1: Model 3
4a Monotonic Model 2 SA, §2.9.2: Model 4(a)
4c Monotonic Model 2 SA, §2.9.2: Model 4(c)
5a Exponential Sobol’ Function 6 SA, §2.9.2: Model 5(a)
5b Exponential Sobol’ Function 20 SA, §2.9.2: Model 5(c)
6a Quotient 2 SA, §2.9.2: Model 6(a)
6b Quotient 2 SA, §2.9.2: Model 6(b)
7 Sobol’ g-Function 8 SA, §2.9.3: Model 7
8 ** missing **
9 Ishigami Function 3 SA, §2.9.3: Model 9
10 Morris Function 20 SA, §2.9.3: Model 10
11 Bungee Jumping Man 3 SAIP, §3.1
12a Distance of Two Spheres 6 SAIP, §3.5
12b Distance of Two Spheres 6 SAIP, §3.5
13a Smooth Switch 2 Milestone
13b Smooth Switch 2 Milestone
The analysis of some of these test function was marked as voluntarily. However, all models were
treated within the same test-bed, where applicable.
Software Setup For our analysis, we used MatLab. All model simulations where driven by the same script. The
analysis consists of the following phases
- Assignment of the model under investigation(“model”), its number of parameters(“k”) and
its input parameter transformation(“trafo”)
2
- Choice of sampling size (“n”), with possible number of repetitions
- Creation of a uniformly distributed sample, either “u=rand(n,k)” using standard random
sampling or “u=lhs(n,k)” for Latin hypercube sampling.
- Transformation of the uniform sample to input parameter space via “x=trafo(u)”
- Model evaluation “y=model(x)”
- Uncertainty Analysis, mean and variance of “y”
- Sensitivity Analysis based on linear regression of “y” on “x”
- Sensitivity Index calculations using a post-processing algorithm, “Si=easi(x,y)”
- (Sensitivity Index calculations using classical methods, “Si=fast(k,n,model,trafo)”)
- Results are written into an ASCII data file
During this process, some graphics were automatically generated. Rank-based indicators were not
computed.
Results In all the following examples the estimates for uncertainty analysis (UA) indicators and sensitivity
analysis (SA) indicators have been generated from 25 run of 50000 samples each. This should provide
us with a precision (way beyond practical purposes) which can be compared with published values.
However, runs of sample sizes 500, 1000, 5000, and 10000 are also available and may be used for
further analysis. Two flavours of data, one for standard random sampling and one for Latin
hypercube sampling, have been generated. For presentation purposes, we show the results of one of
the model runs for each sample size.
Model 1 Model 1 is given by
The results for simple random sampling can be found in the table below.
Model 1-SRS mean variance R
expected 13.00 7.583 1.00 0.1048 0.3145 0.9435 1.00 1.00 1.00 0.0110 0.0989 0.8901
estimated 13.00 7.583 1.00 0.1039 0.3143 0.9434 1.00 1.00 1.00 0.1048 0.3146 0.9436 0.0110 0.0990 0.8898
500 13.14 7.597 1.00 0.0886 0.3150 0.9401 1.00 1.00 1.00 0.1035 0.3271 0.9444 0.0417 0.1236 0.8866
1,000 12.86 7.395 1.00 0.0850 0.3116 0.9423 1.00 1.00 1.00 0.1068 0.3179 0.9464 0.0310 0.1103 0.8892
5,000 12.96 7.691 1.00 0.0947 0.3261 0.9440 1.00 1.00 1.00 0.1053 0.3137 0.9404 0.0109 0.1092 0.8909
10,000 12.99 7.489 1.00 0.0937 0.3124 0.9426 1.00 1.00 1.00 0.1060 0.3168 0.9454 0.0094 0.0991 0.8883
50,000 13.00 7.628 1.00 0.1067 0.3169 0.9435 1.00 1.00 1.00 0.1044 0.3144 0.9425 0.0116 0.1007 0.8899
Pear PCC SRC SI
The results for Latin hypercube sampling can be found in the table below.
Model 1-LHS mean variance R
expected 13.00 7.583 1.00 0.1048 0.3145 0.9435 1.00 1.00 1.00 0.0110 0.0989 0.8901
estimated 13.00 7.585 1.00 0.1056 0.3144 0.9435 1.00 1.00 1.00 0.1048 0.3145 0.9434 0.0114 0.0991 0.8899
500 13.00 8.047 1.00 0.1446 0.3833 0.9468 1.00 1.00 1.00 0.1019 0.3056 0.9169 0.0636 0.1530 0.8998
1,000 13.00 7.690 1.00 0.1507 0.3113 0.9463 1.00 1.00 1.00 0.1042 0.3125 0.9374 0.0383 0.1096 0.8961
5,000 13.00 7.480 1.00 0.0730 0.3056 0.9426 1.00 1.00 1.00 0.1056 0.3167 0.9500 0.0080 0.0976 0.8885
10,000 13.00 7.599 1.00 0.1083 0.3169 0.9433 1.00 1.00 1.00 0.1047 0.3142 0.9425 0.0126 0.1014 0.8898
50,000 13.00 7.565 1.00 0.1099 0.3097 0.9432 1.00 1.00 1.00 0.1050 0.3149 0.9446 0.0123 0.0962 0.8893
Pear PCC SRC SI
No real advantages asides from the good estimates of the mean can be spotted when using
hypercube sampling. As , we know from the UA that Model 1is a linear model. This is also
visible by the fact that the squares of the Pearson indices give the sensitivity indices. The sensitivity
indices add up to 1, as a linear model is also an additive model.
.U(4.5,9.0)~X,U(1.5,4.5)~X,U(0.5,1.5)~X ,X+X+X=Y 321321
3
Model 2 Model 2 is a linear model given by
Model 2 needs special treatment as it is the only example in which the factors are not independent.
Hence sample generation is done via a special function that handles the dependency. Only a sample
size of 50000 has been generated. The UA reproduces the associated indicators well. Sensitivity
indices were computed with EASI using the generated interdependent sample set (ignoring the
dependence in the input data). However, a slight systematic error seems to be introduced in this
way.
Model 2-SRS mean variance R
expected 1.00 0.2917 1.00 0.94 0.94 1.00 1.00 0.5345 0.5345 0.9286 0.9286
estimated 1.00 0.2910 1.00 0.94 0.94 1.00 1.00 0.5351 0.5351 0.9148 0.9150
50000 1.00 0.2913 1.00 0.94 0.94 1.00 1.00 0.5349 0.5349 0.9142 0.9152
50000 1.00 0.2897 1.00 0.94 0.94 1.00 1.00 0.5364 0.5364 0.9148 0.9150
50000 1.00 0.2912 1.00 0.94 0.94 1.00 1.00 0.5350 0.5350 0.9152 0.9152
50000 1.00 0.2903 1.00 0.94 0.94 1.00 1.00 0.5358 0.5358 0.9147 0.9145
50000 1.00 0.2926 1.00 0.94 0.94 1.00 1.00 0.5336 0.5336 0.9151 0.9151
50000 1.00 0.2932 1.00 0.94 0.94 1.00 1.00 0.5332 0.5332 0.9154 0.9149
Pear PCC SRC SI
Clearly, as and , this is a definite sign that this linear model has dependencies in the
input data as otherwise we would expect .
Model 3 Model 3 is the linear Sobol‘ function given by
A linear regression already shows all details needed, as .
Model 3-LHS mean variance R
expected 0.00 5442.25 1.00 0.153 0.100 0.063 0.037 0.020 0.010 0.004 0.001 0.000 0.000 0.000 0.000 0.000 0.001 0.004 0.010 0.020 0.037 0.063 0.100 0.153 0.224
estimated 0.00 5445.12 1.00 0.153 0.101 0.064 0.037 0.020 0.010 0.004 0.001 0.001 0.000 0.000 0.000 0.001 0.002 0.004 0.010 0.020 0.037 0.062 0.101 0.153 0.224
500 0.00 5347.77 1.00 0.187 0.133 0.115 0.050 0.019 0.035 0.041 0.042 0.034 0.020 0.041 0.018 0.032 0.034 0.016 0.038 0.062 0.072 0.068 0.098 0.165 0.253
1000 0.00 5415.56 1.00 0.172 0.103 0.056 0.063 0.027 0.024 0.023 0.017 0.017 0.016 0.020 0.014 0.008 0.017 0.010 0.020 0.029 0.045 0.052 0.113 0.170 0.277
5000 0.00 5383.01 1.00 0.155 0.092 0.055 0.032 0.016 0.010 0.006 0.003 0.002 0.002 0.001 0.002 0.003 0.007 0.005 0.012 0.022 0.043 0.064 0.106 0.159 0.247
10000 0.00 5451.07 1.00 0.158 0.109 0.064 0.039 0.022 0.013 0.005 0.004 0.001 0.001 0.001 0.001 0.002 0.001 0.005 0.010 0.024 0.039 0.064 0.098 0.147 0.219
50000 0.00 5419.32 1.00 0.158 0.099 0.067 0.035 0.021 0.010 0.004 0.002 0.001 0.000 0.000 0.000 0.000 0.001 0.003 0.010 0.020 0.036 0.060 0.101 0.150 0.222
SI
Only the results for the computation of the sensitivity indices are shown. The estimates for larger
sensitivity indices are good for small sampling sizes. However, to fix decimal places for the small SI
requires a large amount of samples.
Model 4a Model 4 is a monotonic model given by
pdf.joint with X+X=Y 21
22.=k,11)-(i=cU(0,1),~X ,0.5)-(X c= Y 2
ii
k
1i
ii
U(0,1).~X ,X+X=Y i
4
21
4
The linear regression only gives , hence the indicators Pear, PCC, and SRC are still
significant but do not explain everything. In this example, even small sampling sizes lead to good
estimates.
Model 4a-SRS mean variance R
expected 0.700 0.1544 0.89 0.7346 0.5876 0.9078 0.8660 0.5396 0.4604
estimated 0.701 0.1547 0.88 0.7345 0.5878 0.9078 0.8660 0.7345 0.5877 0.5395 0.4593
500 0.693 0.1557 0.88 0.7147 0.6237 0.8987 0.8717 0.7026 0.6098 0.5241 0.4857
1,000 0.704 0.1436 0.88 0.7124 0.5468 0.9065 0.8638 0.7615 0.6082 0.5173 0.4224
5,000 0.707 0.1521 0.88 0.7204 0.5844 0.9047 0.8669 0.7343 0.6014 0.5203 0.4589
10,000 0.698 0.1550 0.89 0.7378 0.5872 0.9097 0.8672 0.7363 0.5854 0.5453 0.4601
50,000 0.699 0.1550 0.89 0.7359 0.5898 0.9080 0.8662 0.7333 0.5865 0.5415 0.4611
Pear PCC SRC SI
Model 4c Model 4c exchanges the sampling distribution of Model 4a by using . The second input
parameter should now be significantly of more influence. This is clear when looking at the UA/SA
indicators.
Model 4c-SRS mean variance R
expected 127.50 27780 0.75 0.0087 0.8660 0.0173 0.8660 0.0001 0.9999
estimated 127.14 27689 0.75 0.0079 0.8658 0.0173 0.8658 0.0087 0.8658 0.0003 0.9962
500 121.71 27245 0.74 0.0377 0.8599 0.0725 0.8605 0.0370 0.8598 0.0293 0.9941
1,000 125.68 27596 0.75 0.0188 0.8631 0.0420 0.8633 0.0212 0.8631 0.0136 0.9953
5,000 125.91 26847 0.75 0.0077 0.8669 0.0104 0.8669 0.0052 0.8669 0.0038 0.9952
10,000 127.01 27722 0.75 0.0068 0.8666 0.0195 0.8666 0.0097 0.8666 0.0019 0.9966
50,000 126.76 27666 0.75 0.0078 0.8656 0.0215 0.8656 0.0108 0.8656 0.0004 0.9962
Pear PCC SRC SI
Model 5a Model 5 is an exponential function due to Sobol’, given by
For (a), we consider parameters with and . From this choice of parameters, the second to last input parameters should be treated roughly the same, which is validated by the data. For presentational purposes, only the indicators of the first, second, third and last input parameters are shown. Model 5a-SRS mean variance
expected 0.00 427.28 0.80 0.51 0.32 0.32 0.32 0.76 0.58 0.58 0.58 0.2870 0.1057 0.1057 0.1057
estimated 0.01 429.08 0.80 0.53 0.32 0.32 0.32 0.76 0.58 0.58 0.58 0.53 0.32 0.32 0.32 0.2866 0.1069 0.1059 0.1062
500 0.85 465.36 0.79 0.58 0.33 0.33 0.29 0.78 0.57 0.54 0.54 0.56 0.32 0.29 0.29 0.3756 0.1168 0.1490 0.1084
1000 -0.29 441.10 0.80 0.50 0.35 0.35 0.29 0.76 0.59 0.60 0.58 0.53 0.33 0.33 0.32 0.2730 0.1414 0.1428 0.0930
5000 -0.01 438.41 0.79 0.52 0.34 0.32 0.33 0.75 0.57 0.58 0.57 0.51 0.32 0.32 0.32 0.2883 0.1240 0.1144 0.1126
10000 0.06 437.38 0.79 0.52 0.33 0.32 0.31 0.75 0.57 0.57 0.58 0.53 0.32 0.32 0.32 0.2850 0.1115 0.1086 0.0975
50000 0.02 430.22 0.80 0.53 0.32 0.33 0.32 0.76 0.58 0.58 0.58 0.52 0.32 0.32 0.32 0.2887 0.1045 0.1086 0.1052
Pear PCC SRC SI
Model 5b For (b), we consider parameters with and . Only
the results for the first, the second, the last, and the last but one parameter are shown in the table
below. For indicators based on linear regression, few samples suffice to fix two decimal places. The
sensitivity indices, however, require a higher precision and hence more samples.
5
Model 5b-SRS mean variance R
expected 0.00 18022 0.81 0.24 0.24 0.16 0.16 0.47 0.47 0.32 0.32 0.0562 0.0562 0.0250 0.0250
estimated -0.10 18053 0.81 0.24 0.24 0.16 0.16 0.47 0.47 0.34 0.34 0.24 0.24 0.16 0.16 0.0560 0.0563 0.0245 0.0254
500 6.04 22037 0.81 0.21 0.23 0.20 0.22 0.47 0.44 0.32 0.37 0.24 0.22 0.16 0.18 0.0633 0.0865 0.0573 0.0851
1000 4.40 20645 0.82 0.22 0.23 0.21 0.16 0.46 0.47 0.34 0.31 0.23 0.23 0.16 0.14 0.0594 0.0701 0.0547 0.0351
5000 -2.68 18200 0.80 0.24 0.23 0.13 0.16 0.46 0.46 0.33 0.32 0.23 0.23 0.16 0.15 0.0602 0.0550 0.0184 0.0271
10000 -0.49 17723 0.81 0.25 0.24 0.15 0.16 0.48 0.48 0.34 0.34 0.24 0.23 0.16 0.16 0.0628 0.0576 0.0247 0.0273
50000 0.28 18038 0.81 0.24 0.23 0.15 0.16 0.48 0.47 0.33 0.34 0.24 0.24 0.15 0.16 0.0591 0.0541 0.0232 0.0260
Pear PCC SRC SI
Model 6a Model 6a is a quotient of powers given by
As all parameters are near 1, we expect a mostly linear behaviour, which is illustrated by the
following analysis.
Model 6a-SRS mean variance R
expected 1.030 0.0704 0.98 -0.45 0.89 -0.98 0.99 0.2023 0.7864
estimated 1.031 0.0706 0.98 -0.45 0.88 -0.96 0.99 -0.45 0.88 0.2044 0.7864
500 1.010 0.0640 0.98 -0.40 0.88 -0.96 0.99 -0.45 0.91 0.1814 0.7906
1000 1.030 0.0696 0.98 -0.44 0.88 -0.96 0.99 -0.45 0.89 0.2000 0.7863
5000 1.026 0.0705 0.98 -0.45 0.89 -0.96 0.99 -0.45 0.88 0.2043 0.7909
10000 1.028 0.0708 0.98 -0.46 0.89 -0.96 0.99 -0.45 0.88 0.2096 0.7902
50000 1.029 0.0709 0.98 -0.45 0.89 -0.96 0.99 -0.45 0.88 0.2082 0.7881
Pear PCC SRC SI
Model 6b Model 6b exchanges the sampling distribution of Model 6a by using . Hence the
nonlinear character of the powers is of much more influence which can be directly seen in the
following table.
Model 6b-SRS mean variance R
expected 2.017 6.901 0.68 -0.47 0.67 -0.64 0.76 0.2619 0.5110
estimated 2.015 6.891 0.67 -0.47 0.67 -0.63 0.76 -0.47 0.67 0.2606 0.5103
500 2.240 8.454 0.69 -0.46 0.66 -0.67 0.78 -0.50 0.69 0.2704 0.5070
1000 2.010 6.261 0.68 -0.50 0.66 -0.66 0.76 -0.49 0.66 0.3014 0.4880
5000 2.007 6.679 0.68 -0.45 0.68 -0.63 0.77 -0.47 0.69 0.2473 0.5078
10000 1.983 6.542 0.68 -0.47 0.67 -0.64 0.76 -0.47 0.68 0.2630 0.5044
50000 2.025 6.967 0.67 -0.47 0.67 -0.63 0.76 -0.47 0.67 0.2656 0.5072
Pear PCC SRC SI
The first order effects do not account for all of the variance.
Model 7 Model 7 is the so-called g-function of Sobol’, given by
The smaller , the more influential is the parameter . However, it is an even function so that
indicators based on linear regression are of no use.
6
Model 7-SRS mean variance R
expected 1.00 0.47 0.00 0.7165 0.1791 0.0237 0.0072 0.0001 0.0001 0.0001 0.0001
estimated 1.00 0.47 0.00 0.7142 0.1790 0.0240 0.0074 0.0003 0.0003 0.0004 0.0004
500 1.04 0.47 0.02 0.7556 0.2598 0.0483 0.0264 0.0306 0.0389 0.0262 0.0299
1000 0.98 0.47 0.00 0.7418 0.1894 0.0261 0.0171 0.0125 0.0126 0.0179 0.0106
5000 0.99 0.46 0.00 0.7173 0.1669 0.0319 0.0107 0.0025 0.0021 0.0026 0.0037
10000 1.00 0.47 0.00 0.7155 0.1834 0.0220 0.0067 0.0017 0.0007 0.0004 0.0023
50000 1.00 0.47 0.00 0.7158 0.1800 0.0239 0.0083 0.0003 0.0004 0.0004 0.0003
SI
Model 9 Model 9 is the following function suggested by Ishigami,
Model 9-SRS mean variance R
expected 3.50 13.85 0.19 0.44 0.00 0.00 0.3139 0.4424 0.0000
estimated 3.51 13.79 0.19 0.44 0.00 0.00 0.44 0.00 0.00 0.44 0.00 0.00 0.3128 0.4424 0.0003
500 3.69 13.80 0.22 0.46 0.00 -0.01 0.47 -0.06 0.01 0.47 -0.05 0.01 0.3097 0.4563 0.0101
1000 3.44 14.25 0.19 0.44 -0.03 -0.02 0.44 -0.03 -0.04 0.44 -0.03 -0.04 0.3380 0.4392 0.0080
5000 3.50 14.01 0.19 0.43 0.00 -0.02 0.43 0.00 -0.02 0.43 0.00 -0.02 0.3124 0.4505 0.0018
10000 3.49 13.77 0.18 0.43 0.02 0.00 0.43 0.01 0.00 0.43 0.01 0.00 0.3128 0.4511 0.0009
50000 3.53 13.87 0.19 0.44 0.00 0.00 0.44 0.00 0.00 0.44 0.00 0.00 0.3136 0.4378 0.0002
Pear PCC SRC SI
Note, that a first order index of 0 is always over-estimated. Furthermore, this gives a clue on the rate
of convergence.
Model 10 Model 10 is a function with 20 parameters, suggested by Morris. It is given by
The values for the parameters are not reproduced here. This model is normally used to study
screening methods. However, here only the standard UA/SA indicators were computed. The
estimates for and the sensitivity indices are reproduced here.
Model 10-SRS mean variance R
estimated 35.44 1078.06 0.45 0.01 0.01 0.02 0.01 0.02 0.00 0.07 0.10 0.15 0.10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
500 38.23 1047.18 0.52 0.04 0.06 0.03 0.06 0.03 0.02 0.09 0.17 0.22 0.12 0.04 0.02 0.04 0.01 0.02 0.01 0.05 0.03 0.04
1000 35.81 1177.00 0.48 0.02 0.02 0.04 0.02 0.03 0.02 0.11 0.14 0.16 0.08 0.02 0.01 0.02 0.01 0.02 0.01 0.01 0.00 0.02
5000 35.03 1086.69 0.42 0.02 0.01 0.02 0.01 0.02 0.01 0.05 0.09 0.16 0.10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
10000 35.46 1104.66 0.43 0.01 0.01 0.02 0.01 0.02 0.00 0.08 0.09 0.15 0.10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
50000 35.56 1054.79 0.45 0.01 0.01 0.02 0.01 0.02 0.00 0.07 0.10 0.15 0.10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
SI
Only parameters 7, 8, 9, and 10 show some effects
Model 11 This function models the height of a bungee jump depending on the weight of the jumper and other
parameters, given by
7
Model 11-SRS mean variance R
expected 18.24 75.24 0.44 0.01 0.55
estimated 18.24 75.16 0.98 0.67 -0.10 0.73 0.98 -0.62 0.98 0.67 -0.10 0.73 0.44 0.01 0.55
500 17.87 77.73 0.98 0.68 -0.10 0.74 0.98 -0.61 0.98 0.65 -0.10 0.72 0.47 0.04 0.57
1000 18.49 75.78 0.98 0.68 -0.18 0.72 0.98 -0.63 0.98 0.66 -0.11 0.71 0.48 0.04 0.54
5000 18.27 74.12 0.98 0.66 -0.09 0.72 0.98 -0.62 0.98 0.68 -0.10 0.73 0.44 0.01 0.54
10000 18.32 74.42 0.98 0.66 -0.13 0.72 0.98 -0.63 0.98 0.67 -0.11 0.73 0.44 0.02 0.54
50000 18.26 74.79 0.98 0.67 -0.11 0.73 0.98 -0.62 0.98 0.67 -0.11 0.73 0.44 0.01 0.54
Pear PCC SRC SI
Model 12a Model 12 measures the distance of a sample to two spheres, given by
where , and . The linear regression and the first
order effects reveal no detail about the function.
Model 12a-LHS mean variance R
estimated 0.34 0.05 0.00 0.128 0.128 0.127 0.127 0.128 0.128
500 0.34 0.05 0.00 0.161 0.096 0.183 0.140 0.137 0.150
1000 0.34 0.05 0.00 0.146 0.112 0.161 0.114 0.138 0.128
5000 0.34 0.05 0.00 0.147 0.145 0.127 0.130 0.131 0.121
10000 0.34 0.05 0.00 0.136 0.122 0.128 0.132 0.125 0.128
SI
Model 12b For Model 12b, we exchange the normal distribution with .
Model 12b-LHS mean variance R
estimated 0.16 0.02 0.00 0.037 0.037 0.037 0.037 0.037 0.037
500 0.16 0.02 0.01 0.061 0.082 0.065 0.082 0.058 0.054
1000 0.17 0.02 0.01 0.055 0.076 0.052 0.056 0.041 0.045
5000 0.16 0.02 0.00 0.045 0.044 0.036 0.038 0.044 0.041
10000 0.16 0.02 0.00 0.036 0.044 0.039 0.036 0.041 0.038
50000 0.16 0.02 0.00 0.040 0.039 0.040 0.037 0.036 0.039
SI
Model 13a For 13a, we model a smooth switch from 0 to , depending on the sign of , given by
The statistics reveal that the SI algorithm does not work well with fast changing or discontinuous
data.
8
Model 13a-SRS mean variance R
expected 0.50 0.40 0.5951 0.2066
estimated 0.50 0.40 0.67 0.68 0.46 0.76 0.62 0.68 0.45 0.5827 0.2074
500 0.51 0.41 0.67 0.67 0.46 0.76 0.64 0.68 0.47 0.5559 0.2161
1000 0.49 0.42 0.67 0.70 0.45 0.76 0.60 0.68 0.43 0.6141 0.2059
5000 0.50 0.41 0.67 0.68 0.45 0.77 0.62 0.68 0.45 0.5856 0.2044
10000 0.50 0.41 0.67 0.68 0.46 0.76 0.62 0.68 0.45 0.5816 0.2084
50000 0.50 0.40 0.67 0.68 0.46 0.76 0.62 0.68 0.45 0.5846 0.2081
Pear PCC SRC SI
Model 13b Model 13b switches from to , depending on the sign of by setting . For this model,
we predict a 0 first order effect for the second parameter, a fact that shows up in the following table.
Model 13b-SRS mean variance R
expected 0.00 0.32 0.7500 0.0000
estimated 0.00 0.32 0.58 0.76 0.00 0.76 0.00 0.76 0.00 0.7339 0.0003
500 0.04 0.34 0.59 0.76 0.04 0.76 0.03 0.76 0.02 0.7300 0.0124
1000 0.01 0.33 0.59 0.77 0.00 0.77 0.00 0.77 0.00 0.7418 0.0087
5000 0.00 0.32 0.59 0.76 -0.01 0.76 -0.02 0.76 -0.01 0.7353 0.0039
10000 0.00 0.32 0.59 0.77 0.00 0.77 -0.01 0.77 -0.01 0.7351 0.0005
50000 0.00 0.32 0.58 0.76 0.00 0.76 0.00 0.76 0.00 0.7303 0.0004
Pear PCC SRC SI
Here, the non-continuous behaviour leads to worse estimates of the SI for .
JRC’s contribution to the benchmark based on synthetic
PA cases.
R. Bolado & A. Badea
2
1. Introduction .............................................................................................................................3
2.- The Ishigami function (model 9) ............................................................................................4
3.- Hard switch model .................................................................................................................7
4.- Linear model with dependent inputs (model 2) .......................................................................7
5.- Sobol G function .................................................................................................................. 10
6.- Conclusions ......................................................................................................................... 14
7.- References ........................................................................................................................... 14
3
1. Introduction
In this work, JRC has studied the capability of correlation ratios (CRs) to estimate first order
sensitivity indices. The implementation of CRs considered is the one referred to in reference [1]
as CR, which sometimes is referred to in the same reference as VCE (variance of conditional
expected values). We would like to remark that in our understanding, and taking into account the
way we are implementing this technique, there should be no difference between VCE and ECV
(expected value of conditional variances). This is an issue to discuss with TUC.
All agreed models were studied: the Ishigami function (model 9), the hard switch function, the
linear model with dependent inputs (model 2) and the Sobol G function. Two issues are
addressed in this study: to compare our results with the results obtained with other partners using
either the same or other techniques and to study the dependence between the sensitivity indices
obtained and the number of subsamples (the sample at hand is divided in a number of
subsamples) used to compute the CRs. In order to study this dependence, for each sample size
we have considered the possibility of taking different number of subsamples. The cases
considered are summarised and labelled in table 1. Each possible case was run 25 times in order
to get an estimate of the expected sampling dispersion. Results obtained for each specific model
are reported in the next sections of this report. Boxplots are used as the main tool to show the
sampling dispersion. Theoretical values of sensitivity indices are represented in all boxplots as
black stars (*). As an example to interpret correctly many plots in the next pages and the
information contained in table 1, cases reported in boxplots and table 1 as 17 to 24 have been
obtained for samples of size 3000, with 2 to 500 subsamples (corresponding to 1500 to 6
observations per subsample).
Table 1.- Number of subsamples and corresponding subsample sizes for each sample size considered Sample
size Number of subsamples / subsample size
100 2/50 5/20 10/10 20/5 Cases 1 to 4
300 2/150 5/60 10/30 20/15 50/6 Cases 5 to 9
1000 2/500 5/200 10/100 20/50 50/20 100/10 200/5 Cases 10 to 16
3000 2/1500 5/600 10/300 20/150 50/60 100/30 200/15 500/6 ←Cases 17 to 24
and 25 to 34↓
10000 2/5000 5/2000 10/1000 20/500 50/200 100/100 200/50 500/60 1000/10 2000/5
In addition to boxplots, we have used the estimate of the mean squared error (MSE) as a
quantitative measure of results spread. In fact the MSE of any estimator used to estimate a
quantity is
2 22 2
ˆ ˆ ˆ ˆ ˆ ˆ( ) ( )MSE E E E E Bias Var
, (0.1)
4
which means that the dispersion of the estimator around its expected value (which is not
necessarily the value we want to estimate) splits into two parts: the square of the systematic error
committed and the variance of the estimator used.
2.- The Ishigami function (model 9)
Figures 1 to 9 report the results obtained for the Ishigami function. Figures 1, 3 and 5 report the
dispersion of sensitivity indices obtained respectively for input parameters X1, X2 and X3.
Codes 1 to 34 are interpreted as explained in the introduction. Figures 2, 4 and 6 show the MSE
obtained for each case and its split in squared systematic error and variance, Figures 7 to 9 are
respectively the scatterplots of the output versus X1, X2 and X3. These scatterplots are provided
to support and explain some conclusions about the results obtained.
Results for X1 and X2:
1 The dispersion of results decreases as the sample size increases (as expected).
2 In many cases (large and small number of subsamples/intervals) the estimates are biased
(boxplots show underestimation for small number of intervals and overestimation for
large number of intervals), but there is always a number of intervals for which the
estimator is unbiased, such a number is unknown though.
3 For large sample sizes (3000 and more clearly 10000), a kind of plateau is observed.
4 From the last two bullets we may deduce that the selection of the number of intervals is
less critical when the sample size increases. An intermediate number of intervals is
advised.
5 Figure 4, and specially figure 2, show that the bias is the main contributor to the MSE,
except when the estimators are close to being unbiased. In those cases the variance turns
the main contributor.
6 Figure 3 shows that when we consider only two intervals for performing the estimation,
the estimate is almost zero (specially in the case of large sample sizes). This is due to the
fact that the function studied is symmetric in X2 around 0 (see figure 8).
Results for X3:
1 The dispersion of results decreases as the sample size increases (as expected).
2 The best estimates are obtained always (all sample sizes) for the minimum possible
number of intervals (2).
3 For large sample sizes (3000 and more clearly 10000), a kind of plateau is observed.
4 From the last two bullets we may deduce that the selection of the number of intervals is
less critical when the sample size increases. A small number of intervals is advised.
5 Figure 6 shows that the main contributor to MSE is the bias except for very small number
of intervals. In that case, the contribution of both sources of variability is similar.
6 By default, conditional on the structure of the estimator used to compute first order
sensitivity indices (it is lower bounded by 0.0), when Si=0.0, estimates will always be
biased (expected value positive instead of null). The bias will be minimised when the
number of subsamples is 2 and it will decrease as sample size increases.
5
Figure 1.- Figure 2.-
Figure 3.- Figure 4.-
Figure 5.- Figure 6.-
6
Figure 7.-
Figure 8.-
Figure 9.-
7
3.- Hard switch model
Figures 10 to 15 report the results obtained for this model. Figures 10 and 12 report the
dispersion of sensitivity indices obtained respectively for input parameters X1 and X2. Codes 1
to 34 are interpreted as explained in the introduction. Figures 11 and 13 show the MSE obtained
for each case and its split in squared systematic error and variance, Figures 14 and 15 are
respectively the scatterplots of the output versus X1 and X2. These scatterplots are provided to
support and explain some conclusions about the results obtained.
Results for X1:
1 These results and the ones obtained for X1 and X2 in Ishigami’s function are very
similar.
2 When the number of intervals considered is 5 (second box for each sample size), the
sensitivity index is systematically underestimated. This is due to the fact that the interval
in the middle (see figure 14) takes roughly 50% of the output values before the jump
(X1=0.0) and 50% after it. The estimate of the output mean for this interval is very close
to the global output mean (0.0), producing a decrease in the estimation of the variance of
the conditional expected values (VCE). This effect should also appear in non-tested
moderate odd number of intervals such as 3, 7 and 9, and should decrease as the number
of intervals increases.
Results for X2
1 As for X3 in Ishigami’s function.
4.- Linear model with dependent inputs (model 2)
Figures 16 to 18 report the results for this model. Since the model is symmetric in X1 and X2,
and conditional on the way CR work, only results for one of them are reported, which is
represented as Xi.
Results for Xi:
1 Similar results as for the two first input parameters in Ishigami’s function and X1 in the
hard switch model.
2 MSEs are smaller that in the previous models, especially for large number of intervals
because being estimating a value quite close to 1.0.
3 The existence of the plateau is more obvious than for the previous models.
4 Same problem as in the hard switch function for X1 when the number of intervals is 5.
5 The sensitivity index is also systematically underestimated when the number of intervals
is 2.
8
Figure 10.- Figure 11.-
Figure 12.- Figure 13.-
Figure 14.- Figure 15.-
9
Figure 16.-
Figure 17.-
Figure 18.-
10
5.- Sobol G function
Results are reported in figures 19 to 33. Results are only reported for X1, X2, X3, X4 and X5,
given the equal importance of X5, X6, X7 and X8.
Results for X1:
1 Similar results as for other important input parameters in other models.
2 Large MSE, specially for 2 subsamples and for too many subsamples.
3 Existence of the plateau when the sample size is large.
Results for X2 to X8 (shown in pictures only up to X5):
1 Better estimates are continuously shifted towards the region of smaller number of
intervals for all sample sizes.
2 Best results are obtained for X5 to X8 (irrelevant inputs) when the number of intervals is
2.
11
Figure 19.- Figure 20.-
Figure 21.- Figure 22.-
Figure 23.- Figure 24.-
12
Figure 25.- Figure 26.-
Figure 27.- Figure 28.-
Figure 29.- Figure 30.-
13
Figure 31.-
Figure 32.-
Figure 33.-
14
6.- Conclusions
A study has been developed to test the dependence of the estimates of the first order sensitivity
indices using correlation ratios applied to four models. The main conclusions achieved are:
1 The results obtained strongly depend on: the sample size and the number of intervals
selected to apply the technique.
2 The thumb rule reported in reference [1] certainly is not optimal, but it is relatively good
to estimate moderate and large sensitivity indices. There is room for improvement.
3 The thumb rule reported in reference [1] systematically overestimates first order
sensitivity indices corresponding to non-important input parameters.
4 When first order sensitivity indices are null or close to zero, the best estimate is obtained
when the number of intervals is 2.
5 An iterative process should be developed to obtain better estimates. It should be based on
the detection of a plateau (point of inflection) in the estimation of moderate to large
sensitivity indices, in the asymptotic decrease towards zero of the estimates when
decreasing the number of intervals in the case of very small sensitivity indices and in the
asymptotic increase towards one of the estimates when increasing the number of intervals
in the case of very large sensitivity indices
Some updates of these conclusions could be done in the coming days.
7.- References
1 E. Plischke & K-J Röhlig. PAMINA milestone M2.1.D.11: Sensitivity analysis
benchmark based on the use of Synthetic PA cases (topic report).
2 D. Peña. Fundamentos de estadística. Alianza editorial.