![Page 1: Smoothing Spline ANOVA for variable screening](https://reader031.vdocuments.net/reader031/viewer/2022012012/61582c1d291ec148032b268b/html5/thumbnails/1.jpg)
a useful tool for metamodels training
and multi-objective optimization
Smoothing Spline ANOVA
for variable screening
L. Ricco, E. Rigoni, A. Turco
![Page 2: Smoothing Spline ANOVA for variable screening](https://reader031.vdocuments.net/reader031/viewer/2022012012/61582c1d291ec148032b268b/html5/thumbnails/2.jpg)
Outline
RSM • Introduction
• Possible coupling
• Test case
MOO
• MOO with Game Theory
• Possible rankings
• Test Cases
• Interesting Application
![Page 3: Smoothing Spline ANOVA for variable screening](https://reader031.vdocuments.net/reader031/viewer/2022012012/61582c1d291ec148032b268b/html5/thumbnails/3.jpg)
SS-Anova
Smoothing Spline ANOVA (SS-ANOVA) is a statistical
modeling algorithm based on a function decomposition
similar to the classical analysis of variance (ANOVA)
decomposition and the associated notions of main effects
and interactions.
Each term –main effects and interactions– exhibits the
measure of its contribution to the global variance (so its
relative significance).
SS-ANOVA is a suitable screening technique for detecting
important variables in a given dataset
![Page 4: Smoothing Spline ANOVA for variable screening](https://reader031.vdocuments.net/reader031/viewer/2022012012/61582c1d291ec148032b268b/html5/thumbnails/4.jpg)
Univariate case: Cubic Smoothing Spline
In the simple univariate case, the SS-ANOVA model f(x) is
the solution of this minimization problem:
The left term guarantees a good fit to the data.
The right term represents a penalty on the roughness of the
model.
The solution is called Cubic Smoothing Spline, and it
corresponds to the usual natural cubic spline.
n
i
ii dxxfxffn 1
1
0
2''2)()(
1min
![Page 5: Smoothing Spline ANOVA for variable screening](https://reader031.vdocuments.net/reader031/viewer/2022012012/61582c1d291ec148032b268b/html5/thumbnails/5.jpg)
General multivariate case
In the general multivariate case, the model is the solution of
this penalized least square problem:
The ANOVA decomposition can be built into the above
formulation through the proper construction of the
roughness functional J(f).
The theory behind this formulation is based on the so-called
reproducing kernel Hilbert space.
n
i
ii fJxffn 1
2)()(
1min
![Page 6: Smoothing Spline ANOVA for variable screening](https://reader031.vdocuments.net/reader031/viewer/2022012012/61582c1d291ec148032b268b/html5/thumbnails/6.jpg)
Screening with SS-ANOVA
The function ANOVA decomposition of the trained model,
evaluated at sampling points, can be written as:
The relative significance of the different terms can be
assessed by means of the contribution indices
Given the aforementioned formulas, we obtain a sort of
decomposition of unity:
2*
**
f
)f,f( kk
p
k
k
1
** f f
p
k
k
1
1
![Page 7: Smoothing Spline ANOVA for variable screening](https://reader031.vdocuments.net/reader031/viewer/2022012012/61582c1d291ec148032b268b/html5/thumbnails/7.jpg)
Metamodeling
A RSM use the information given by a training database
to predict the response of the system at unsampled
points.
![Page 8: Smoothing Spline ANOVA for variable screening](https://reader031.vdocuments.net/reader031/viewer/2022012012/61582c1d291ec148032b268b/html5/thumbnails/8.jpg)
Metamodeling
PROs:
Training and evaluating a RSM is usually less costly than
running a real simulation, both considering computational
resources and time.
CONs:
The curse of dimensionality is the main restriction. For
example the number of monomials composing a full
polynomial of degree “deg” using “nVar” variables is
equal to:
!deg!
)!(deg
nVar
nVar
![Page 9: Smoothing Spline ANOVA for variable screening](https://reader031.vdocuments.net/reader031/viewer/2022012012/61582c1d291ec148032b268b/html5/thumbnails/9.jpg)
SS-Anova and Metamodeling
SS-Anova is able to scan a complex and possibly large
database. Once the most relevant input variables are
detected, it is possible to restrict the training improving
time and accuracy performances.
![Page 10: Smoothing Spline ANOVA for variable screening](https://reader031.vdocuments.net/reader031/viewer/2022012012/61582c1d291ec148032b268b/html5/thumbnails/10.jpg)
An Example (mimicking a true database)
The introduction of SS-Anova in our software is linked to
a customer request having a very peculiar database. We
cannot show their data, but we built a toy model with
similar properties, which helped us in the development
phase.
We consider a polynomial of
degree 3 in 6 variables, but
we set to zero all the
coefficients of the terms
involving the last 3 variables.
![Page 11: Smoothing Spline ANOVA for variable screening](https://reader031.vdocuments.net/reader031/viewer/2022012012/61582c1d291ec148032b268b/html5/thumbnails/11.jpg)
An Example (mimicking a true database)
We build randomly a training (140 points) and a validation
(60 points) database and we compare the performances of
different RSM before and after the use of SS-Anova.
Interpolation methods performed better on this toy problem,
while Kriging model (including noise) was the best in the
original test.
Mean absolute error comparison
Full training Using SS-ANOVA
Kriging 0.01 0.0001
RBF 3.0E-5 1.0E-7
SVD 2.0E-15 6.0E-16
The original test case
results in a improvement
of about 3 orders of
magnitude
![Page 12: Smoothing Spline ANOVA for variable screening](https://reader031.vdocuments.net/reader031/viewer/2022012012/61582c1d291ec148032b268b/html5/thumbnails/12.jpg)
Multi Objective Optimization
Mathematical formulation:
When m>1 and the functions are in contrast, we speak about multi-objective optimization.
The goal of MOO is to find the Pareto front composed by non-dominated points.
![Page 13: Smoothing Spline ANOVA for variable screening](https://reader031.vdocuments.net/reader031/viewer/2022012012/61582c1d291ec148032b268b/html5/thumbnails/13.jpg)
Game Theory and MOO
![Page 14: Smoothing Spline ANOVA for variable screening](https://reader031.vdocuments.net/reader031/viewer/2022012012/61582c1d291ec148032b268b/html5/thumbnails/14.jpg)
Game Theory and MOO
SS - Anova
![Page 15: Smoothing Spline ANOVA for variable screening](https://reader031.vdocuments.net/reader031/viewer/2022012012/61582c1d291ec148032b268b/html5/thumbnails/15.jpg)
Ranking Variables
We designed two possible ranking and selection strategies.
1) Deterministic:
• Normalize SS-Anova coefficients
• Look for the highest coefficient and record its
variable-objective association.
• Remove all the coefficients related to the assigned
variable and iterate
2) Stochastic:
• Normalize SS-Anova coefficients
• For each variable perform roulette
wheel selection using the different
coefficients as weights in order to
assign the variable to an objective.
In both cases we check for objectives without assigned
variables.
SS-Anova coefficient for variable x
Obj A
Obj B
Obj C
![Page 16: Smoothing Spline ANOVA for variable screening](https://reader031.vdocuments.net/reader031/viewer/2022012012/61582c1d291ec148032b268b/html5/thumbnails/16.jpg)
Test Cases: full separation of variables
k
ki
ii
k
i
ii
xxf
xxf
2
1
2
1
1
)))30(2(sin(3.0)30(
)))20(2(sin(3.0)20(
This problem has a clear and unique answer to the
variable-objective partition problem, but it is a difficult
optimization task, since the objective functions exhibit
several local optima.
k=2 # eval = 300
min(f1+f2)
Old MOGT 1.3536
New MOGT 0.0071
MOGA2 6.2607
NSGA2 1.7086
k=5 # eval = 1600
min(f1+f2)
Old MOGT 5.8995
New MOGT 2.6502
MOGA2 2.7829
NSGA2 2.6939
k=10 # eval = 6000
min(f1+f2)
Old MOGT 14.365
New MOGT 3.0213
MOGA2 4.4598
NSGA2 2.2674
![Page 17: Smoothing Spline ANOVA for variable screening](https://reader031.vdocuments.net/reader031/viewer/2022012012/61582c1d291ec148032b268b/html5/thumbnails/17.jpg)
Test Cases: shared variables
100
1
38.0
2
99
1
2
1
2
1
sin5
2.0exp10
i
ii
i
ii
xxf
xxf
KUR100 problem:
As highlighted in the
original paper, this
algorithm is able to
reach almost-optimal
configurations in few
iterations.
The stochastic variable-
objective coupling is
used for this problem.
![Page 18: Smoothing Spline ANOVA for variable screening](https://reader031.vdocuments.net/reader031/viewer/2022012012/61582c1d291ec148032b268b/html5/thumbnails/18.jpg)
Application: boomerang optimization
![Page 19: Smoothing Spline ANOVA for variable screening](https://reader031.vdocuments.net/reader031/viewer/2022012012/61582c1d291ec148032b268b/html5/thumbnails/19.jpg)
Application: boomerang optimization
The optimization follows a bi-level scheme, the outer loop
tries to optimize the shape of the boomerang, the inner one
ensures a satisfactory trajectory.
With the new MOGT algorithm we are trying to go further
and to fully optimize the launch: maximize range, minimize
force, loop constraint.
![Page 20: Smoothing Spline ANOVA for variable screening](https://reader031.vdocuments.net/reader031/viewer/2022012012/61582c1d291ec148032b268b/html5/thumbnails/20.jpg)