Robust Pareto Design of GMDH-type Neural Networks for Systems with Probabilistic Uncertainties

N. Nariman-zadeh, F. Kalantary, A. Jamali, F. Ebrahimi

Faculty of Engineering,

The University of Guilan

•System identification techniques are applied in many fields in order to model and predict the behaviors of unknown and/or very complex systems based on given input-output data

•GMDH is a self-organizing approach by which gradually complicated models are generated based on the evaluation of their performances on a set of multi-input-single-output data

•In order to obtain more robust models, it is required to consider all the conflicting objectives, namely, training error (TE), prediction error (PE) in the sense of multi-objective Pareto optimization process

•For multi-objective optimization problems, there is a set of optimal solutions, known as Pareto optimal solutions or Pareto front

Introduction

•System Identification Techniques Are Applied in Many Fields in order to Model and Predict the Behaviors of Unknown and/or Very Complex Systems Based on Given Input-Output Data.

•Group Method of Data Handling (GMDH) Algorithm is Self-Organizing Approach by which Gradually Complicated Models are Generated Based on the Evaluation of their Performances on a set of Multi-Input-Single-Output Data Pairs (i=1, 2, …, M)

X1

X2

Xn

Y1

.

.Ym

Modelling Using GMDH-type Networks

The classical GMDH algorithm can be represented as set of neurons in which different pairs of them in each layer are connected through a quadratic polynomial and thus produce new neurons in the next layer.

G1

G2

G4

G6

X1

X2

X3

X4

A Feedforward GMDH-Type Network

G3

G5

Input Layer

Output Layer

Hidden Layer(s)

Modelling Using GMDH-type Networks

A Generalized GMDH Network Structure of a Chromosome

a

c

b

d

ad

bc

adbc

a d b c b c b c

LayerHiddenNeuronofLength 2.

Application of Genetic Algorithm in the Topology Design of GMDH-type NNs

a

c

b

d

ad

bc

adbc

a d b c d d d d

Application of Genetic Algorithm in the Topology Design of GMDH-type NNs

Crossover operation for two individuals in GS-GMDH networks

Application of Singular Value Decomposition to the Design of GMDH-type Networks

SVD is the method for solving most linear least squares problems that some singularities may exist in the normal equations YA a

The SVD of a matrix, , is a factorization of the matrix into the product of three matrices, matrix , diagonal matrix with non-negative elements (Singular Values), and orthogonal matrix such that :

M NA R M NU R

N NW R N NV R

. . TA U W V YUw

diagV T

j

..)1

(.a

Genetic Algorithms and Multi-objective Pareto Optimization

Genetic algorithms are iterative and stochastic

optimization techniques.

In the optimization of complex In the optimization of complex real-world problems, there are real-world problems, there are

several objective functions to be several objective functions to be optimized simultaneously.optimized simultaneously.

In the optimization of complex In the optimization of complex real-world problems, there are real-world problems, there are

several objective functions to be several objective functions to be optimized simultaneously.optimized simultaneously.

There is There is no single optimal solutionno single optimal solution as the best because objectives as the best because objectives

conflict each other.conflict each other.

There is There is no single optimal solutionno single optimal solution as the best because objectives as the best because objectives

conflict each other.conflict each other.There is a set of optimal solutions, There is a set of optimal solutions,

well known as well known as Pareto optimal Pareto optimal solutions solutions oror Pareto front. Pareto front.

There is a set of optimal solutions, There is a set of optimal solutions, well known as well known as Pareto optimal Pareto optimal

solutions solutions oror Pareto front. Pareto front.

Modelling error

Multi-objective optimization

Modelling error

Multi-objective optimization

Difference between robust optimization and traditional optimization

Design Variable

Obj

ectiv

e F

unct

ion

Feasible

Infeasible

Optimal solution

Robust optimal solution

Random variable

0.25

0.50

0.75

1.00

PDF

CDF

p

x

XX dxxfxXxF Pr

xf X

dxxfxxdFXE XX

dxxfXExX X

2

For the discrete sampling:

N

iixN

XE1

1

N

ii XEx

NX

1

22

11

Stochastic Robust Analysis

•Modelling and prediction of soil shear strength, Su , based on 5 input parameters, namely, SPT number (Standard Penetration Test) N′, effective overburden stress s/

0, moisture content percent W , LL liquid limit, and PL plastic limit of fine-graded clay soil

•The data used in this study were gathered from the National Iranian Geotechnical Database, which has been set up in the Building and Housing Research Centre (BHRC)

•The database has been established under a mandate from the Management and Planning Organization (MPORG), which supervises the professional activities of all of the consultancy firms in Iran

Comparison of actual values with the evolved GMDH model corresponding to optimum point C (nominal table)

Training set Prediction set

Point Network’s structure TE PE Mean of TE Mean of PE Variance of TE Variance of PE

A bbaebcacbcaeacee 133.12 48.49 323.76 161.49 174862.64 42019.59

B bcaebacdbcbbadde 79.20 260.15 73785.2 17844.7 3.8e11 3.3e9

C bcaebccdbdbcaccd 89.79 75.30 28366.5 709.8 3.7e10 2.6e6

Objective functions and structure of networks of different optimum design points

Point Network’s structure TE PE Mean of TE

Mean of PE Variance of TE Variance of PE

A bbaebcacbcaeacee 133.12 48.49 323.76 161.49 174862.64 42019.59B bcaebacdbcbbadde 79.20 260.15 73785.2 17844.7 3.8e11 3.3e9C bcaebccdbdbcaccd 89.79 75.30 28366.5 709.8 3.7e10 2.6e6D abeecddd 132.79 237.59 234 .61 248.03 178.77 1174.283

Objective functions and structure of networks of different optimum design points

Y1

Y4

Y3

Y5Y2

Point CPoint D

The structure of network corresponding to point C and D

Y1=-5.94+ 0.65 N’ + 0.76 σ0’ -0.0083 N’2 - 0.0019 σ0

’2 + 0.0013 N’ σ0’

Y2= 25.42 - 2.76w + 1.86LL - 0.019w2 - 0.045LL2 + 0.11w(LL)

Y3= 16.99 + 0.82Y2 - 1.27LL - 0.0015Y22 + 0.016(LL)2 + 0.015(Y2)(LL)

Y4= 10.16 + 0.74Y1 - 0.22PL - 0.019Y12 - 0.034PL2 + 0.056(Y1)(PL)

Y5= 16.12 + 0.83Y4 - 0.64Y3 - 0.0004Y42 + 0.0060Y32+ 0.0036(Y4)(Y3)

Conclusion

• A multi-objective genetic algorithm was used to optimally design GMDH-type neural networks from a robustness point

of view in a probabilistic approach.

• Multi-objective optimization of robust GMDH models led to the discovering some important trade-off among those

objective functions.

• The framework of this work is very promising and can be generally used in the optimum design of GMDH models in

real-world complex systems with probabilistic uncertainties.

Thanks for your attention…