neural network concepts and paradigms · computational intelligence in complex decision systems g....
TRANSCRIPT
![Page 1: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/1.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
Neural Network Concepts
and Paradigms
![Page 2: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/2.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
Biological NeuronNeurons: nerve cells; consist of dendrites, body and an axon;
signals flow through synapses.
Axon(output)
dendrites => body => axon
Axons connects to dendrites via synapses Synapses
vary in strength may be excitatory or inhibitory (+ or -)
an electrically excitable cell that processes and transmits
information through electrical and chemical signals.
![Page 3: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/3.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
Human Brain
on average the human brain has
• on average 86 billions neurons
• ~1000 main modules /~500 neural network each /~100 000 neurons
Properties of the brain:
It can learn, reorganize itself from experience
It adapts to the environment
It is robust and fault tolerant
![Page 4: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/4.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
Artificial Neural Networks
Some differences between biological neurons and
artificial neurons (processing elements - PE):
* Signs of weights (+ or -)
* Signals are ac in biological neurons, dc in PEs
* Many types of neurons in a system; usually only
a few at most in neural networks
* Basic cycle time for PC (<1 ns) faster than
brain (10-100ms) {as far as we know!}
An analysis paradigm very roughly modeled after themassively parallel structure of the brain.
Simulates a highly interconnected, parallel computationalstructure with numerous relatively simple individualprocessing elements (PE ).
![Page 5: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/5.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
What ANNs are and why they are useful
In the simplest terms, ANNs map input vectors to output vectors
ANNs consist of processing elements (PEs) or
neurons and weighted connections
• Operations performed by ANNs can include:* Fitting (approximation) and modeling* Pattern recognition (Classification)* Prediction* Pattern matching and completion* Noise removal* Optimization* Simulation* Control
![Page 6: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/6.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
What NNs are and why they are useful
• Connection weights store the information
• Weight values usually determined by a learning procedure
• Each PE acts independently of all others
• Each PE relies only on local information
• Connection pattern provides redundancy and facilitates fault tolerance
• ANNs are able to “learn” arbitrary non-linear mappings
![Page 7: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/7.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
7 / 20
Housing Price
Prediction
https://www.coursera.org/learn/neural-networks-deep-learning/lecture/eAE2G/what-is-a-neural-network
![Page 8: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/8.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
8 / 20
https://www.coursera.org/learn/neural-networks-deep-learning/lecture/2c38r/supervised-learning-with-neural-networks
![Page 9: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/9.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
9 / 20
https://www.coursera.org/learn/neural-networks-deep-learning/lecture/2c38r/supervised-learning-with-neural-networks
![Page 10: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/10.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
10 / 20
https://www.coursera.org/learn/neural-networks-deep-learning/lecture/2c38r/supervised-learning-with-neural-networks
People are just really good at interpreting unstructured data.
It has been much harder for computers to make sense of unstructured data compared to structured data.
Thanks to deep learning, thanks to neural networks, computers are now much better at interpreting unstructured data
![Page 11: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/11.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
11 / 20
![Page 12: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/12.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
Feedforward Neural Network
Input layer Hidden layer Output layer
![Page 13: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/13.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
Principal elements required to specify ANNs
Topology - layers and interconnections
Learning - how network is configured to store information
Recall - How stored information is retrieved
![Page 14: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/14.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
Neuron (PE) Model
- summing component
f - activation function
𝑠 = 𝑤1𝑥1 + 𝑤2𝑥2 + …+ 𝑤𝑁𝑥𝑁 + 𝑏
If N = 1, the weighted sum is in fact the simple linear regression
𝑠 = 𝑤1𝑥1 + 𝑏
If N > 1, the weighted sum is in fact the multiple linear regression
b is the intercept
![Page 15: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/15.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
Neuron (PE)
Model
TNxxxx ,...,, 21
Nwwww ,...,, 21
Input vector
Weight vector
Bias
N
i
ii bxwfysfy1
)(
bwxsbxwsN
i
ii 1
- summing component
f - activation function
b
multiple linear regression
![Page 16: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/16.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
Problem
5.2];5.025.0[
;4
2;2
bw
xN
)(sf
s = ?; y = ?
Implement in a programing language the above neuron
![Page 17: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/17.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
Neuron Model as Logistic Regression
TNxxxx ,...,, 21Input vector
Binary classification
1 – positive (in the class)
0 – negative (not in the class)
features vector (of the objects)
Output 𝑦 ∈ 0, 1
We want our model to predict the chance for the object
described by a set of features x to be in the class
𝑦 ∈ [0, 1] 0 ≤ 𝑦 ≤ 1
![Page 18: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/18.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
Neuron Model as Logistic Regression
TNxxxx ,...,, 21
Binary classification
𝑦 ∈ [0, 1] 0 ≤ 𝑦 ≤ 1
Activation function f
Sigmoidal (logistic function)
𝑦 =1
1+e−𝑠
![Page 19: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/19.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
19 / 20
Phyton 3.5 (Spyder) implementation using NumPy module
![Page 20: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/20.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
20 / 20
Phyton 3.5 (Spyder) results
x=24
w = [0.25, -0.5]b = 2.5
s= 1.0y = 0.7615941559557649
x=24-0.3
w = [0.25, -0.5, 1]b = 0.5
s= -1.3y = -0.8617231593133063
x=24-0.8
w = [0.25, -0.5, 1]b = -0.99
s=-3.29y = -0.9972281483227239
![Page 21: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/21.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
Activation(Transfer) Functions
hardlims
0,1
0,1)(
s
ssy
)(sf
)(sf
![Page 22: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/22.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
Activation(Transfer) Functions
)(sf
)(sf
![Page 23: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/23.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
Simulink Transfer Functions
![Page 24: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/24.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
ANN with 1 layer 2 inputs, 3 outputs
2
1
x
xx
3231
2221
1211
ww
ww
ww
w
3
2
1
b
b
b
b
3
2
1
s
s
s
s
3
2
1
o
o
o
o
bwxs )(sfo
3
2
1
2
1
3231
2221
1211
3
2
1
b
b
b
x
x
ww
ww
ww
s
s
s
![Page 25: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/25.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
Exercise
3
5.0
5.0
5.075.0
11.0
15.0
bw
0
10
4
2
4
1321 xxx
sesf
1
1)(
Compute the output vector o, for each input vector xi
Implement in a programing language the above example
![Page 26: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/26.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
26 / 20
Phyton 3.5 (Spyder) implementation
![Page 27: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/27.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
27 / 20
The one-layer user-defined function is written in a separate
file (ANNfunc.py)
![Page 28: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/28.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
28 / 20
The input is:[[-1. ][-2. ][ 0.33]]
The weights matrix is:[[-0.5 -0.15 -0.1 ][ 0.23 0.11 -0.5 ][-0.5 1. -0.1 ][ 0.045 0.15 -0.05 ]]
The bias is:[[ 0.2 ][-0.5 ][-0.03][ 0.36]]
The activation function is: tanh
The output is:[[ 0.747383 ][-0.80582272][-0.91590514][-0.0015 ]]
Phyton 3.5 (Spyder) results
![Page 29: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/29.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
It is common for different layers to have different numbers of neurons.
Multiple-layer networks are quite powerful.
Feedforward Neural Network
An ANN of two layers, where the first layer is sigmoid and the second layer is linear, can be trained to approximate any function (with a finite number of discontinuities) arbitrarily well - universal aproximator
![Page 30: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/30.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
Two layers: one hidden layer and one output layer
N inputs; H neurons in the hidden layer;
K outputs (neurons in the output layer)
hidden
layer
output
layer
output input
1x
hW
2x
Nx
1
hb1
hs1 hy1
1
1
hf
1
1
1
1o
oW
hs2
h
Hs
hf
hf
of
of
of
hb2
Hb
hy2
h
Hy
os1
os2
o
Ks
2o
Ko
ob1
ob2
o
Kb
∑
∑
∑
∑
∑
∑
![Page 31: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/31.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
hidden layer:
h
HN
h
H
h
H
h
N
hh
h
N
hh
h
www
www
www
w
21
22221
11211
a
H
a
a
a
b
b
b
b2
1
hidden
layer
output
layer
output input
1x
hW
2x
Nx
1
hb1
hs1 hy1
1
1
hf
1
1
1
1o
oW
hs2
h
Hs
hf
hf
of
of
of
hb2
Hb
hy2
h
Hy
os1
os2
o
Ks
2o
Ko
ob1
ob2
o
Kb
∑
∑
∑
∑
∑
∑
![Page 32: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/32.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
output layer:
o
KH
o
K
o
K
o
H
oo
o
H
oo
o
www
www
www
w
21
22221
11211
o
K
o
o
o
b
b
b
b2
1
hidden
layer
output
layer
output input
1x
hW
2x
Nx
1
hb1
hs1 hy1
1
1
hf
1
1
1
1o
oW
hs2
h
Hs
hf
hf
of
of
of
hb2
Hb
hy2
h
Hy
os1
os2
o
Ks
2o
Ko
ob1
ob2
o
Kb
∑
∑
∑
∑
∑
∑
![Page 33: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/33.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
hhhh bxwfy Th
H
hhh yyyy 21
ohoo bywfO
ohhhoo bbxwfwfO
Transfer function:
hidden
layer
output
layer
output input
1x
hW
2x
Nx
1
hb1
hs1 hy1
1
1
hf
1
1
1
1o
oW
hs2
h
Hs
hf
hf
of
of
of
hb2
Hb
hy2
h
Hy
os1
os2
o
Ks
2o
Ko
ob1
ob2
o
Kb
∑
∑
∑
∑
∑
∑
![Page 34: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/34.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
Exporting an ANN trained in Matlabto simulate it in another environment (e.g. Python)
In Matlab, training a ANN generally involves a normalization step that is
applied to both the input vectors and the target vectors in the data set.
The network output always falls into a normalized range, [-1, 1].
The network output can then be reverse transformed back into the units
of the original target data when the network is put to use in the field
(simulated in Matlab).
For correct export of the ANN, remove all preprocessing and
postprocessing functions that MATLAB executes on inputs and outputs,
(or implement those function in the new environment)
For a two-layer ANN, in the script (function) used to create and train the
ANN, write:
net.inputs{1}.processFcns = {}; % for inputs
net.outputs{2}.processFcns = {}; % for targets
Default setting is: net.inputs{1}.processFcns = {'removeconstantrows','mapminmax'};
net.outputs{2}.processFcns = {'removeconstantrows','mapminmax’};
![Page 35: Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons](https://reader035.vdocuments.net/reader035/viewer/2022071008/5fc5fc88f0e9cf459235aa94/html5/thumbnails/35.jpg)
Computational Intelligence in Complex Decision Systems G. Oltean
Exporting an ANN trained in Matlabto simulate it in another environment (e.g. Python) – cont.
The parameters of the trained ANN (e.g. for a two-layer ANN)
wh=cell2mat(net.IW) %weights of the hidden layer
wo=cell2mat(net.LW) %weights of the output layer
bh=cell2mat(net.b(1)) % bias of the hidden layer
bo=cell2mat(net.b(2)) % bias of the output layer
afunh=net.layers{1}.transferFcn % activation function in the
hidden layer
afunh=net.layers{2}.transferFcn % activation function in the
output layer
In Matlab:
Observation: “net” is the name of the ANN