jump to first page. the generalised mapping regressor (gmr) neural network for inverse discontinuous...
TRANSCRIPT
Jump to first page
Jump to first page
The Generalised Mapping Regressor (GMR) neural
network for inverse discontinuous problems
The Generalised Mapping Regressor (GMR) neural
network for inverse discontinuous problems
Student : Chuan LU
Promotor : Prof. Sabine Van Huffel
Daily Supervisor : Dr. Giansalvo Cirrincione
Jump to first page
Mapping Approximation Problem
Feedforward neural networks are : universal approximators of nonlinear continuous
functions (many-to-one, one-to-one) they don’t yield multiple solutions they don’t yield infinite solutions they don’t approximate mapping discontinuities
Jump to first page
Inverse and Discontinuous Problems
Mapping : multi-valued, complex structure.
conditional average of the target data
Poor representation of the mapping by least squares approach (sum-of-squares error function) for feedforward neural networks.
Mapping with discontinuities.
Jump to first page
gatinggatingnetworknetwork
Network 1 Network 2 Network 3
inputinput
outputoutputmixture-of-experts
It partitions the solution between several networks. It uses a separate network to determine the parameters of each kernel, with a further network to determine the coefficients.
winner-take-all
• Jacobs and Jordan• Bishop (ME extension)
kernel blending
Jump to first page
Example #1
ME
MLP
Jump to first page
Example #2
ME
MLP
Jump to first page
Example #3
ME
MLP
Jump to first page
Example #4
ME
MLP
Jump to first page
Generalised Mapping Regressor( GMR )
(G. Cirrincione and M. Cirrincione, 1998)
approximate every kind of function or relation.
input : collection of components of x and y output : estimation of the remaining components output all solutions, mapping branches, equilevel hypersurfaces.
Characteristics :
nm yxyxM :),(
Jump to first page
coarse-to-fine learning incremental competitive based on mapping recovery (curse of dimensionality)
topological neuron linking distance direction
linking tracking branches contours
open architecture
function approximation pattern recognition
Z (augmented) space unsupervised learning
GMR Basic Ideas
clusters mapping branches
Jump to first page
GMR four phases
object merged
Object Merging
Learning Recall-ing
branch 1branch 2
INPUTINPUT
Linking
links
object 1
pool of neurons
object 2object 3
TrainingTrainingSetSet
Jump to first page
EXIN Segmentation Neural Network (EXIN SNN)
clustering
(G. Cirrincione, 1998)
w4= x4
vigilance threshold
x
Input/weight space
Z (augmented) space
coarse quantization• EXIN SNN• high z ( say 1 )
branch (object)neuron
GMR Learning
Z (augmented) space
• production phase• Voronoi sets domain setting
GMR Learning
Z (augmented) space
• secondary EXIN SNNs• z = 2 < 1
TS#1
TS#2
TS#3
TS#4
TS#5
Other levels are possible
fine quantization
GMR Learning
-0.6 -0.4 -0.2 0 0.2 0.4 0.6-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1PLN Level 1 1=0.2, epoch1=3
x
-0.6 -0.4 -0.2 0 0.2 0.4 0.6-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1PLN Level 1 1=0.2, epoch1=3
x
GMR Coarse to fine Learning ( Example)
-0.6 -0.4 -0.2 0 0.2 0.4 0.6-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1PLN level 1-2, 1=0.2, epoch1=3;2=0.1, epoch2=3
* 1st PLN: 13*
x
y
* 2nd PLN: 24*
-0.6 -0.4 -0.2 0 0.2 0.4 0.6-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1PLN level 1-2, 1=0.2, epoch1=3;2=0.1, epoch2=3
* 1st PLN: 13*
x
y
* 2nd PLN: 24*
object neuron
fine VQ neurons
object neuron Voronoi set
Jump to first page
GMR Linking Voronoi set: setup of the neuron radius (domain variable)
neuron i
ri
asymmetric radius
Task 1 :Task 1 : Task 1 :Task 1 :
Jump to first page
Weight Space
GMR Linking For one TS presentation:
zi
d1
w1
w5
w3
w4
d1
w2
d5
d3
d4
d2
branch and bound search technique
k-nn
Linking candidates
distance test direction test create a link or strengthen a link
Task 2 :Task 2 : Task 2 :Task 2 :
Linking direction
Jump to first page
Branch and Bound Accelerated Linking
neuron tree constructed during learning phase (multilevel EXIN SNN learning)
methods in linking candidate step (k-nearest-neighbors computation): -BnB : < d1 , ( : linking factor predefined) k-BnB : k predefined.
Jump to first page
44 43
3127
6459
5547
7681 80 83
0,00%10,00%20,00%30,00%40,00%50,00%60,00%70,00%80,00%90,00%
2-D (TS 2k): 8 2-D(TS 4k): 24 3-D (TS 3k): 199 linking flops (x100,000)
percents of linking flops saved by branch and bound
2-level d-BnB
2-level k-BnB
3-level d-BnB
3-level k-BnB
GMR Linking
branch-and-bound in linking experimental results:83 %
Jump to first page
branch and bound (cont.)
Apply branch and bound in learning phase ( labelling ) :
Tree construction k-means EXIN SNN
Experimental results (in the 3-D example) 50% of labeling flops are saved
GMR Linking Example
-0.6 -0.4 -0.2 0 0.2 0.4 0.6-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
x
y
Linking: = 2.5
-0.6 -0.4 -0.2 0 0.2 0.4 0.6-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
x
y
Linking: = 2.5
link
GMR Merging Example
-0.6 -0.4 -0.2 0 0.2 0.4 0.6-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
x
y
Merging: threshold = 1
Obj: 13 -> 3
-0.6 -0.4 -0.2 0 0.2 0.4 0.6-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
x
y
Merging: threshold = 1
Obj: 13 -> 3
-0.6 -0.4 -0.2 0 0.2 0.4 0.6-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
x
y
x = 0.2
Level 1 neurons: 3
-0.6 -0.4 -0.2 0 0.2 0.4 0.6-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
x
y
x = 0.2
Level 1 neurons: 3
GMR Recalling Example
)04.0
01.0)2(sin(
4
1)(
2
xxxfy )
04.0
01.0)2(sin(
4
1)(
2
xxxfy
level 1 neuron
level 2 neuron
branch 1
branch 2
-0.6 -0.4 -0.2 0 0.2 0.4 0.6-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
x
y
y = 0.6
Level 1 neurons: 1
-0.6 -0.4 -0.2 0 0.2 0.4 0.6-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
x
y
y = 0.6
Level 1 neurons: 1
level one neurons : input within their domain level two neurons : only connected ones level zero neurons : isolated (noise)
Experiments
spiral of Archimedes = a (a = 1)
spiral of Archimedes = a (a = 1)
Experiments
Sparse regions
further normalizing + higher mapping resolution
)04.0
01.0)2(sin(
4
1)(
2
xxxfy )
04.0
01.0)2(sin(
4
1)(
2
xxxfy
Experiments noisy data
1 Bernoulli of lemniscate
222222
a
yxayx 1 Bernoulli of lemniscate
222222
a
yxayx
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
x
y
Solutions for y = -0.5
Level 1 neurons: 6
Experiments
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
x
y
Solutions for y = -0.1
Level 1 neurons: 10
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
x
y
Solutions for y = 0.5
Level 1 neurons: 5
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
x
y
Solutions for y = 1
Level 1 neurons: 19
5,3 Lissajous of curve
sin,cos
ba
btyatx 5,3 Lissajous of curve
sin,cos
ba
btyatx
Experiments
contours : links among
level one neurons
GMR mapping of 8 spheres in a 3-D scene.
Jump to first page
Conclusions
GMR is able to : solve inverse discontinuous problems approximate every kind of mapping
yield all the solutions and the corresponding branches
GMR can be accelerated by applying tree search techniques
GMR needs : interpolation techniques kernels or projection techniques for high dimensional data adaptive parameters
Jump to first page
Thank you !(shi-a shi-a)
l1 = 0b1 = 0
l1 = 0b1 = 0
l6 = 0b6 = 0
l6 = 0b6 = 0
l5 = 0b5 = 0
l5 = 0b5 = 0
l2= 0b2= 0
l2= 0b2= 0
l3 = 0b3 = 0
l3 = 0b3 = 0 l4 = 0
b4 = 0
l4 = 0b4 = 0
l7 = 0b7 = 0
l7 = 0b7 = 0
l8= 0b8 = 0
l8= 0b8 = 0
l3 = 2b3 = 1
l3 = 2b3 = 1
GMR Recall
input
w1
w2
w3
w7
w8
w4
w5
w6
r1
l1 = 1b1 = 1
l1 = 1b1 = 1
linking tracking
restricted distance
level one test
connected neuron :level zero level two
branch the winner branch
GMR Recall
input
w1
w2
w3
w7
w8
l1 = 0b1 = 0
l1 = 0b1 = 0
l6 = 0b6 = 0
l6 = 0b6 = 0
l5 = 0b5 = 0
l5 = 0b5 = 0
l2= 0b2= 0
l2= 0b2= 0
l3 = 0b3 = 0
l3 = 0b3 = 0 l4 = 0
b4 = 0
l4 = 0b4 = 0
l7 = 0b7 = 0
l7 = 0b7 = 0
l8= 0b8 = 0
l8= 0b8 = 0
w4
w5
w6
r2
l1 = 1b1 = 1
l1 = 1b1 = 1
l3 = 2b3 = 1
l3 = 2b3 = 1
l2= 1b2= 2
l2= 1b2= 2 l2= 1b2= 1
l2= 1b2= 1
level one test
linking tracking
branchcross
GMR Recall
l6 = 0b6 = 0
l6 = 0b6 = 0 l6 = 2b6 = 4
l6 = 2b6 = 4 l6 = 1b6 = 6
l6 = 1b6 = 6
input
w1
w2
w3
l1 = 0b1 = 0
l1 = 0b1 = 0
l5 = 0b5 = 0
l5 = 0b5 = 0
l2= 0b2= 0
l2= 0b2= 0
l3 = 0b3 = 0
l3 = 0b3 = 0 l4 = 0
b4 = 0
l4 = 0b4 = 0
l7 = 0b7 = 0
l7 = 0b7 = 0
l8= 0b8 = 0
l8= 0b8 = 0
w4
w5
w6
l1 = 1b1 = 1
l1 = 1b1 = 1
l3 = 2b3 = 1
l3 = 2b3 = 1
l2= 1b2= 2
l2= 1b2= 2 l2= 1b2= 1
l2= 1b2= 1
l4 = 1b4 = 4
l4 = 1b4 = 4
l5 = 2b5 = 4
l5 = 2b5 = 4 l4 = 1b4 = 5
l4 = 1b4 = 5 l4 = 1b4 = 4
l4 = 1b4 = 4
… until completion of the candidates
level one neurons : input within their domain level two neurons : only connected ones level zero neurons : isolated (noise)
w7
w8
l6 = 1b6 = 4
l6 = 1b6 = 4
clipping
Tow Branches
Tow Branches
Two Branches
Two Branches
GMR Recall
input
w1
w2
w3
w7
w8
l7 = 0b7 = 0
l7 = 0b7 = 0
l8= 0b8 = 0
l8= 0b8 = 0
w4
w5
w6
Output = weight complements of the level one neurons Output interpolation
l1 = 1b1 = 1
l1 = 1b1 = 1
l3 = 2b3 = 1
l3 = 2b3 = 1
l2= 1b2= 1
l2= 1b2= 1
l4 = 1b4 = 4
l4 = 1b4 = 4
l4 = 1b4 = 4
l4 = 1b4 = 4 l6 = 1
b6 = 4
l6 = 1b6 = 4