![Page 1: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/1.jpg)
Structure learning
with deep neuronal networks
6th Network Modeling Workshop, 6/6/2013
Patrick Michl
![Page 2: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/2.jpg)
Page 26/6/2013
Patrick MichlNetwork Modeling
Agenda
Autoencoders
Biological Model
Validation & Implementation
![Page 3: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/3.jpg)
Page 36/6/2013
Patrick MichlNetwork Modeling
Real world data usually is high dimensional …
x1
x2
Dataset Model
Autoencoders
![Page 4: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/4.jpg)
Page 46/6/2013
Patrick MichlNetwork Modeling
… which makes structural analysis and modeling complicated!
x1
x2
x1
x2
Dataset Model
?
Autoencoders
![Page 5: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/5.jpg)
Page 56/6/2013
Patrick MichlNetwork Modeling
Dimensionality reduction techinques like PCA …
x1
x2
PCA
Dataset Model
Autoencoders
![Page 6: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/6.jpg)
Page 66/6/2013
Patrick MichlNetwork Modeling
… can not preserve complex structures!
x1
x2
PCA
Dataset Model
x1
x2
𝑥2=α 𝑥1+β
Autoencoders
![Page 7: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/7.jpg)
Page 76/6/2013
Patrick MichlNetwork Modeling
Therefore the analysis of unknown structures …
x1
x2
Dataset Model
Autoencoders
![Page 8: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/8.jpg)
Page 86/6/2013
Patrick MichlNetwork Modeling
… needs more considerate nonlinear techniques!
x1
x2
Dataset Model
x1
x2
𝑥2= 𝑓 (𝑥1)
Autoencoders
![Page 9: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/9.jpg)
Page 96/6/2013
Patrick MichlNetwork Modeling
Autoencoders are artificial neuronal networks …
Autoencoder
• Artificial Neuronal Network
Autoencoders
input data X
output data X‘
Perceptrons
Gaussian Units
![Page 10: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/10.jpg)
Page 106/6/2013
Patrick MichlNetwork Modeling
Autoencoders are artificial neuronal networks …
Autoencoder
• Artificial Neuronal Network
Autoencoders
input data X
output data X‘
Perceptrons
Gaussian Units
Perceptron1
0
Gauss UnitsR
![Page 11: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/11.jpg)
Page 116/6/2013
Patrick MichlNetwork Modeling
Autoencoders are artificial neuronal networks …
Autoencoder
• Artificial Neuronal Network
Autoencoders
input data X
output data X‘
Perceptrons
Gaussian Units
![Page 12: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/12.jpg)
Page 126/6/2013
Patrick MichlNetwork Modeling
Autoencoder
• Artificial Neuronal Network• Multiple hidden layers
Autoencoders
… with multiple hidden layers.
Gaussian Units
input data X
output data X‘
Perceptrons
(Visible layers)
(Hidden layers)
![Page 13: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/13.jpg)
Page 136/6/2013
Patrick MichlNetwork Modeling
Autoencoder
• Artificial Neuronal Network• Multiple hidden layers
Autoencoders
Such networks are called deep networks.
Gaussian Units
input data X
output data X‘
Perceptrons
(Visible layers)
(Hidden layers)
![Page 14: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/14.jpg)
Page 146/6/2013
Patrick MichlNetwork Modeling
Autoencoder
• Artificial Neuronal Network• Multiple hidden layers
Autoencoders
Such networks are called deep networks.
Gaussian Units
input data X
output data X‘
Perceptrons
(Visible layers)
(Hidden layers)Definition (deep network)
Deep networks are artificial neuronal networks with multiple hidden layers
![Page 15: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/15.jpg)
Page 156/6/2013
Patrick MichlNetwork Modeling
Autoencoder
Autoencoders
Gaussian Units
input data X
output data X‘
Perceptrons
(Visible layers)
(Hidden layers)
Such networks are called deep networks.
• Deep network
![Page 16: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/16.jpg)
Page 166/6/2013
Patrick MichlNetwork Modeling
Autoencoder
Autoencoders
Autoencoders have a symmetric topology …
Gaussian Units
input data X
output data X‘
Perceptrons
(Visible layers)
(Hidden layers)
• Deep network• Symmetric topology
![Page 17: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/17.jpg)
Page 176/6/2013
Patrick MichlNetwork Modeling
Autoencoder
Autoencoders
… with an odd number of hidden layers.
Gaussian Units
input data X
output data X‘
Perceptrons
(Visible layers)
(Hidden layers)
• Deep network• Symmetric topology
![Page 18: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/18.jpg)
Page 186/6/2013
Patrick MichlNetwork Modeling
Autoencoder
Autoencoders
The small layer in the center works lika an information bottleneck
input data X
output data X‘
• Deep network• Symmetric topology• Information bottleneck
Bottleneck
![Page 19: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/19.jpg)
Page 196/6/2013
Patrick MichlNetwork Modeling
Autoencoder
Autoencoders
... that creates a low dimensional code for each sample in the input data.
input data X
output data X‘
• Deep network• Symmetric topology• Information bottleneck
Bottleneck
![Page 20: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/20.jpg)
Page 206/6/2013
Patrick MichlNetwork Modeling
Autoencoder
Autoencoders
The upper stack does the encoding …
input data X
output data X‘
• Deep network• Symmetric topology• Information bottleneck• Encoder
Encoder
![Page 21: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/21.jpg)
Page 216/6/2013
Patrick MichlNetwork Modeling
Autoencoder
Autoencoders
… and the lower stack does the decoding.
input data X
output data X‘
• Deep network• Symmetric topology• Information bottleneck• Encoder• Decoder
Encoder
Decoder
![Page 22: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/22.jpg)
Page 226/6/2013
Patrick MichlNetwork Modeling
• Deep network• Symmetric topology• Information bottleneck• Encoder• Decoder
Autoencoder
Autoencoders
… and the lower stack does the decoding.
input data X
output data X‘
Encoder
Decoder
Definition (deep network)
Deep networks are artificial neuronal networks with multiple hidden layers
Definition (autoencoder)
Autoencoders are deep networks with a symmetric topology and an odd number of hiddern layers, containing a encoder, a low dimensional representation and a decoder.
![Page 23: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/23.jpg)
Page 236/6/2013
Patrick MichlNetwork Modeling
Autoencoder
Autoencoders
Autoencoders can be used to reduce the dimension of data …
input data X
output data X‘
Problem: dimensionality of data
Idea:1. Train autoencoder to minimize the distance
between input X and output X‘2. Encode X to low dimensional code Y3. Decode low dimensional code Y to output X‘4. Output X‘ is low dimensional
![Page 24: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/24.jpg)
Page 246/6/2013
Patrick MichlNetwork Modeling
Autoencoder
Autoencoders
… if we can train them!
input data X
output data X‘
Problem: dimensionality of data
Idea:1. Train autoencoder to minimize the distance
between input X and output X‘2. Encode X to low dimensional code Y3. Decode low dimensional code Y to output X‘4. Output X‘ is low dimensional
![Page 25: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/25.jpg)
Page 256/6/2013
Patrick MichlNetwork Modeling
Autoencoder
Autoencoders
In feedforward ANNs backpropagation is a good approach.
input data X
output data X‘
Training
Backpropagation
![Page 26: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/26.jpg)
Page 266/6/2013
Patrick MichlNetwork Modeling
Backpropagation
Autoencoder
Autoencoders
input data X
output data X‘
Training
Definition (autoencoder)
Backpropagation
(1) The distance (error) between current output X‘ and wanted output Y is computed. This gives a error function
error
In feedforward ANNs backpropagation is a good approach.
![Page 27: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/27.jpg)
Page 276/6/2013
Patrick MichlNetwork Modeling
Backpropagation
Autoencoder
Autoencoders
In feedforward ANNs backpropagation is the choice
input data X
output data X‘
Training
Definition (autoencoder)
Backpropagation
(1) The distance (error) between current output X‘ and wanted output Y is computed. This gives a error function
Example (linear neuronal unit with two inputs)
![Page 28: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/28.jpg)
Page 286/6/2013
Patrick MichlNetwork Modeling
Backpropagation
Autoencoder
Autoencoders
input data X
output data X‘
Training
Definition (autoencoder)
Backpropagation
(1) The distance (error) between current output X‘ and wanted output Y is computed. This gives a error function
(2) By calculating we get a vector that shows in a direction which decreases the error
(3) We update the parameters to decrease the error
In feedforward ANNs backpropagation is a good approach.
![Page 29: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/29.jpg)
Page 296/6/2013
Patrick MichlNetwork Modeling
Backpropagation
Autoencoder
Autoencoders
In feedforward ANNs backpropagation is the choice
input data X
output data X‘
Training
Definition (autoencoder)
Backpropagation
(1) The distance (error) between current output X‘ and wanted output Y is computed. This gives a error function
(2) By calculating we get a vector that shows in a direction which decreases the error
(3) We update the parameters to decrease the error(4) We repeat that
![Page 30: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/30.jpg)
Page 306/6/2013
Patrick MichlNetwork Modeling
Autoencoder
Autoencoders
… the problem are the multiple hidden layers!
input data X
output data X‘
Training
Backpropagation
Problem: Deep Network
![Page 31: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/31.jpg)
Page 316/6/2013
Patrick MichlNetwork Modeling
Autoencoder
Autoencoders
input data X
output data X‘
Training
Backpropagation is known to be slow far away from the output layer …
Backpropagation
Problem: Deep Network• Very slow training
![Page 32: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/32.jpg)
Page 326/6/2013
Patrick MichlNetwork Modeling
Autoencoder
Autoencoders
input data X
output data X‘
Training
… and can converge to poor local minima.
Backpropagation
Problem: Deep Network• Very slow training• Maybe bad solution
![Page 33: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/33.jpg)
Page 336/6/2013
Patrick MichlNetwork Modeling
Autoencoder
Autoencoders
input data X
output data X‘
Training
Backpropagation
Problem: Deep Network• Very slow training• Maybe bad solution
Idea: Initialize close to a good solution
The task is to initialize the parameters close to a good solution!
![Page 34: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/34.jpg)
Page 346/6/2013
Patrick MichlNetwork Modeling
Autoencoder
Autoencoders
input data X
output data X‘
Training
Backpropagation
Problem: Deep Network• Very slow training• Maybe bad solution
Idea: Initialize close to a good solution• Pretraining
Therefore the training of autoencoders has a pretraining phase …
![Page 35: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/35.jpg)
Page 356/6/2013
Patrick MichlNetwork Modeling
Autoencoder
Autoencoders
input data X
output data X‘
Training
Backpropagation
Problem: Deep Network• Very slow training• Maybe bad solution
Idea: Initialize close to a good solution• Pretraining• Restricted Boltzmann Machines
… which uses Restricted Boltzmann Machines (RBMs)
![Page 36: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/36.jpg)
Page 366/6/2013
Patrick MichlNetwork Modeling
Autoencoder
Autoencoders
input data X
output data X‘
Training
Backpropagation
Problem: Deep Network• Very slow training• Maybe bad solution
Idea: Initialize close to a good solution• Pretraining• Restricted Boltzmann Machines
… which uses Restricted Boltzmann Machines (RBMs)
Restricted Boltzmann Machine
• RBMs are Markov Random Fields
![Page 37: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/37.jpg)
Page 376/6/2013
Patrick MichlNetwork Modeling
Autoencoder
Autoencoders
input data X
output data X‘
Training
Backpropagation
Problem: Deep Network• Very slow training• Maybe bad solution
Idea: Initialize close to a good solution• Pretraining• Restricted Boltzmann Machines
… which uses Restricted Boltzmann Machines (RBMs)
Restricted Boltzmann Machine
• RBMs are Markov Random Fields
Markov Random Field
Every unit influences every neighbor
The coupling is undirected
Motivation (Ising Model)A set of magnetic dipoles (spins)
is arranged in a graph (lattice)
where neighbors are
coupled with a given strengt
![Page 38: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/38.jpg)
Page 386/6/2013
Patrick MichlNetwork Modeling
Autoencoder
Autoencoders
input data X
output data X‘
Training
Backpropagation
Problem: Deep Network• Very slow training• Maybe bad solution
Idea: Initialize close to a good solution• Pretraining• Restricted Boltzmann Machines
… which uses Restricted Boltzmann Machines (RBMs)
Restricted Boltzmann Machine
• RBMs are Markov Random Fields• Bipartite topology: visible (v), hidden (h)• Use local energy to calculate the probabilities of values
Training:contrastive divergency(Gibbs Sampling)
h1
v1 v2 v3 v4
h2 h3
![Page 39: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/39.jpg)
Page 396/6/2013
Patrick MichlNetwork Modeling
Autoencoder
Autoencoders
input data X
output data X‘
Training
Backpropagation
Problem: Deep Network• Very slow training• Maybe bad solution
Idea: Initialize close to a good solution• Pretraining• Restricted Boltzmann Machines
… which uses Restricted Boltzmann Machines (RBMs)
Restricted Boltzmann Machine
Gibbs Sampling
![Page 40: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/40.jpg)
Page 406/6/2013
Patrick MichlNetwork Modeling
Autoencoders
Autoencoder
The top layer RBM transforms real value data into binary codes.
Top
Training
![Page 41: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/41.jpg)
Page 416/6/2013
Patrick MichlNetwork Modeling
Autoencoders
Autoencoder
Top
Therefore visible units are modeled with gaussians to encode data …
h2
v1 v2 v3 v4
h3 h4 h5h1
Training
![Page 42: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/42.jpg)
Page 426/6/2013
Patrick MichlNetwork Modeling
Autoencoders
Autoencoder
Top
… and many hidden units with simoids to encode dependencies
h2
v1 v2 v3 v4
h3 h4 h5h1
Training
![Page 43: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/43.jpg)
Page 436/6/2013
Patrick MichlNetwork Modeling
Autoencoders
Autoencoder
Top
The objective function is the sum of the local energies.
Local Energy
𝐸𝑣≔−∑h
𝑤 h𝑣
𝑥𝑣
𝜎𝑣
𝑥h
+(𝑥𝑣−𝑏𝑣 )2
2𝜎 𝑣2
h2
v1 v2 v3 v4
h3 h4 h5h1
Training
![Page 44: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/44.jpg)
Page 446/6/2013
Patrick MichlNetwork Modeling
Autoencoders
Autoencoder
Reduction
The next RBM layer maps the dependency encoding…
Training
![Page 45: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/45.jpg)
Page 456/6/2013
Patrick MichlNetwork Modeling
Autoencoders
Autoencoder
Reduction
… from the upper layer …
v
h1
v1 v2 v3 v4
h2 h3
Training
![Page 46: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/46.jpg)
Page 466/6/2013
Patrick MichlNetwork Modeling
Autoencoders
Autoencoder
Reduction
… to a smaller number of simoids …
h
h1
v1 v2 v3 v4
h2 h3
Training
![Page 47: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/47.jpg)
Page 476/6/2013
Patrick MichlNetwork Modeling
Autoencoders
Autoencoder
Reduction
… which can be trained faster than the top layer
Local Energy𝐸𝑣≔−∑
h
𝑤 h𝑣 𝑥𝑣 𝑥h+𝑥h𝑏h
𝐸h≔−∑𝑣
𝑤 h𝑣 𝑥𝑣 𝑥h+𝑥𝑣𝑏𝑣
h1
v1 v2 v3 v4
h2 h3
Training
![Page 48: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/48.jpg)
Page 486/6/2013
Patrick MichlNetwork Modeling
Autoencoders
Autoencoder
Unrolling
The symmetric topology allows us to skip further training.
Training
![Page 49: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/49.jpg)
Page 496/6/2013
Patrick MichlNetwork Modeling
Autoencoders
Autoencoder
Unrolling
The symmetric topology allows us to skip further training.
Training
![Page 50: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/50.jpg)
Page 506/6/2013
Patrick MichlNetwork Modeling
After pretraining backpropagation usually finds good solutions
Autoencoders
Autoencoder
Training
• PretrainingTop RBM (GRBM)Reduction RBMsUnrolling
• FinetuningBackpropagation
![Page 51: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/51.jpg)
Page 516/6/2013
Patrick MichlNetwork Modeling
The algorithmic complexity of RBM training depends on the network size
Autoencoders
Autoencoder
Training
• Complexity: O(inw)i: number of iterationsn: number of nodesw: number of weights
• Memory Complexity: O(w)
![Page 52: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/52.jpg)
Page 526/6/2013
Patrick MichlNetwork Modeling
Agenda
Autoencoders
Biological Model
Validation & Implementation
![Page 53: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/53.jpg)
Page 536/6/2013
Patrick MichlNetwork Modeling Network Modeling
Restricted Boltzmann Machines (RBM)
How to model the topological structure?
S
E
TF
![Page 54: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/54.jpg)
Page 546/6/2013
Patrick MichlNetwork Modeling
We define S and E as visible data Layer …
S
E
TF
Network ModelingRestricted Boltzmann Machines (RBM)
![Page 55: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/55.jpg)
Page 556/6/2013
Patrick MichlNetwork Modeling
S E
TF
Network ModelingRestricted Boltzmann Machines (RBM)
We identify S and E with the visible layer …
![Page 56: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/56.jpg)
Page 566/6/2013
Patrick MichlNetwork Modeling
S E
… and the TFs with the hidden layer in a RBM
TF
Network ModelingRestricted Boltzmann Machines (RBM)
![Page 57: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/57.jpg)
Page 576/6/2013
Patrick MichlNetwork Modeling
S E
The training of the RBM gives us a model
TF
Network ModelingRestricted Boltzmann Machines (RBM)
![Page 58: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/58.jpg)
Page 586/6/2013
Patrick MichlNetwork Modeling
Agenda
Autoencoder
Biological Model
Implementation & Results
![Page 59: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/59.jpg)
Page 596/6/2013
Patrick MichlNetwork Modeling
Results
Validation of the results
• Needs information about the true regulation• Needs information about the descriptive power of the data
![Page 60: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/60.jpg)
Page 606/6/2013
Patrick MichlNetwork Modeling
Results
Validation of the results
• Needs information about the true regulation• Needs information about the descriptive power of the data
Without this infomation validation can only be done,
using artificial datasets!
![Page 61: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/61.jpg)
Page 616/6/2013
Patrick MichlNetwork Modeling
Results
Artificial datasets
We simulate data in three steps:
![Page 62: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/62.jpg)
Page 626/6/2013
Patrick MichlNetwork Modeling
Results
Artificial datasets
We simulate data in three steps
Step 1
Choose number of Genes (E+S) and create random bimodal distributed data
![Page 63: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/63.jpg)
Page 636/6/2013
Patrick MichlNetwork Modeling
Results
Artificial datasets
We simulate data in three steps
Step 1
Choose number of Genes (E+S) and create random bimodal distributed data
Step 2
Manipulate data in a fixed order
![Page 64: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/64.jpg)
Page 646/6/2013
Patrick MichlNetwork Modeling
Results
Artificial datasets
We simulate data in three steps
Step 1
Choose number of Genes (E+S) and create random bimodal distributed data
Step 2
Manipulate data in a fixed order
Step 3
Add noise to manipulated data
and normalize data
![Page 65: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/65.jpg)
Page 656/6/2013
Patrick MichlNetwork Modeling
Simulation
Results
Step 1Number of visible nodes 8 (4E, 4S)
Create random data:
Random {-1, +1} + N(0,
![Page 66: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/66.jpg)
Page 666/6/2013
Patrick MichlNetwork Modeling
Simulation
Results
NoiseStep 2Manipulate data
![Page 67: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/67.jpg)
Page 676/6/2013
Patrick MichlNetwork Modeling
Simulation
Results
Step 3Add noise: N(0,
![Page 68: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/68.jpg)
Page 686/6/2013
Patrick MichlNetwork Modeling
Results
We analyse the data Xwith an RBM
![Page 69: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/69.jpg)
Page 696/6/2013
Patrick MichlNetwork Modeling
Results
We train an autoencoder with 9 hidden layersand 165 nodes:
Layer 1 & 9: 32 hidden unitsLayer 2 & 8: 24 hidden unitsLayer 3 & 7: 16 hidden unitsLayer 4 & 6: 8 hidden unitsLayer 5: 5 hidden units
input data X
output data X‘
![Page 70: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/70.jpg)
Page 706/6/2013
Patrick MichlNetwork Modeling
Results
We transform the data from X to X‘And reduce the dimensionality
![Page 71: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/71.jpg)
Page 716/6/2013
Patrick MichlNetwork Modeling
Results
We analyse thetransformed data X‘with an RBM
![Page 72: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/72.jpg)
Page 726/6/2013
Patrick MichlNetwork Modeling
Results
Lets compare the models
![Page 73: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/73.jpg)
Page 736/6/2013
Patrick MichlNetwork Modeling
Results
Another Example with more nodes and larger autoencoder
![Page 74: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/74.jpg)
Page 746/6/2013
Patrick MichlNetwork Modeling
Conclusion
Conclusion
• Autoencoders can improve modeling significantly by reducing the dimensionality of data
• Autoencoders preserve complex structures in their multilayer perceptron network. Analysing those networks (for example with knockout tests) could give more structural information
• The drawback are high computational costsSince the field of deep learning is getting more popular (Face recognition / Voice recognition, Image transformation). Many new improvements in facing the computational costs have been made.
![Page 75: Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl](https://reader038.vdocuments.net/reader038/viewer/2022103123/56649d1f5503460f949f2c96/html5/thumbnails/75.jpg)
Page 756/6/2013
Patrick MichlNetwork Modeling
Acknowledgement
eilsLABS
Prof. Dr. Rainer König
Prof. Dr. Roland Eils
Network Modeling Group