![Page 1: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/1.jpg)
http://bit.ly/idl2018
Xavier [email protected]
Associate ProfessorUniversitat Politecnica de CatalunyaTechnical University of Catalonia
The Neural Network Zoo(& some research at UPC)
Day 4 Lecture 4
#DLUPC
![Page 2: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/2.jpg)
2
Acknowledgements
Fjodor Van Veen, “The Neural Network Zoo”
The Asimov Institute (2016)
![Page 3: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/3.jpg)
3
Acknowledgements
![Page 4: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/4.jpg)
4
A Perceptron
J. Alammar, “A visual and interactive guide to the Basics of Neural Networks” (2016)F. Van Veen, “The Neural Network Zoo” (2016)
![Page 5: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/5.jpg)
5
Neural Network = Multi Layer Perceptron
2 layers of perceptronsF. Van Veen, “The Neural Network Zoo” (2016)
![Page 6: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/6.jpg)
6
Deep Neural Network (DNN)
F. Van Veen, “The Neural Network Zoo” (2016)
![Page 7: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/7.jpg)
7Alex Graves, “Supervised Sequence Labelling with Recurrent Neural Networks”
The hidden layers and the output depend from previous
states of the hidden layers
Recurrent Neural Network (RNN)
![Page 8: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/8.jpg)
8
Recurrent Neural Network (RNN)
F. Van Veen, “The Neural Network Zoo” (2016)
![Page 9: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/9.jpg)
9
Recurrent Neural Network (RNN)
F. Van Veen, “The Neural Network Zoo” (2016)
![Page 10: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/10.jpg)
10
Recurrent Neural Network (RNN)
Víctor Campos, Brendan Jou, Jordi Torres, Xavier Giró-i-Nieto, Shih-Fu Chang, “Skip RNN: Learning to Skip State Updates in RNNs”. NIPS Workshop on Time Series 2018.
![Page 11: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/11.jpg)
11
Recurrent Neural Network (RNN)
Deep Learning TV, “Recurrent Neural Networks - Ep. 9”
![Page 12: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/12.jpg)
12
Convolutional Neural Network (CNN)
F. Van Veen, “The Neural Network Zoo” (2016)
![Page 13: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/13.jpg)
13
Convolutional Neural Network (CNN)
LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278-2324.
LeNet-5
![Page 14: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/14.jpg)
14
Convolutional Neural Network (CNN)
Orange
A Krizhevsky, I Sutskever, GE Hinton “Imagenet classification with deep convolutional neural networks” NIPS 2012.
AlexNet
![Page 15: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/15.jpg)
15
Convolutional Neural Network (CNN)
Junting Pan, Kevin McGuinness, Elisa Sayrol, Noel O'Connor, and Xavier Giro-i-Nieto. "Shallow and Deep Convolutional Networks for Saliency Prediction." CVPR 2016.
JuntingNet
![Page 16: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/16.jpg)
16
Convolutional Neural Network (CNN)
Marc Assens, Kevin McGuinness, Noel E. O’Connor, Xavier Giró-i-Nieto, “SaltiNet: Scan-path Prediction on 360 Degree Images using Saliency Volumes”. ICCVW 2017. (winner Salient360 2017)
[ 600 x 300 ]
SaltiNet
![Page 17: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/17.jpg)
17
Convolutional Neural Network (CNN)
Deep Learning TV, “Convolutional Neural Networks - Ep. 8”
![Page 18: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/18.jpg)
18
Deconvolutional Neural Network (Deconv)
Junting Pan, SalGAN (2017)F. Van Veen, “The Neural Network Zoo” (2016)
![Page 19: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/19.jpg)
19
Deep Residual Learning / Skip connections
F. Van Veen, “The Neural Network Zoo” (2016)
He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Deep Residual Learning for Image Recognition." CVPR 2016 [slides]
![Page 20: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/20.jpg)
20Figure: Kilian Weinberger
Deep Residual Learning / Skip connections
![Page 21: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/21.jpg)
21Long, Jonathan, Evan Shelhamer, and Trevor Darrell. "Fully convolutional networks for semantic segmentation." CVPR 2015 & PAMI 2016.
Deep Residual Learning / Skip connections
![Page 22: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/22.jpg)
22Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation." MICCAI 2015
U-Net
Deep Residual Learning / Skip connections
![Page 23: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/23.jpg)
23
CNN + RNN + Skip connections
Amaia Salvador, Míriam Bellver, Manel Baradad, Ferran Marqués, Jordi Torres, Xavier Giró-i-Nieto, “Recurrent Neural Networks for Semantic Instance Segmentation”. NIPS WiML 2017.
![Page 24: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/24.jpg)
24
Dense connections
Dense Block of 5-layers with a growth rate of k=4
Connect every layer to every other layer of the same filter size.
Huang, Gao, Zhuang Liu, Kilian Q. Weinberger, and Laurens van der Maaten. "Densely connected convolutional networks." CVPR 2017. [code]
DenseNet
![Page 25: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/25.jpg)
25
Dense connections
Huang, Gao, Zhuang Liu, Kilian Q. Weinberger, and Laurens van der Maaten. "Densely connected convolutional networks." CVPR 2017. [code]
DenseNet
![Page 26: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/26.jpg)
26
Dense connections
Huang, Gao, Zhuang Liu, Kilian Q. Weinberger, and Laurens van der Maaten. "Densely connected convolutional networks." CVPR 2017 [code]
![Page 27: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/27.jpg)
27
Dense connections
Jégou, Simon, Michal Drozdzal, David Vazquez, Adriana Romero, and Yoshua Bengio. "The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation." CVPRW 2017 [code] [slides]
![Page 28: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/28.jpg)
28
Autoencoder (AE)
F. Van Veen, “The Neural Network Zoo” (2016)
![Page 29: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/29.jpg)
29
Variational Autoencoder (VAE)
F. Van Veen, “The Neural Network Zoo” (2016)
![Page 30: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/30.jpg)
30F. Van Veen, “The Neural Network Zoo” (2016)
Restricted Boltzmann Machine (RBM)
![Page 31: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/31.jpg)
31
Deep Belief Networks (DBN)
F. Van Veen, “The Neural Network Zoo” (2016)
![Page 32: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/32.jpg)
32
Adversarial Networks
Goodfellow, Ian, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. "Generative adversarial nets." NIPS 2014Goodfellow, Ian. "NIPS 2016 Tutorial: Generative Adversarial Networks." arXiv preprint arXiv:1701.00160 (2016).
F. Van Veen, “The Neural Network Zoo” (2016)
![Page 33: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/33.jpg)
33
Adversarial Networks
Junting Pan, Cristian Canton, Kevin McGuinness, Noel E. O’Connor, Jordi Torres, Elisa Sayrol and Xavier Giro-i-Nieto. “SalGAN: Visual Saliency Prediction with Generative Adversarial Networks.” CVPRW 2017.
Introduce an additional term to the less to improve performance.
![Page 34: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/34.jpg)
34F. Van Veen, “The Neural Network Zoo” (2016)
Graves, Alex, Greg Wayne, and Ivo Danihelka. "Neural turing machines." arXiv preprint arXiv:1410.5401 (2014). [slides] [code]
Differentiable Neural Computers (DNC)
![Page 35: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/35.jpg)
35
von Neumann architecture (1952) Neural Turing Machine (2014)
Differentiable Neural Computers (DNC)
Graves, Alex, Greg Wayne, and Ivo Danihelka. "Neural turing machines." arXiv preprint arXiv:1410.5401 (2014). [slides] [code]
End-to-end trainable
![Page 36: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/36.jpg)
36
Differentiable Neural Computers (DNC)Add a trainable external memory to a neural network.
Graves, Alex, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo et al. "Hybrid computing using a neural network with dynamic external memory." Nature 538, no. 7626 (2016): 471-476. [Post by DeepMind]
![Page 37: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/37.jpg)
37
Differentiable Neural Computers (DNC)DNC can solve tasks reading information from a trained memory.
Graves, Alex, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo et al. "Hybrid computing using a neural network with dynamic external memory." Nature 538, no. 7626 (2016): 471-476. [Post by DeepMind]
![Page 38: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/38.jpg)
38
Differentiable Neural Computers (DNC)
[Siraj Raval on YouTube]
![Page 39: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/39.jpg)
39
More architectures to come...
Yann LeCun, “A Path to AI”, Beneficial AI 2017.
von Neumann architecture (1952)
![Page 40: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/40.jpg)
40
The Full Story
F. Van Veen, “The Neural Network Zoo” (2016)
![Page 41: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/41.jpg)
41
The Prequel
Fjodor Van Veen, “Neural Network Zoo Prequel: Cells and Layers”(2017)
![Page 42: Day 4 Lecture 4 - raw.githubusercontent.com · Differentiable Neural Computers (DNC) Add a trainable external memory to a neural network. Graves, Alex, Greg Wayne, Malcolm Reynolds,](https://reader033.vdocuments.net/reader033/viewer/2022041421/5e1ef860c9c89262d54fcb6d/html5/thumbnails/42.jpg)
42
Questions ?