deep learning / representation learning
TRANSCRIPT
28.02.2019
1
Deep/representation learning – concept
1
Deep learning / representationlearning
• Machine learning – standard approach
• Deep learning
– Learning representation
– Perspectives
28.02.2019
2
Machine learning – standard approach
• Machine learning:develop patterns from data– E-mail spam recognition
– Prediction of credit risk
– Risk of reoccurrence of cancer (MammaPrint)
– Prediction of chemiotherapy results
– Recommendation as to cesarean section
Challenges for ML
• Face recognition
• What is shown on the image?
• Sentiment analysis
• Speech recognition
• …
• These problems are: easy - for a human (solved intuitively) hard - to describe by formal rules
• Formal rules available computer outperform humans
Deep
Learning
28.02.2019
3
Example – spam detection
X1, X2,..., Xd
...
...
...
...
...
...
...
........................... ..................
...Y
Generate features
... ? Model
Model training
(Machine learning)
Spam detection – possible features
X1, X2,..., Xd
...
...
...
...
...
...
...
........................... ..................
...Y
Generate features
... ? Model
Uczenie modelu
(Machine learning)
28.02.2019
4
Spam detection – sample model
X1, X2,..., Xd
...
...
...
...
...
...
...
........................... ..................
...Y
Generacja cech
... ? Model
Uczenie modelu
(Machine learning)
Other models:
Logistic regression
Naive Bayes
Neural net, MLP
SVM
k-NN (MBR)
LDA
Lasso, ElasticNet
…
Challenge: how to generate features?
Brain tumor?
Pulmonary micro-embolism?
28.02.2019
5
Challenge: how to generate features
ImageNet
Large Scale Visual Recognition Challenge
1.2 mln annotated pictures
1000 categories
(ILSVRC-2012)
Deep NN for ImageNet (ILSVRC-2012)
• Krizhevsky, A., Sutskever, I., and Hinton, G. (2012). ImageNet classification with deepconvolutional neural networks. In NIPS’2012
• Top-5 Error 15.3% (second best 26.2%)
• 60 mln parameters in the model
• Training: 6 days on two GPUs (GTX 580 3 GB)
• Input: whole pictures (150 528 inputs, RGB values of pixels)
28.02.2019
6
Deep NN for ImageNet (ILSVRC-2012)
Krizhevsky et al. (2012). ImageNet classification with deep convolutional NN.
Deep NN for ImageNet (ILSVRC-2012)
Krizhevsky et al. (2012). ImageNet classification with deep convolutional NN.
28.02.2019
7
Representation learning
Feature extraction Classification
Representation learning
Standard machine learning
Deep learning / representation learning
Input
Hand-
designed
features
Mapping
features Y
Output
Y
Input FeaturesMapping
features Y
Output
Y
28.02.2019
8
Deep learning growth factors
• Data– Internet, ImageNet project (14 mln pictures)
• Processing power– GPU, parallel computation (Apache Spark)
– cuDNN (nVidia), Caffe, theano
• Improvements in training algorithms– autoencoder
– ReLU – prevents vanishing gradient effect in deep net training
– Dropout – regularization technique (prevents overfitting)
Dataset size
Goodfellow et al. Deep Learning. MIT Press 2016
28.02.2019
9
Model size
• Number of neurons doubles every 2.4 years
Goodfellow et al. Deep Learning. MIT Press 2016
18: Krizhevsky 2012
20: GoogleLeNet 2014
Perspectives
• Deep learning models surpass human performance?– MNIST digit recognition: error < 0.3%
– ImageNet: error ca. 3.6 %
• Need to understand why it works– Meaning of features?
– How to design the architecture of deep networks?
Zeiler, M. D. and Fergus, R. (2014). Visualizing and understanding convolutional networks. In ECCV’14