introduction to deep learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf ·...
TRANSCRIPT
![Page 1: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/1.jpg)
Introduction to Deep LearningConvolutional Neural Networks (1)
Prof. Songhwai OhECE, SNU
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 1
![Page 2: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/2.jpg)
ALEXNET
Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenetclassification with deep convolutional neural networks." NIPS. 2012.
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 2
![Page 3: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/3.jpg)
ImageNet Large‐Scale Visual Recognition Challenge, 2012
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 3
Tasks:• Decide whether a given image contains a particular type of object or not. For example, a
contestant might decide that there are cars in this image but no tigers. • Find a particular object and draw a box around it. For example, a contestant might decide that
there is a screwdriver at a certain position with a width of 50 pixels and a height of 30 pixels.
• 1000 different categories• Over 1 million images• Training set: 456,567 images
Year Winning Error Rate
2010 28.2%
2011 25.8%
2012 16.4% (2nd 25.2%)
2013 11.2%
2014 6.7%
2015 3.57%
Human About 5.1%
ImageNet Large Scale Visual Recognition Challenge. Russakovsky et al. arXiv preprint arXiv:1409.0575. URL: http://arxiv.org/abs/1409.0575v1
![Page 4: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/4.jpg)
ImageNet Dataset
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 4
Source: https://cs.stanford.edu/people/karpathy/cnnembed/cnn_embed_full_1k.jpg
AlexNet on ImageNet
![Page 5: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/5.jpg)
Architecture
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 5
5 convolutional layers3 fully
connected layers
Key ideas: • Rectified Linear Unit (ReLU): an activation function• GPU implementation (2 GPUs)• Local response normalization, Overlapping pooling• Data augmentation, Dropout
Learned 11x11x3 filters
![Page 6: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/6.jpg)
CONVOLUTION
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 6
![Page 7: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/7.jpg)
Convolution
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 7
![Page 8: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/8.jpg)
2D Convolution
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 8
K (3x3 filter) I (7x7 image) Output (5x5)
* =
![Page 9: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/9.jpg)
2D Convolution
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 9
K (3x3 filter) I (7x7 image) Output (5x5)
* =
![Page 10: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/10.jpg)
2D Convolution
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 10
K (3x3 filter) I (7x7 image) Output (5x5)
* =
![Page 11: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/11.jpg)
2D Convolution
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 11
K (3x3 filter) I (7x7 image) Output (5x5)
* =
![Page 12: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/12.jpg)
2D Convolution
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 12
K (3x3 filter) I (7x7 image) Output (5x5)
* =
![Page 13: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/13.jpg)
2D Convolution
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 13
K (3x3 filter) I (7x7 image) Output (5x5)
* =
![Page 14: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/14.jpg)
2D Convolution
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 14
K (3x3 filter) I (7x7 image) Output (5x5)
* =
……
![Page 15: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/15.jpg)
2D Convolution
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 15
K (3x3 filter) I (7x7 image) Output (5x5)
* =
……
![Page 16: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/16.jpg)
RGB Image Convolution
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 16
32x32x3 Image5x5x3 filter
* =
28x28x1 feature
32
32
3
5
5
3
28
28
1
tensor
![Page 17: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/17.jpg)
RGB Image Convolution
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 17
32x32x3 Image 28x28x4 feature
Four 5x5x3 filters
*
![Page 18: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/18.jpg)
Convolutional Neural Network
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18
32
32
3
28
28
4
24
24
10
CONV (4 5x5x3filters),ReLU
CONV (10 5x5x4filters),ReLU
ReLU: Rectified Linear Unit
![Page 19: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/19.jpg)
Stride
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 19
stride 1 3x3 filter
![Page 20: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/20.jpg)
Stride
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 20
stride 1 3x3 filter
![Page 21: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/21.jpg)
Stride
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 21
stride 1 3x3 filter
![Page 22: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/22.jpg)
Stride
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 22
stride 1 3x3 filter
![Page 23: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/23.jpg)
Stride
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 23
stride 1 3x3 filter=> 5x5 output
![Page 24: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/24.jpg)
Stride
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 24
stride 2 3x3 filter
![Page 25: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/25.jpg)
Stride
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 25
stride 2 3x3 filter
![Page 26: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/26.jpg)
Stride
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 26
stride 2 3x3 filter=> 3x3 output
![Page 27: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/27.jpg)
Output Size
• N = input size• F = filter size• S = stride
• Output size = ( N – F ) / S + 1
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 27
![Page 28: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/28.jpg)
Zero Padding
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 28
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0
0
0
0
0
0
0
00
0
0
0
0
0
7x7 inputZero padding with 1 pixel border3x3 filter
=> 7x7 output
![Page 29: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/29.jpg)
Output Size
• N = input size• F = filter size• S = stride• P = padding size
• Output size = ( N + 2P – F ) / S + 1
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 29
![Page 30: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/30.jpg)
1x1 Convolution
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 30
256
256
128
256
256
32
CONV (32 1x1x128filters)
• Dimension reduction• Same output size (H x W)
![Page 31: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/31.jpg)
ReLU Activation
• Preserves properties of linear models– Easy to optimize with gradient descent– Good generalization– Large and consistent gradients
• Overcomes the vanishing gradient problem
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 31
![Page 32: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/32.jpg)
Other Activation Functions
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 32
Sigmoid/Logistic
1
1
tanh (hyperbolic tangent)tanh
Leaky ReLUmax , , 1
maxoutmax , )
![Page 33: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/33.jpg)
Pooling
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 33
1 2 1 0
5 0 0 3
8 0 0 5
0 2 2 0
5 3
8 5max pooling with 2x2 filterwith stride 2
1 2 1 0
5 0 0 3
8 0 0 5
0 2 2 0
2 1
2.5 1.4average pooling with 2x2 filterwith stride 2
• Poolingmakes features invariant to local translations of input• Dimension reduction
![Page 34: Introduction to Deep Learningcpslab.snu.ac.kr/courses/deep-learning-2018/files/03_dl_cnn_p1.pdf · Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 18 32 32 3 28 28 4 24](https://reader036.vdocuments.net/reader036/viewer/2022062508/6053c3aad59d2404876eacd2/html5/thumbnails/34.jpg)
Wrap Up
• Convolutional Neural Networks– Convolution– Activation function: ReLU– Pooling
Prof. Songhwai Oh (ECE, SNU) Introduction to Deep Learning 34