applied deep learning 11/03 convolutional neural networks
TRANSCRIPT
![Page 1: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/1.jpg)
SlidecreditfromMarkChang 1
![Page 2: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/2.jpg)
ConvolutionalNeuralNetworks• Weneedacoursetotalkaboutthistopic◦ http://cs231n.stanford.edu/syllabus.html
• However, we only have a lecture
2
![Page 3: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/3.jpg)
Outline• CNN(ConvolutionalNeuralNetworks)Introduction
• EvolutionofCNN
• VisualizingtheFeatures
• CNNasArtist
• SentimentAnalysisbyCNN
3
![Page 4: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/4.jpg)
Outline• CNN(ConvolutionalNeuralNetworks)Introduction
• EvolutionofCNN
• VisualizingtheFeatures
• CNNasArtist
• SentimentAnalysisbyCNN
4
![Page 5: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/5.jpg)
ImageRecognition
http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf
5
![Page 6: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/6.jpg)
ImageRecognition
6
![Page 7: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/7.jpg)
LocalConnectivity
Neuronsconnecttoasmallregion
7
![Page 8: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/8.jpg)
Parameter Sharing• Thesamefeatureindifferentpositions
Neuronssharethesameweights
8
![Page 9: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/9.jpg)
Parameter Sharing• Differentfeaturesinthesameposition
Neuronshavedifferentweights
9
![Page 10: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/10.jpg)
Convolutional Layers
depth
widthwidthdepth
weights weights
height
sharedweight
10
![Page 11: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/11.jpg)
Convolutional Layers
c1
c2
b1
b2
a1
a2
a3
wb1
wb2
b1 =wb1a1+wb2a2
wb1
wb2
b2 =wb1a2+wb2a3
wc1
wc2
c1 =wc1a1+wc2a2
wc2
wc1
c2 =wc1a2+wc2a3
depth=2depth=1
11
![Page 12: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/12.jpg)
Convolutional Layers
c1
b1
b2
a1
a2
d1
b3
a3
c2
d2
depth=2 depth=2
wc1
wc2
wc3
wc4
c1 = a1wc1 + b1wc2
+ a2wc3 + b2wc4
wc1
wc2
wc3
wc4
c2 = a2wc1 + b2wc2
+ a3wc3 + b3wc4
12
![Page 13: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/13.jpg)
Convolutional Layers
c1
b1
b2
a1
a2
d1
b3
a3
c2
d2
c1 = a1wc1 + b1wc2
+ a2wc3 + b2wc4
c2 = a2wc1 + b2wc2
+ a3wc3 + b3wc4
wd1
wd2
wd3
wd4 d1 = a1wd1 + b1wd2
+ a2wd3 + b2wd4
wd1
wd2
wd3
wd4 d2 = a2wd1 + b2wd2
+ a3wd3 + b3wd4
depth=2 depth=2
13
![Page 14: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/14.jpg)
Convolutional LayersA B C
A B C D
14
![Page 15: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/15.jpg)
Hyper-parametersofCNN• Stride • Padding
0 0
Stride=1
Stride=2
Padding=0
Padding=1
15
![Page 16: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/16.jpg)
Example
OutputVolume(3x3x2)
InputVolume(7x7x3)
Stride=2
Padding=1
http://cs231n.github.io/convolutional-networks/
Filter(3x3x3)
16
![Page 17: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/17.jpg)
Convolutional Layers
http://cs231n.github.io/convolutional-networks/
17
![Page 18: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/18.jpg)
Convolutional Layers
http://cs231n.github.io/convolutional-networks/
18
![Page 19: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/19.jpg)
Convolutional Layers
http://cs231n.github.io/convolutional-networks/
19
![Page 20: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/20.jpg)
RelationshipwithConvolution
y[n] =X
k
x[k]w[n� k]
x[n]
w[n]
n
n
y[n]
x[k]
k
k
w[0� k]
n
y[0] = x[�2]w[2] + x[�1]w[1] + x[0]w[0]
20
![Page 21: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/21.jpg)
RelationshipwithConvolution
y[n] =X
k
x[k]w[n� k]
x[n]
w[n]
n
n
y[n]
x[k]
k
k
n
w[1� k]
y[1] = x[�1]w[2] + x[0]w[1] + x[2]w[0]
21
![Page 22: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/22.jpg)
RelationshipwithConvolution
y[n] =X
k
x[k]w[n� k]
x[n]
w[n]
n
n
y[n]
x[k]
k
k
n
y[2] = x[0]w[2] + x[1]w[1] + x[2]w[0]
w[2� k]
22
![Page 23: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/23.jpg)
RelationshipwithConvolution
y[n] =X
k
x[k]w[n� k]
x[n]
w[n]
n
n
y[n]
x[k]
k
k
n
w[4� k]
y[4] = x[2]w[2] + x[3]w[1] + x[4]w[0]
23
![Page 24: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/24.jpg)
Nonlinearity• RectifiedLinear(ReLU)
nout
=
⇢nin
if nin
> 0
0 otherwise
nin n
2
664
14�31
3
775
2
664
1401
3
775ReLU
24
![Page 25: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/25.jpg)
WhyReLU?• Easytotrain
• Avoidgradientvanishingproblem
Sigmoidsaturated
gradient≈0 ReLU notsaturated
25
![Page 26: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/26.jpg)
WhyReLU?• Biologicalreason
strong stimulationReLU
weak stimulation
neuron t
v
strong stimulation
neuron t
v
weak stimulation
26
![Page 27: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/27.jpg)
PoolingLayer
1 3 2 4
5 7 6 8
0 0 3 3
5 5 0 0
4 5
5 3
7 8
5 3
MaximumPooling
AveragePooling
Max(1,3,5,7)=7 Avg(1,3,5,7)=4
nooverlap
noweights
depth=1
Max(0,0,5,5)=5
27
![Page 28: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/28.jpg)
Why“Deep” Learning?
28
![Page 29: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/29.jpg)
VisualPerceptionofHuman
http://www.nature.com/neuro/journal/v8/n8/images/nn0805-975-F1.jpg
29
![Page 30: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/30.jpg)
VisualPerceptionofComputer
ConvolutionalLayer
ConvolutionalLayer Pooling
Layer
PoolingLayer
ReceptiveFieldsReceptiveFields
InputLayer
30
![Page 31: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/31.jpg)
VisualPerceptionofComputer
Input Layer
ConvolutionalLayerwith
ReceptiveFields:
Max-poolingLayerwith
Width=3,Height=3
FilterResponses
FilterResponses
Input Image31
![Page 32: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/32.jpg)
Fully-Connected Layer• Fully-ConnectedLayers:Globalfeatureextraction
• Softmax Layer:Classifier
ConvolutionalLayer Convolutional
LayerPoolingLayer
PoolingLayer
InputLayer
InputImage
Fully-ConnectedLayer Softmax
Layer
5
7
ClassLabel
32
![Page 33: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/33.jpg)
VisualPerceptionofComputer• Alexnet
http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf
http://vision03.csail.mit.edu/cnn_art/data/single_layer.png
33
![Page 34: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/34.jpg)
Training• ForwardPropagation
n2 n1
n1(out)n2(out) n2(in) w21
34
n2(in) = w21n1(out)
n2(out) = g(n2(in)), g is activation function
![Page 35: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/35.jpg)
Training• Updateweights
n2 n1JCost
function:
@J
@w21=
@J
@n2(out)
@n2(out)
@n2(in)
@n2(in)
@w21
w21 w21 � ⌘@J
@w21
) w21 w21 � ⌘@J
@n2(out)
@n2(out)
@n2(in)
@n2(in)
@w21
n2(out) n2(in) w21 n1(out)
35
@J
@w21=
@J
@n2(out)
@n2(out)
@n2(in)
@n2(in)
@w21
w21 w21 � ⌘@J
@w21
) w21 w21 � ⌘@J
@n2(out)
@n2(out)
@n2(in)
@n2(in)
@w21
![Page 36: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/36.jpg)
Training• Updateweights
n2 n1JCost
function:
n2(out) n2(in) w21 n1(out)
w21 w21 � ⌘@J
@n2(out)
@n2(out)
@n2(in)
@n2(in)
@w21
) w21 w21 � ⌘@J
@n2(out)g0(n2(in))n1(out)
n2(out) = g(n2(in)), n2(in) = w21n1(out)
)@n2(out)
@n2(in)= g0(n2(in)),
@n2(in)
@w21= n1(out)
36
![Page 37: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/37.jpg)
Training• Propagatetothepreviouslayer
n2 n1JCost
function:
@J
@n1(in)=
@J
@n2(out)
@n2(out)
@n2(in)
@n2(in)
@n1(out)
@n1(out)
@n1(in)
n2(out) n2(in) n1(out) n1(in)
37
![Page 38: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/38.jpg)
TrainingConvolutional Layers• example:
a3
a2
a1
b2
b1wb1
wb1
wb2
wb2
output input
ConvolutionalLayer
To simplify the notations, in the following slides, we make:
b1 means b1(in), a1 means a1(out), and so on.
38
![Page 39: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/39.jpg)
TrainingConvolutional Layers• Forwardpropagation
a3
a2
a1
b2
b1input
ConvolutionalLayer
b1 = wb1a1 + wb2a2
b2 = wb1a2 + wb2a3
wb1
wb1
wb2
wb2
39
![Page 40: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/40.jpg)
TrainingConvolutional Layers• Updateweights
JCost
function:
a3
a2
a1
b2
b1
@J
@b1
@J
@b2
wb1
wb1
@b1@wb1
@b2@wb1
wb1 wb1 � ⌘(@J
@b1
@b1@wb1
+@J
@b2
@b2@wb1
)
40
![Page 41: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/41.jpg)
TrainingConvolutional Layers• Updateweights
a3
a2
a1
b2
b1 wb1
wb1
b1 = wb1a1 + wb2a2
b2 = wb1a2 + wb2a3
@b1@wb1
= a1
@b2@wb1
= a2
wb1 wb1 � ⌘(@J
@b1a1 +
@J
@b2a2)
@J
@b1
@J
@b2
JCost
function:
41
![Page 42: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/42.jpg)
TrainingConvolutional Layers• Updateweights
a3
a2
a1
b2
b1
wb2 wb2 � ⌘(@J
@b1
@b1@wb2
+@J
@b2
@b2@wb2
)
@b1@wb2
@b2@wb2
wb2
wb2
@J
@b1
@J
@b2
JCost
function:
42
![Page 43: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/43.jpg)
TrainingConvolutional Layers• Updateweights
a3
a2
a1
b2
b1wb2
wb2
@b1@wb2
= a2
@b2@wb2
= a3
wb2 wb2 � ⌘(@J
@b1a2 +
@J
@b2a3)
b1 = wb1a1 + wb2a2
b2 = wb1a2 + wb2a3
@J
@b1
@J
@b2
JCost
function:
43
![Page 44: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/44.jpg)
TrainingConvolutional Layers• Propagatetothepreviouslayer
JCost
function:
a3
a2
a1
b2
b1
@J
@b1
@J
@b2
@b1@a1
@b1@a2
@b2@a2
@b2@a3
@J
@b1
@b1@a1
@J
@b2
@b2@a3
@J
@b1
@b1@a2
+@J
@b2
@b2@a2
44
![Page 45: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/45.jpg)
TrainingConvolutional Layers• Propagatetothepreviouslayer
JCost
function:
a3
a2
a1
b2
b1
@J
@b1
@J
@b2
b1 = wb1a1 + wb2a2
b2 = wb1a2 + wb2a3
@b1@a1
= wb1@b1@a2
= wb2
@b2@a2
= wb1
@b2@a3
= wb2
@J
@b1wb1
@J
@b1wb1 +
@J
@b2wb2
@J
@b2wb2
45
![Page 46: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/46.jpg)
Max-Pooling LayersduringTraining• Poolinglayershavenoweights
• Noneedtoupdateweights
a3
a2
a1
b2
b1 a1 > a2
a2 > a3b2 = max(a2, a3)
b1 = max(a1, a2)
Max-pooling
46
=
⇢a2 if a2 � a3a3 otherwise @b2
@a2=
⇢1 if a2 � a30 otherwise
![Page 47: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/47.jpg)
Max-Pooling LayersduringTraining• Propagatetothepreviouslayer
a3
a2
a1
b2
b1
@J
@b1
@J
@b2
@b1@a1
= 1
a2 > a3
@b2@a2
= 1
a1 > a2
@J
@b1@J
@b2
@b1@a2
= 0
@b2@a3
= 0
J
Costfunction:
47
![Page 48: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/48.jpg)
Max-Pooling LayersduringTraining• ifa1 =a2 ??◦ Choosethenodewithsmallerindex
a3
a2
a1
b2
b1
@J
@b1
@J
@b2
@J
@b1@J
@b2J
Costfunction:
a1 = a2 = a3
48
![Page 49: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/49.jpg)
Avg-Pooling LayersduringTraining• Poolinglayershavenoweights
• Noneedtoupdateweights
a3
a2
a1
b2
b1b1 =
1
2(a1 + a2)
b2 =1
2(a2 + a3)
@b2@a2
=1
2
@b2@a3
=1
2
Avg-pooling
49
![Page 50: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/50.jpg)
Avg-Pooling LayersduringTraining• Propagatetothepreviouslayer
JCost
function:
a3
a2
a1
b2
b1
@J
@b1
@J
@b2
@b1@a1
=@b1@a2
=1
2
@b2@a2
=@b2@a3
=1
2
1
2
@J
@b1
1
2(@J
@b1+
@J
@b2)
1
2
@J
@b2
50
![Page 51: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/51.jpg)
ReLUduringTraining
nout
=
⇢nin
if nin
> 0
0 otherwise
nin n
@nout
@nin
=
⇢1 if n
in
> 1
0 otherwise
51
![Page 52: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/52.jpg)
Training CNN
52
![Page 53: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/53.jpg)
Outline• CNN(ConvolutionalNeuralNetworks)Introduction
• EvolutionofCNN
• VisualizingtheFeatures
• CNNasArtist
• SentimentAnalysisbyCNN
53
![Page 54: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/54.jpg)
LeNet◦ Paper:http://vision.stanford.edu/cs598_spring07/papers/Lecun98.pdf
YannLeCun http://yann.lecun.com/exdb/lenet/
54
![Page 55: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/55.jpg)
ImageNetChallenge• ImageNetLargeScaleVisualRecognitionChallenge◦ http://image-net.org/challenges/LSVRC/
• Dataset:◦ 1000categories◦ Training:1,200,000◦ Validation:50,000◦ Testing:100,000
http://vision.stanford.edu/Datasets/collage_s.png
55
![Page 56: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/56.jpg)
ImageNetChallenge
http://www.qingpingshan.com/uploads/allimg/160818/1J22QI5-0.png
56
![Page 57: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/57.jpg)
AlexNet (2012)• Paper:
http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf
• TheresurgenceofDeepLearning
GeoffreyHintonAlexKrizhevsky57
![Page 58: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/58.jpg)
VGGNet (2014)• Paper:https://arxiv.org/abs/1409.1556
D:VGG16E:VGG19Allfiltersare3x3
58
![Page 59: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/59.jpg)
VGGNet• Morelayers&smallerfilters(3x3)isbetter
• Morenon-linearity,fewerparameters
One5x5filter• Parameters:5x5=25
• Non-linear:1
Two3x3filters• Parameters:3x3x2=18
• Non-linear:2
59
![Page 60: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/60.jpg)
VGG19
depth=643x3convconv1_1conv1_2
maxpool
depth=1283x3convconv2_1conv2_2
maxpooldepth=2563x3convconv3_1conv3_2conv3_3conv3_4
depth=5123x3convconv4_1conv4_2conv4_3conv4_4
depth=5123x3convconv5_1conv5_2conv5_3conv5_4
maxpool maxpool maxpool
size=4096FC1FC2
size=1000softmax
60
![Page 61: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/61.jpg)
GoogLeNet (2014)• Paper:
http://www.cs.unc.edu/~wliu/papers/GoogLeNet.pdf
22layersdeepnetwork
InceptionModule
61
![Page 62: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/62.jpg)
Inception Module• Bestsize?◦ 3x3?5x5?
• Usethemall,andcombine
62
![Page 63: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/63.jpg)
Inception Module
1x1convolution
3x3convolution
5x5convolution
3x3max-pooling
Previouslayer
FilterConcatenate
63
![Page 64: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/64.jpg)
Inception ModulewithDimensionReduction
• Use1x1filterstoreducedimension
64
![Page 65: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/65.jpg)
Inception ModulewithDimensionReduction
Previouslayer
1x1convolution(1x1x256x128)
Reduceddimension
Inputsize1x1x256
128256
Outputsize1x1x128
65
![Page 66: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/66.jpg)
ResNet (2015)• Paper:https://arxiv.org/abs/1512.03385
• ResidualNetworks
• 152layers
66
![Page 67: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/67.jpg)
ResNet• Residuallearning:abuildingblock
Residualfunction
67
![Page 68: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/68.jpg)
Residual LearningwithDimensionReduction
• using1x1filters
68
![Page 69: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/69.jpg)
PretrainedModelDownload• http://www.vlfeat.org/matconvnet/pretrained/◦ Alexnet:◦ http://www.vlfeat.org/matconvnet/models/imagenet-matconvnet-alex.mat
◦ VGG19:◦ http://www.vlfeat.org/matconvnet/models/imagenet-vgg-verydeep-19.mat
◦ GoogLeNet:◦ http://www.vlfeat.org/matconvnet/models/imagenet-googlenet-dag.mat
◦ ResNet◦ http://www.vlfeat.org/matconvnet/models/imagenet-resnet-152-dag.mat
69
![Page 70: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/70.jpg)
UsingPretrainedModel• Lowerlayers:edge,blob,texture(moregeneral)
• Higherlayers:objectpart(morespecific)
http://vision03.csail.mit.edu/cnn_art/data/single_layer.png
70
![Page 71: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/71.jpg)
Transfer Learning• ThePretrained Modelis
trainedonImageNetdataset
• IfyourdataissimilartotheImageNet data
◦ FixallCNNLayers◦ TrainFClayer
Conv layer
FClayer FClayer
LabeleddataLabeleddataLabeleddataLabeleddataLabeleddataImageNetdata
Yourdata
…
Conv layer
Conv layer…
Conv layer
Yourdata
… …
71
![Page 72: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/72.jpg)
Transfer Learning• ThePretrainedModelis
trainedonImageNetdataset
• IfyourdataisfardifferentfromtheImageNet data
◦ FixlowerCNNLayers◦ TrainhigherCNNandFClayers
Conv layer
FClayer FClayer
LabeleddataLabeleddataLabeleddataLabeleddataLabeleddataImageNetdata
Yourdata
…
Conv layer
Conv layer…
Conv layer
Yourdata
… …
72
![Page 73: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/73.jpg)
Tensorflow Transfer LearningExample• https://www.tensorflow.org/versions/r0.11/how_tos/styl
e_guide.html
daisy634
photos
dandelion899
photos
roses642
photos
tulips800
photos
sunflowers700
photoshttp://download.tensorflow.org/example_images/flower_photos.tgz
![Page 74: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/74.jpg)
Tensorflow Transfer LearningExampleFixtheselayers Trainthislayer
![Page 75: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/75.jpg)
Outline• CNN(ConvolutionalNeuralNetworks)Introduction
• EvolutionofCNN
• VisualizingtheFeatures
• CNNasArtist
• SentimentAnalysisbyCNN
75
![Page 76: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/76.jpg)
Visualizing CNN
http://vision03.csail.mit.edu/cnn_art/data/single_layer.png
76
![Page 77: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/77.jpg)
Visualizing CNN
CNN
CNN
flower
randomnoise
filterresponse
filterresponse
77
![Page 78: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/78.jpg)
filterresponse:
GradientAscent• Magnifythefilterresponse
randomnoise:
x f
score: F =X
i,j
fi,j
fi,j
lowerscore
higherscore
x
F
gradient:@F
@x
78
![Page 79: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/79.jpg)
filterresponse:
GradientAscent• Magnifythefilterresponse
randomnoise:
x f
x
gradient:
fi,j
lowerscore
higherscore
F
@F
@x
update x
learningrate
x x+ ⌘@F
@x
79
![Page 80: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/80.jpg)
GradientAscent
80
![Page 81: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/81.jpg)
Different LayersofVisualization
CNN
81
![Page 82: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/82.jpg)
Multiscale ImageGeneration
visualize resize visualize
resize
visualize
82
![Page 83: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/83.jpg)
Multiscale ImageGeneration
83
![Page 84: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/84.jpg)
DeepDream• https://research.googleblog.com/2015/06/inceptionism-
going-deeper-into-neural.html
• Sourcecode:https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/deepdream/deepdream.ipynb
http://download.tensorflow.org/example_images/flower_photos.tgz
84
![Page 85: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/85.jpg)
DeepDream
85
![Page 86: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/86.jpg)
DeepDream
86
![Page 87: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/87.jpg)
Outline• CNN(ConvolutionalNeuralNetworks)Introduction
• EvolutionofCNN
• VisualizingtheFeatures
• CNNasArtist
• SentimentAnalysisbyCNN
87
![Page 88: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/88.jpg)
NeuralArt
• Paper:https://arxiv.org/abs/1508.06576
• Sourcecode:https://github.com/ckmarkoh/neuralart_tensorflow
content
style
artwork
http://www.taipei-101.com.tw/upload/news/201502/2015021711505431705145.JPG
https://github.com/andersbll/neural_artistic_style/blob/master/images/starry_night.jpg?raw=true
88
![Page 89: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/89.jpg)
TheMechanismofPainting
BrainArtist
Scene Style ArtWork
Computer NeuralNetworks
89
![Page 90: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/90.jpg)
Misconception
90
![Page 91: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/91.jpg)
ContentGenerationBrainArtistContent
CanvasMinimize
thedifference
NeuralStimulation
Draw
91
![Page 92: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/92.jpg)
ContentGeneration
92
FilterResponsesVGG19
Updatethecolorofthepixels
Content
Canvas
Result
Width*HeightDepth
Minimizethe
difference
![Page 93: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/93.jpg)
ContentGeneration
Layerl’sFilterlResponses:
Layerl’sFilterResponses:
InputPhoto: Input
Canvas:
Width*Height (j)
Depth(i)
Width*Height (j)
Depth(i)
93
![Page 94: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/94.jpg)
ContentGeneration• BackwardPropagation
Layerl’sFilterlResponses:
InputCanvas:
VGG19
UpdateCanvas
LearningRate
94
![Page 95: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/95.jpg)
ContentGeneration
95
![Page 96: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/96.jpg)
ContentGeneration
VGG19
conv1_2 conv2_2 conv3_4 conv4_4 conv5_2conv5_1
96
![Page 97: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/97.jpg)
StyleGeneration
VGG19Artwork
G
G
FilterResponses GramMatrix
Width*Height
Depth
Depth
Depth
Position-dependent
Position-independent
97
![Page 98: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/98.jpg)
StyleGeneration
1. .5
.5
.5
1.
1. .5 .25 1.
.5 .25 .5
.25 .25
1. .5 1.
Width*Height
Depth
k1 k2
k1
k2
Depth
Depth
Layerl’sFilterResponsesGramMatrix
G
98
![Page 99: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/99.jpg)
StyleGeneration
Layerl’sFilterResponses
Layerl’sGramMatrix
Layerl’sGramMatrix
InputArtwork:
InputCanvas:
99
![Page 100: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/100.jpg)
StyleGeneration
VGG19Filter
ResponsesGramMatrix
Minimizethe
difference
G
G
Style
Canvas
UpdatethecolorofthepixelsResult
100
![Page 101: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/101.jpg)
StyleGeneration
101
![Page 102: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/102.jpg)
StyleGenerationVGG19
Conv1_1 Conv1_1Conv2_1
Conv1_1Conv2_1Conv3_1
Conv1_1Conv2_1Conv3_1Conv4_1
Conv1_1Conv2_1Conv3_1Conv4_1Conv5_1
102
![Page 103: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/103.jpg)
ArtworkGenerationFilterResponsesVGG19
GramMatrix
103
![Page 104: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/104.jpg)
ArtworkGeneration
VGG19 VGG19
Conv1_1Conv2_1Conv3_1Conv4_1Conv5_1
Conv4_2
104
![Page 105: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/105.jpg)
ArtworkGeneration
105
![Page 106: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/106.jpg)
Contentv.s.Style
0.15 0.05
0.02 0.007
106
![Page 107: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/107.jpg)
NeuralDoodle• Paper:https://arxiv.org/abs/1603.01768
• Sourcecode:https://github.com/alexjc/neural-doodle
style
content resultsemanticmaps
107
![Page 108: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/108.jpg)
NeuralDoodle• Imageanalogy
108
![Page 109: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/109.jpg)
NeuralDoodle• Imageanalogy
恐怖連結,慎入!https://raw.githubusercontent.com/awentzonline/image-analogies/master/examples/images/trump-image-analogy.jpg
109
![Page 110: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/110.jpg)
Real-timeTextureSynthesis• Paper:https://arxiv.org/pdf/1604.04382v1.pdf◦ GAN:https://arxiv.org/pdf/1406.2661v1.pdf◦ VAE:https://arxiv.org/pdf/1312.6114v10.pdf
• SourceCode:https://github.com/chuanli11/MGANs
110
![Page 111: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/111.jpg)
Outline• CNN(ConvolutionalNeuralNetworks)Introduction
• EvolutionofCNN
• VisualizingtheFeatures
• CNNasArtist
• SentimentAnalysisbyCNN
111
![Page 112: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/112.jpg)
AConvolutional NeuralNetworkforModellingSentences
• Paper:https://arxiv.org/abs/1404.2188
• Sourcecode:https://github.com/FredericGodin/DynamicCNN
112
![Page 113: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/113.jpg)
Drawbacks ofRecursiveNeuralNetworks(RvNN)
• Needhuman-labeledsyntaxtreeduringtraining
This is a dog
TrainRvNN
Wordvector This is a dog
RvNN
RvNN
RvNN
113
![Page 114: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/114.jpg)
Drawbacks ofRecursiveNeuralNetworks(RvNN)
• Ambiguityinnaturallanguage
http://3rd.mafengwo.cn/travels/info_weibo.php?id=2861280
114
http://www.appledaily.com.tw/realtimenews/article/new/20151006/705309/
![Page 115: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/115.jpg)
Element-wise 1Doperationsonwordvectors
• 1DConvolutionor1DPooling
This is a
operationoperation
This is a
Representedby
115
![Page 116: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/116.jpg)
FromRvNN toCNN• RvNN • CNN
This is a dog
conv3
conv2
conv1conv1conv1
conv2
SameRvNN
Differentconv layers
This is a dog
RvNN
RvNN
RvNN
116
![Page 117: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/117.jpg)
CNNwithMax-Pooling Layers• Similartosyntaxtree
• Buthuman-labeledsyntaxtreeisnotneeded
This is a dog
conv2
pool1
conv1conv1conv1
pool1
This is a dog
conv2
pool1
conv1conv1
pool1
MaxPooling
117
![Page 118: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/118.jpg)
SentimentAnalysisbyCNN• Use softmax layer to classify the sentiments
positive
This movie is awesome
conv2
pool1
conv1conv1conv1
pool1
softmax
negative
This movie is awful
conv2
pool1
conv1conv1conv1
pool1
softmax
118
![Page 119: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/119.jpg)
SentimentAnalysisbyCNN• Buildthe “correct syntax tree” by training
negative
This movie is awesome
conv2
pool1
conv1conv1conv1
pool1
softmax
negative
This movie is awesome
conv2
pool1
conv1conv1conv1
pool1
softmax
Backwardpropagation
error
119
![Page 120: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/120.jpg)
SentimentAnalysisbyCNN• Buildthe “correct syntax tree” by training
negative
This movie is awesome
conv2
pool1
conv1conv1conv1
pool1
softmax
positive
This movie is awesome
conv2
pool1
conv1conv1conv1
pool1
softmax
Updatetheweights
120
![Page 121: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/121.jpg)
MultipleFilters• RicherfeaturesthanRNN
This is
filter11 Filter13filter12
a
filter11 Filter13filter12
filter21 Filter23filter22
121
![Page 122: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/122.jpg)
Sentencecan’t be easily resized• Imagecanbeeasilyresized • Sentence can’t be easily
resized
全台灣最高樓在台北
resize
resize
全台灣最高的高樓在台北市
全台灣最高樓在台北市
台灣最高樓在台北
122
![Page 123: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/123.jpg)
Various InputSize• Convolutionallayersandpoolinglayers◦ canhandleinputwithvarioussize
This is a dog
pool1
conv1conv1conv1
pool1
the dog run
pool1
conv1conv1
123
![Page 124: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/124.jpg)
Various InputSize• Fully-connectedlayerandsoftmax layer◦ needfixed-sizeinput
The dog run
fc
softmax
This is a
fc
softmax
dog
124
![Page 125: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/125.jpg)
k-maxPooling• choosethek-maxvalues
• preservetheorderofinputvalues
• variable-sizeinput,fixed-sizeoutput
3-maxpooling
13 4 1 7 812 5 21 15 7 4 9
3-maxpooling
12 21 15 13 7 8
125
![Page 126: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/126.jpg)
WideConvolution• Ensuresthatallweightsreachtheentiresentence
conv conv conv conv conv convconv conv
Wide convolution Narrow convolution
126
![Page 127: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/127.jpg)
Dynamick-maxPooling
wideconvolution&k-maxpooling
wideconvolution&k-maxpooling
kl
ktop
L
s
ktop
and L are constants
l : index of current layer
kl
: k of current layer
ktop
: k of top layer
L : total number of layers
s : length of input sentence
k
l
= max(ktop
, dL� l
L
se)
127
![Page 128: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/128.jpg)
Dynamick-maxPooling
s = 10
L = 2
k1 = max(3, d2� 1
2⇥ 10e) = 5
ktop
= 3
k
l
= max(ktop
, dL� l
L
se)
conv &pooling
conv &pooling
128
![Page 129: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/129.jpg)
Dynamick-maxPooling
conv &pooling
conv &pooling
L = 2
ktop
= 3
k
l
= max(ktop
, dL� l
L
se)
k1 = max(3, d2� 1
2⇥ 14e) = 7
s = 14
129
![Page 130: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/130.jpg)
Dynamick-maxPooling
conv &pooling
conv &pooling
L = 2
ktop
= 3
k
l
= max(ktop
, dL� l
L
se)
s = 8
k1 = max(3, d2� 1
2⇥ 8e) = 4
130
![Page 131: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/131.jpg)
Dynamick-maxPooling
Wideconvolution&Dynamick-maxpooling
131
![Page 132: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/132.jpg)
Convolutional NeuralNetworksforSentenceClassification
• Paper:http://www.aclweb.org/anthology/D14-1181
• Sourcee code:https://github.com/yoonkim/CNN_sentence
132
![Page 133: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/133.jpg)
Static&Non-StaticChannel• Pretrained byword2vec
• Static:fixthevaluesduringtraining
• Non-Static:updatethevaluesduringtraining
133
![Page 134: Applied Deep Learning 11/03 Convolutional Neural Networks](https://reader033.vdocuments.net/reader033/viewer/2022051709/587071671a28ab48378b7a1f/html5/thumbnails/134.jpg)
AbouttheLecturer
MarkChang
• Email:ckmarkoh at gmail dot com• Blog: http://cpmarkchang.logdown.com• Github:https://github.com/ckmarkoh• Slideshare:http://www.slideshare.net/ckmarkohchang• Youtube:https://www.youtube.com/channel/UCckNPGDL21aznRhl3EijRQw
134
HTCResearch&HealthcareDeepLearningAlgorithmsResearchEngineer