neural network background - github pages
TRANSCRIPT
Neural Network
BackgroundZhifei Zhang and Hairong Qi
3/29/18 2
Algorithms whose performance improve as they are exposed to more data over time
A program that can sense, reason, act, and adapt
Multi-layer neural networks learn from vast amounts of data
• Naive Bayes• kNN• Decision Tree• SVM• K-Means• Dimensionality Reduction,
e.g., FLD, PCA
3/29/18 3For more detailed history of deep learning https://www.import.io/post/history-of-deep-learning/
3/29/18 4
Intuition of Neural Network
https://www.quora.com/What-is-an-intuitive-explanation-for-neural-networks
3/29/18 5
Perceptron
• Weights shows the strength of the particular node.
• A bias value allows you to shift the activation function to the left or right.
• The activation function map the input between the required values, e.g., [0, 1].
Perceptron is usually used to classify the data into two parts. Therefore, it is also known as a Linear Binary Classifier.
Activation Function
WeightedSum
Bias
ActivationFunction
3/29/18 6
Perceptron
WeightedSum
Bias
ActivationFunction
0
1 1
1 1
1 T
n n
n n
wx w
sx wx w- -
æ ö æ öç ÷ ç ÷ç ÷ ç ÷ç ÷ ç ÷´ =ç ÷ ç ÷ç ÷ ç ÷ç ÷ ç ÷è ø è ø
! !
Common activation functions:
ˆ( )F s y=
3/29/18 7
Perceptron Learning Rule
WeightedSum
Bias
ActivationFunction
0
1 1
1 1
1 T
n n
n n
wx w
sx wx w- -
æ ö æ öç ÷ ç ÷ç ÷ ç ÷ç ÷ ç ÷´ =ç ÷ ç ÷ç ÷ ç ÷ç ÷ ç ÷è ø è ø
! !
!" ≔ !" − & '( − ' )"
* = 12 ' − '( .
∆!" = 01023
= 0104(
04(05
05023
Objective:
Partial derivative of * w.r.t. !":
∆!" = '( − ' 6 7′(:) 6 )"
Where < is the learning rate that is a positive value in the range of (0, 1).
Assume the input = = (=>, … , =A), ground truth B, and output of perceptron BC, the weight is updated as follow:
Assume 1, directly pass the error backward
ˆ( )F s y=
3/29/18 8
XOR Problem
3/29/18 9
Multi-layer Perceptron --- Neural Network
Hidden layerInput layer Output layer
Neural Network in more practice
3/29/18 10
Online demo: https://lecture-demo.ira.uka.de/neural-network-demo/
Multi-layer Perceptron --- Solve the XOR Problem
3/29/18 11
BackpropagationHow the multi-layer perceptron (neural network) is learned?
Back propagate the error layer-by-layer ℎ"
#$
%" & = 12 # − #$ +
Output error:,-
%-
3/29/18 12
BackpropagationHow those multi-layer perceptron (neural network) is learned?
Back propagate the error layer-by-layer
∆"# = %& − % ( )′(,-) ( ℎ#
ℎ#
%&
"# 0 = 12 % − %& 3
Output error:45
"5
3/29/18 13
BackpropagationHow those multi-layer perceptron (neural network) is learned?
Back propagate the error layer-by-layer, and
!"
# = 12 ! − !" (
Output error:)*
∆,* = - . /′(23) . )*-
Hidden nodes are independent from each other
- = 5#5ℎ7
= 5#5!"
5!"528
525ℎ7
,*
- = !" − ! . /′(28) . ,7
ℎ7
,7
∆,7 = !" − ! . /′(28) . ℎ7
Back propagated error:
3/29/18 14
Convolutional Neural Network (CNN)
Handle images which need to preserve the special structure
3/29/18 15
Convolutional Neural Network (CNN)
LeNet-5 by Yann LeCun et al., 1998
A demo code in Matlab: https://github.com/ZZUTK/An-Example-of-CNN-on-MNIST-dataset
3/29/18 16
Appearance of CNN is early (back to 1998) but it booms around 2010s, WHY?
Convolutional Neural Network (CNN)
From 2009 From 2010
Large dataset (over 10 million) Powerful computing unit
3/29/18 17
Convolutional Neural Network (CNN)
Appearance of CNN is early (back to 1998) but it booms around 2010s, WHY?
Deep convolutional neural networks with competitive performance to human
AlexNet, 2012
3/29/18 18
Convolutional Neural Network (CNN)
Since 2010s, the deep convolutional neural networks are mostly referred to as deep learning, and it flourishes in computer vision area until today.
3/29/18 19
Convolutional Neural Network (CNN)Convolution and Pooling
Receptive field
Where is the activation function?
3/29/18 20
Convolutional Neural Network (CNN)
Why deep? What does each layer learn?
3/29/18 21
Convolutional Neural Network (CNN)
ImageNet Large Scale Visual Recognition Challenge (ILSVRC) results show deep learning surpassing human levels of accuracy
3/29/18 22
Convolutional Neural Network (CNN)
How to learn a CNN? Again, Backpropagation.
The same as multi-layer perceptronNo parameters, do not need update. Directly pass the error backward.
3/29/18 23
!
Convolutional Neural Network (CNN)
How to learn a CNN? Again, Backpropagation.Multi-layer Perceptron Convolutional Neural Network
" =
∆! = % & ' ∆" = '()*+,- ∗ %
For simplicity, assume /′ 1 = 2 here.
' '
% %
https://github.com/ZZUTK/An-Example-of-CNN-on-MNIST-dataset/blob/master/doc/Derivation_of_Backpropagation_in_CNN.pdf
3 = % & ! 3 = % ∗ "()*+,-
3/29/18 24