from conventional machine learning to deep learning and beyond.pptx
TRANSCRIPT
From Conventional Machine Learning to Deep Learning
and Beyond
by GB Chun Hao, Chang
Email: [email protected]
Who am I ?I graduated from IDEA lab 4 months ago with a Master degree.
As a junior Machine Learning Engineer, my job is to understand the latest Deep
Learning research on Computer Vision and implement/modify them for the
Company’s needs
The following contents are my extremely limited experiences(less than 4 months) and summary.
Please correct me if you found any error ;)
Overview1. Brief introduction of Deep Learning
2. Two Popular Networks: CNN and RNN
3. Conventional Learning Model vs Deep Learning
4. The Strength of DNN
5. The Flaws of DNN
6. Beyond Regular Machine Learning Tasks
7. Appendix: Hardwares for DNN
What is Deep LearningDeep Learning refers to Deep Neural Network.
It belongs to Neural Networks family, a branch of Machine Learning
Two popular branches in DNN
Convolutional Neural NetworkFor Computer Vision Tasks
Recurrent Neural NetworkFor Sequential Problem
Convolutional Neural NetworkThe most popular Neural Network in Computer VisionCNN is able to capture different levels of feature representation
The following pictures are generated purely by the weights of learned models
Image of the Bird Saxophone
CNN in Typical Computer Vision Tasks
CNN outperforms hand-crafted methods on Image Classification and Object Detection tasks
Recurrent Neural NetworkRecurrent Neural Network is able to memorize and recall the memory.
RNN are suitable for sequential data:
RNN is good at Time Series and Natural Language Processing. RNN is Turing Complete
RNN Example - Word Embeddings
Mapping words into continuous vectors (Some word embedding techniques does not need DNN at all. For example Word2Vec)
RNN Example - Language Translation
Shallow Neural Network
Shallow Neural Network is similar to most of the conventional supervised models
Pros:1.Easy to train and test2.Able to approach any continuous function
Cons:1.Performance depends on well-designed features 2.Difficult to generalize the prediction
Deep Neural NetworkPros:1.Automatically learn the High-level features representation 2.Modularity: DNN can be composed like LEGO bricks3.Able to do Transfer Learning
Cons:1.Requires tons of data for training2.Expensive computation power for training and testing (no CV)
What makes DNN so popular?
It has the three advantages:
1. Self-learned high-level features representations2. Modularity3. Transfer Learning
Why are these so important?
Self-Learned High-Level Feature Representation (⅓)It means that you no longer need hand-crafted "Feature Engineering"
In the past, it takes several experts years to do the Feature Engineering on a specific task
Self-Learned High-Level Feature Representation(⅔)DNN simplify the pipelines of Machine Learning tasks (by removing Feature Engineering), therefore researcher can use similar tools to solve different kinds of tasks.
You can address different kinds of problems with similar DNN models!
Self-Learned High-Level Feature Representation(3/3)
Example: The following LRCN model (CNN + RNN) is able to handle three different kinds of problem: Activity Recognition, Image Description and Video Description
Modularity (¼)
DNN models can be composed just like building LEGO buildings
Modularity (2/4) : For example, we want a DNN model for Object Detection, but we only have a Image Classification DNN model(VGG-16).
We can either construct a complicated pipeline with conventional detectors or compose a bigger DNN model with other DNN modules.
VGG-16
For Image Classification only
Object Detection Task
Modification
Modularity (¾) : From Image Classification to Object detection VGG-16
VGG-16
VGG-16 + ROI Polling Layer
VGG-16 + Region Proposal Network+ ROI Polling Layer
SVM
Selective Search
Selective Search
R-CNN
Fast R-CNN
Faster R-CNN
For Image Classification only
Complicated and slow
End-to-End and Fast
Modularity (4/4) It’s also simple to replace one component in DNN
For example, ResNet-52 outperforms VGG-16 on Image Classification, we can replace the VGG-16 with ResNet-52 in Faster R-CNN in order to improve the overall performance in Object Detection
VGG-16 + Region Proposal Network+ ROI Polling Layer
Faster R-CNN
ResNet-52 + Region Proposal Network+ ROI Polling Layer
Faster R-CNN
VGG-16
ResNet-52Replace
The learned knowledge of one task can be used in another task.
An apple detection model or data can help you to do orange detection!
Transfer Learning(¼)
Transfer Learning(2/4)
Conventional Machine Learning models are difficult to apply Transfer Learning. The model depends on domain specific features and is sensitive to the data distribution. Furthermore, different kinds of model(SVM, decision trees) can not share their weights easily
Difficult
Transfer Learning(¾)Use a pre-trained model and fine-tune it to adapt new domain tasks is a
common practice in DNN.
Example:
VGG-16
ImageNet Model(Classification)
Style Recognition
Pedestrian Detection
Fine-Tuning
Fine-Tuning
Transfer Learning(4/4): Wisdom of the DNN Models
Many researchers not only publish their thesis and source code. The Trained Models are also shared on-line.
You can download the Pre-trained Models, combine them or fine-tune them.
Caffe Model Zoo
DNN Minimize the gaps between areas
Text Data
Voice and other Signals
Image and Videos
Reinforcement Learning
DNN can help you get into other research areas easier!
Are DNN models really that magical?
Hehehe, NO
DNN models are nice tools but not that magical.
Let me give you some clues and my experience
Flaws of DNN (⅓): MS-CNN Compromise on the Data Distribution
Do you find anything conflicting with the advantages of DNN models?
Flaws of DNN (⅔): TSN Compromise on Self-Learning Feature
Do you find anything conflicting with the advantages of DNN models?
Flaws of DNN(3/3): Adversarial Examples
Maybe it’s not a big deal, but Adversarial Examples somehow breaks the belief of “DNN models are similar to human brains”
DNN are still wonderful tools to use
The above problems are not serious and have solutions.
Furthermore, there are tons of researchers are working on better and smarter DNN models.
Beyond Regular Machine Learning TasksExcept for the regular classification and regression tasks. DNN models more than that.
Once you can define and evaluate the task mathematically, you can apply DNN models
Think DNN models as
“Trainable Program that does everything”
CNN model - Google Deep Dream
RNN - Generated Kanji
RNN - Composing Music
Combine CNN with RNN
Image + Time Series = Videos and varieties of applications!
CNN+RNN: Image CaptioningDemonstration Attention Mechanism to sequentialize an image by focusing on different spots
RNN module decide which part of image to focus
Deep Q-Networks (Reinforcement Learning)
DNN that implements Q-Learning algorithms
Neural Turing Machine
A Turing machine that is differentiable and can be trained on gradient descent
A DNN model that able to control external storage
Memory Network End-to-End
Zero-Shot & One-Shot Learning
Models that depends on small dataset and able to detects never-seen classesResearcher are using DNN to do Zero-Shot & One-Shot Learning tasks
Any Question? :)
Hardwares: Nvidia - Biggest Player in DNN
For normal users, the only choice to run DNN is Nvidia video cards
Nvidia will be focusing more on Intelligence Computations and Self Driving Cars in future
Hardwares: Google Tensor Processing Unit
This Google homemade TPU are mysterious and not accessible for outsiders.
Hardwares: Movidius Fathom
A USB stick for DNN computations from Movidus
Movidius was purchased Intel recently
Hardware: Physical Neural Network
Started from 1960, PNN is hardware electrically similar to neural networks.
A BrainChip From NeuromorThings