![Page 1: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/1.jpg)
1January 5, 2017
Deep Learning
Er. Shiva K. Shrestha, ME Computer, NCIT
![Page 2: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/2.jpg)
2
Slide Credito Jeff Dean, Google, Large Scale Deep Learningo Andrew Ng, Deep Learning o Aditya Khosla & Joseph Lim, Visual Recognition through ML Competition
January 5, 2017
![Page 3: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/3.jpg)
3
Structure◦ General Questions of the World ◦ What is Deep Learning?◦ Why Deep Learning?◦ Deep Neural Network Architectures◦ Deep Learning Applications◦ Conclusions, Recommendations
January 5, 2017
![Page 4: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/4.jpg)
4
How Can We Build More Intelligent Computer Systems?
According to Jeff Dean, Google:o Need to perceive and understand the worldo Basic speech and vision capabilitieso Language understandingo User behavior predictiono …
January 5, 2017
![Page 5: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/5.jpg)
5
How can we do this? According to Jeff Dean, Google:o Cannot write algorithms for each task we want to accomplish separately.o Need to write general algorithms that learn from observationso Can we build systems that:o Generate understanding from raw datao Solve difficult problems to improve productso Minimize software engineering effort
January 5, 2017
![Page 6: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/6.jpg)
6
Plenty of Datao Text: trillions of words of English + other languageso Visual: billions of images and videos o Audio: thousands of hours of speech per dayo User Activity: queries, result page clicks, map requests, etc.o Knowledge Graph: billions of labelled relation tripleso …
January 5, 2017
![Page 7: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/7.jpg)
7
Image Models
January 5, 2017
![Page 8: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/8.jpg)
8
What are these numbers?
January 5, 2017
![Page 9: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/9.jpg)
9
What are all these words?
January 5, 2017
![Page 10: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/10.jpg)
10
How about these words?
January 5, 2017
![Page 11: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/11.jpg)
11
Textual Understanding “This movie should have NEVER been made. From the poorly done animation, to the beyond bad acting. I am not sure at what point the people behind this movie said "Ok, looks good! Lets do it!" I was in awe of how truly horrid this movie was.”
January 5, 2017
![Page 12: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/12.jpg)
12
General Machine Learning Approacheso Learning by labeled example: Supervised Learningo e.g. An email spam detector o amazingly effective if you have lots of examples
o Discovering patterns: Unsupervised Learningo e.g. data clusteringo difficult in practice, but useful if you lack labeled examples
o Feedback right/wrong: Reinforcement Learningo e.g. learning to play chess by winning or losingo works well in some domains, becoming more important
January 5, 2017
![Page 13: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/13.jpg)
13
Machine Learningo For many of these problems, we have lots of datao gives computers the ability to learn without being explicitly programmed
January 5, 2017
Approacheso Decision tree learningo Association rule learningo Artificial neural networkso Deep learningo Inductive logic programmingo Support vector machineso Clusteringo Bayesian networks
Approaches …o Reinforcement learningo Representation learningo Similarity and metric learningo Sparse dictionary learningo Genetic algorithmso Rule-based machine learningo Learning classifier systems
![Page 14: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/14.jpg)
14
Typical Goal of Machine LearningLabel: “Motorcycle”Suggest tagsImage search…
Speech recognitionMusic classificationSpeaker identification…
Web searchAnti-spamMachine translation…
text
audio
images/video
I/p O/pML
ML
ML
January 5, 2017
![Page 15: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/15.jpg)
15
Basic Idea of Deep Learning Is there some way to extract meaningful features from data even without knowing the task to be performed?
Then, throw in some hierarchical ‘stuff’ to make it ‘deep’
January 5, 2017
![Page 16: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/16.jpg)
16
What is Deep Learning?o The modern reincarnation of ANNs from the 1980s and 90s.o A collection of simple trainable mathematical units, which collaborate to compute a complicated function.oCompatible with (3) General ML Approaches
January 5, 2017
![Page 17: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/17.jpg)
17
What is Deep Learning? (2)o Loosely inspired by what (little) we know about the biological brain.o AKA:o Deep Structure Learningo Hierarchical Learningo Deep M/c Learning
January 5, 2017
![Page 18: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/18.jpg)
18
Deep Learning DefinitionsDeep learning is characterized as a class of machine learning algorithms that o use a cascade of many layers of nonlinear processing units for feature extraction and transformation. o are based on the learning of multiple levels of features or representations of the data. o are part of the broader machine learning field of learning representations of data.o learn multiple levels of representations that correspond to different levels of abstraction;
January 5, 2017
![Page 19: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/19.jpg)
19
DL - Why is this hard?You see this:
But the camera sees this:
January 5, 2017
![Page 20: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/20.jpg)
20
Pixel-based Representation
Input
Raw image
Motorbikes“Non”-Motorbikes
Learningalgorithm
pixel 1
pixe
l 2pixel 1
pixel 2
January 5, 2017
![Page 21: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/21.jpg)
21
Pixel-based Representation (2)
InputMotorbikes“Non”-Motorbikes
Learningalgorithm
pixel 1
pixe
l 2pixel 1
pixel 2
Raw image
January 5, 2017
![Page 22: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/22.jpg)
22
Pixel-based Representation (2)
InputMotorbikes“Non”-Motorbikes
Learningalgorithm
pixel 1
pixe
l 2pixel 1
pixel 2
Raw image
January 5, 2017
![Page 23: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/23.jpg)
23
What We Want
InputMotorbikes“Non”-Motorbikes
Learningalgorithm
pixel 1
pixe
l 2Feature
representation
handlebars
wheelE.g., Does it have Handlebars? Wheels?
Handlebars
Whe
els
Raw image Features
January 5, 2017
![Page 24: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/24.jpg)
24
Some Feature Representations
SIFT Spin image
HoG RIFT
Textons GLOHJanuary 5, 2017
![Page 25: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/25.jpg)
25
Some Feature Representations (2)
SIFT Spin image
HoG RIFT
Textons GLOH
Coming up with features is often difficult, time-consuming, and requires expert knowledge.
January 5, 2017
![Page 26: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/26.jpg)
26
The Brain: Potential Motivation for Deep Learning
[Roe et al., 1992]
Auditory Cortex learns to see!
Auditory Cortex
January 5, 2017
![Page 27: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/27.jpg)
27
The Brain adapts!
[BrainPort; Welsh & Blasch, 1997; Nagel et al., 2005; Constantine-Paton & Law, 2009]
Seeing with your Tongue Human Echolocation (Sonar)
Haptic belt: Direction Sense Implanting a 3rd Eye
January 5, 2017
![Page 28: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/28.jpg)
28
Feature Learning Problem Given a 14x14 image patch x, can represent it using 196 real numbers.
Problem: Can we find a learn a better feature vector to represent this?
255989387899148…
January 5, 2017
![Page 29: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/29.jpg)
29
Why Deep Learning?Method AccuracyHessian + ESURF [Williems et al 2008] 38%Harris3D + HOG/HOF [Laptev et al 2003, 2004]
45%
Cuboids + HOG/HOF [Dollar et al 2005, Laptev 2004]
46%
Hessian + HOG/HOF [Laptev 2004, Williems et al 2008]
46%
Dense + HOG / HOF [Laptev 2004] 47%Cuboids + HOG3D [Klaser 2008, Dollar et al 2005]
46%
Unsupervised Feature Learning (DL) 52%
[Le, Zhou & Ng, 2011]
Task: Video Activity Recognition
January 5, 2017
![Page 30: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/30.jpg)
30
Deep Neural Network Architectureso GMDH: 1st DLN of 1965o Convolutional NNo Neural history compressoro Recursive NNo Long short-term memory (LSTM)o Deep belief networks (DBN)o Convolutional deep belief networkso Large memory storage & retrieval NNo Deep Boltzmann machines
o Stacked (de-noising) auto-encoderso Deep stacking networkso Tensor deep stacking networkso Spike-and-slab RBMso Compound hierarchical-deep modelso Deep coding networkso Deep Q-networkso Networks with separate memory structures
January 5, 2017
![Page 31: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/31.jpg)
31
Neural Network (NN)x1
x2
x3
+1 +1
Layer 1 Layer 2
Layer 4+1
Layer 34 layer network with 2 output units:
January 5, 2017
![Page 32: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/32.jpg)
Unsupervised Feature Learning with a NN
x4
x5
x6
+1
x1
x2
x3
+1
a1
a2
a3
+1
b1
b2
b3
+1
c1
c2
c3
New representation for input.
Use [c1, c3, c3] as representation to feed to learning algorithm.
![Page 33: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/33.jpg)
Deep Belief NetworkDBN is algorithm for learning a feature hierarchy.
Building Block: 2-layer graphical model (Restricted Boltzmann Machine).
Can then learn additional layers one at a time. Schematic overview of
a deep belief net.
![Page 34: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/34.jpg)
34
Deep Belief Network (2)
Input [x1, x2, x3, x4]
Layer 2. [a1, a2, a3]
Layer 3. [b1, b2, b3]
Similar to a sparse auto-encoder in many ways. Stack RBMs on top of each other to get DBN.
Train with approximate maximum likelihood (often with sparsity constraint on ai’s):
January 5, 2017
![Page 35: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/35.jpg)
35
Convolutional DBN for AudioS
pect
rogr
am
Detection units
Max pooling unit
January 5, 2017
![Page 36: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/36.jpg)
36
Convolutional DBN for Audio (2)S
pect
rogr
am
January 5, 2017
![Page 37: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/37.jpg)
37
Convolutional DBN for Images
January 5, 2017
![Page 38: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/38.jpg)
38
Going Deep
Pixels
Object Models
[Honglak Lee]
Training Set: Alignedimages of faces.
January 5, 2017
Edges
Object Parts(combination of edges)
![Page 39: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/39.jpg)
39
Applicationso Computer Vision: Object Detection & Recognitiono Speech Recognitiono Speaker Identificationo Web Searcheso Text Classification - Sentiment Analysis
o Translationso Miscellaneouso Fine-grained Classificationo Generalizationo Generating Image Captions from
Pixelso …
January 5, 2017
![Page 40: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/40.jpg)
40
Applications (2)
January 5, 2017
![Page 41: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/41.jpg)
41
Speech Recognition on Android
January 5, 2017
![Page 42: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/42.jpg)
42
Impact on Speech Recognition
January 5, 2017
![Page 43: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/43.jpg)
43
Text Classifications
January 5, 2017
Results for IMDB Sentiment Classification (long paragraphs)
![Page 44: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/44.jpg)
44
Translationo Google Translate:o As Reuters noted for the first time in July, the seating configuration is exactly
what fuels the battle between the latest devices.
o Neural LSTM Model:o As Reuters reported for the first time in July, the configuration of seats is
exactly what drives the battle between the latest aircraft.
o Human Translation:o As Reuters first reported in July, seat layout is exactly what drives the battle
between the latest jets.
January 5, 2017
![Page 45: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/45.jpg)
45
Good Fine-grained Classification
January 5, 2017
![Page 46: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/46.jpg)
46
Good Generalization
January 5, 2017
![Page 47: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/47.jpg)
47
Sensible Errors
January 5, 2017
![Page 48: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/48.jpg)
48
Generating Image Captions from Pixels
January 5, 2017
Work by Oriol Vinyals et al.
![Page 49: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/49.jpg)
49
Generating Image Captions from Pixels(2)
January 5, 2017
![Page 50: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/50.jpg)
50
Generating Image Captions from Pixels(3)
January 5, 2017
![Page 51: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/51.jpg)
51
Generating Image Captions from Pixels(4)
January 5, 2017
![Page 52: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/52.jpg)
52
ConclusionDeep Neural Networks are very effective for wide range of taskso By using parallelism, we can quickly train very large and effective deep neural models on very large datasetso Automatically build high-level representations to solve desired taskso By using embedding, can work with sparse datao Effective in many domains: speech, vision, language modeling, user prediction, language understanding, translation, advertising, …
January 5, 2017
An important tool in building Intelligent Systems !
![Page 53: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/53.jpg)
53
Thank You !
Q/A ?
January 5, 2017
![Page 54: Deep Learning for Artificial Intelligence (AI)](https://reader035.vdocuments.net/reader035/viewer/2022062412/587db9a81a28ab1b498b45ab/html5/thumbnails/54.jpg)
54
Recommendationso Le, Ranzato, Monga, Devin, Chen, Corrado, Dean, & Ng. Building High-Level Features Using Large Scale Unsupervised Learning, ICML 2012.
o Dean, Corrado, et al. , Large Scale Distributed Deep Networks, NIPS 2012.
o Mikolov, Chen, Corrado and Dean. Efficient Estimation of Word Representations in Vector Space, http://arxiv.org/abs/1301.3781.
o Distributed Representations of Sentences and Documents, by Quoc Le and Tomas Mikolov, ICML 2014, http://arxiv.org/abs/1405.4053
o Vanhoucke, Devin and Heigold. Deep Neural Networks for Acoustic Modeling, ICASSP 2013.
o Sequence to Sequence Learning with Neural Networks, Ilya Sutskever, Oriol Vinyals, and Quoc Le. http://arxiv.org/abs/1409.3215. To appear in NIPS, 2014.
o http://research.google.com/papers
o http://research.google.com/people/jeff
January 5, 2017