neural networks and deep learning - cse - iit kanpur€¦ · neural networks and deep learning ....
TRANSCRIPT
![Page 1: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/1.jpg)
Neural Networks and
Deep Learning
![Page 2: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/2.jpg)
Example Learning Problem
![Page 3: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/3.jpg)
Example Learning Problem
Celebrity Faces in the Wild
![Page 4: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/4.jpg)
Machine Learning Pipeline
Features • critical for accuracy • traditionally hand-crafted
Instead of designing features, try to design feature
detectors
Raw data Feature
extract.
Feature
comput-
ation
Inference:
prediction,
recognition
![Page 5: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/5.jpg)
Machine Learning Pipeline
Raw data Feature
discovery.
Feature
compos-
ition
Inference:
prediction,
recognition
Low level features Mid-level features Face-like features
![Page 6: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/6.jpg)
Logistic Regression unit
h(x; w,b)
= 1/ e-(wTx + b)
w1 w2
w3
b
Objective:
determine parameters w,b
![Page 7: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/7.jpg)
Training a neural network
Given training set (x1, t1), (x2, t2), (x3, t3 ), ….
minimize error = h (xi; w,b) – ti by adjusting parameters (w,b)
over all nodes
Use gradient descent : “Backpropagation”: local optima
![Page 8: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/8.jpg)
MLP with Back-propagation
input vector
hidden
layers
outputs
Back-propagate
error to get gradient
for parameter
updating
Compare outputs with
correct answer to get
error vector
[Hinton 09 ] DBN tutorial
![Page 9: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/9.jpg)
Why “Deep”?
• Brains are very deep
• Humans organize their ideas hierarchically, through composition of simpler ideas
• Insufficiently deep architectures can be exponentially inefficient
• functions computable with a polynomial-size circuit of depth k may require exponential size at depth k-1 [Hastad 86].
• Deep architectures help share features
![Page 10: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/10.jpg)
Why learn features??
• In the brain, very few filters are hard-coded
• irreversible damage produced in kittens by early visual deprivation [Hubel Wiesel 63]
• Avoids different feature extraction schemes for different kinds of input data
• Hypothesis :
Good Reconstruction Good Recognition
![Page 11: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/11.jpg)
Drawbacks of Back-propagation
• Purely discriminative
Get all the information from the labels
And the labels don’t give so much of information
Need a substantial amount of labeled data
• Gradient descent with random
initialization leads to poor local minima
![Page 12: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/12.jpg)
Deep Belief Networks
• Pre-train network from input-data
alone (generative step)
• Use weights of pre-trained network
as the initial point for traditional back-
propagation
• Leads to quicker convergence
• Pre-training is fast; fine-tuning can be
slow
![Page 13: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/13.jpg)
Deep Autoencoder
2-l
aye
r A
uto
- E
ncoders
(
coars
e)
Auto
-Encoders
sta
ck
![Page 14: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/14.jpg)
Pre-Training: Maximum likelihood
learning
jiji
ij
hvhvw
vp 0)(log
Start with a training vector on the visible units.
Then alternate between updating all the hidden units in
parallel and updating all the visible units in parallel.
0 jihv jihv
i
j
i
j
i
j
i
j
t = 0 t = 1 t = 2 t = infinity
[Hinton 09 ] DBN tutorial
![Page 15: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/15.jpg)
)( 10 jijiij hvhvw
For each training vector
Update all hidden units in
parallel
Update the all the visible units
in parallel to get a
“reconstruction”.
Update the hidden units again.
0 jihv 1 jihv
i
j
i
j
t = 0 t = 1 reconstruction data
Pre-Training: Maximum likelihood
learning
[Hinton 09 ] DBN tutorial
![Page 16: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/16.jpg)
Deep Belief Networks
Random
Initial position
Pre-trained
weights
Good
Solution
Input-based
pre-training
Slow
Fine-tuning
Very slow
Back-propagation
[MS Ram]
![Page 17: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/17.jpg)
Deep Belief Networks
• Pre-train network from input-data
alone (generative step)
• Use weights of pre-trained network
as the initial point for traditional back-
propagation
• Leads to quicker convergence
• Pre-training is fast; fine-tuning can be
slow
![Page 18: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/18.jpg)
Searching in parameter space
One layer :1000 input + 1000 hidden
≈ 1 million weights
million-dimensional optimization
need to find global (or at least good)
optimum from random initialization)
Impossibly slow for Gradient descent
![Page 19: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/19.jpg)
Deep Belief Networks
Random
Initial position
Good
Solution
Very slow
Back-propagation
million-dimensional space
![Page 20: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/20.jpg)
Searching in parameter space
One layer :1000 input + 1000 hidden
≈ 1 million weights
million-dimensional optimization
Impossibly slow for Gradient descent to find
global optimum from random initialization
• Added complications:
• gradient magnitude vanishingly small in lower
parts of network
• deep networks tend to have more local minima
than shallow networks
![Page 21: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/21.jpg)
In practice : MLP vs DBN (MNIST)
MLP (1 Hidden Layer)
1 hour: 2.18%
14 hours: 1.65%
DBN
1 hour: 1.65%
14 hours: 1.10%
21 hours: 0.97%
Intel QuadCore 2.83GHz, 4GB RAM
[MS Ram]
![Page 22: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/22.jpg)
Deep network architecture (MNIST)
…
…
…
0.02 mn
…
…
1 mn
0.25 mn
0.4 mn
Output: 10
Input: 784
500
500
2000
Pre
-tra
inin
g
Fin
e-t
unin
g
# weights # nodes
![Page 23: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/23.jpg)
Designing DBNs
Relative importance of Depth of network :
Seems quite important
Layer-wise pre-training Counter-example: MNIST: 6-layer MLP: 784-2500-2000-
1500-1000-500-10 (on GPU, w elastic distortions)
Error Rate: 0.35% [Ciresan et al 2010]
“No fashionable unsupervised pre-training is necessary! “ - Jürgen Schmidhuber
Amount of labeled training data Affine and Elastic distortions
Main benefit: DBNs work w less training data
![Page 24: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/24.jpg)
Kernel Methods
Bishop, Ch.6
R & N ch 18.6
![Page 25: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/25.jpg)
Parametric: Discriminative :
![Page 26: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/26.jpg)
Parametric: Generative
image from [Herbrich 2002]
![Page 27: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/27.jpg)
Parametric vs Memory models
• Parametric models:
– learn model for data: parameter vector w /
posterior distribution p(w | t1..t
N)
– discard training set t
– e.g. linear classifiers (perceptron)
• Non-Parametric :
– models on data e.g. k-NN
– memory-based: some or all of the training
data is saved
– SVM: save a set of “support vectors”
![Page 28: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/28.jpg)
MNIST dataset
0
1
image from [Herbrich 2002]
![Page 29: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/29.jpg)
k-NN
Test 5 Nearest neighbours
![Page 30: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/30.jpg)
Kernel methods
• feature space mapping φ(x):
k(x, x') = φ(x)Tφ(x')
• symmetric: k(x, x') = k(x', x)
• linear kernel: φ(x) = x
• stationary kernel: k(x, x') = k(x − x') [stationary
under translation]
• homogeneous kernel:
k(x, x') = k(|x − x'|) (e.g. RBF)
![Page 31: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/31.jpg)
Support Vector Machines
• Main idea:
– linear classifier, but in φ(x) kernel space.
– criterion for decision: max-margin
• Algorithm
– user specifies kernel function
– learn weights for instances
– no actual computation in high-dim space
• Classification
– average of the instance labels, weighted
by a) proximity b) instance weight.
![Page 32: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/32.jpg)
Example: XOR
* X-OR problem
φ(x1,x
2) = {x
1, x
2, x
1x
2}
Better:
φ(x1,x
2) = {1, √2x
1, √2x
2, x
12
, √2 x
1x
2, x
22 }
![Page 33: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/33.jpg)
Margin maximization
yi = +1
yi = -1
Decision hyperplane:
wT x + b = 0
If ti = {+1,-1}, margin m=
1/||w|| mini ti (wT xi + b)
w must satisfy the
constraint that for all
data (xi,ti) :
ti (wT xi + b ) > m
Margin is maximized
when 1/||w|| is maximum,
ie. minimize ||w||
![Page 34: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/34.jpg)
Support Vector Machines
• Main idea:
– linear classifier, but in φ(x) kernel space.
– criterion for decision: max-margin
• Algorithm
– user specifies kernel function
– learn weights for instances
– no actual computation in high-dim space
• Classification
– average of the instance labels, weighted
by a) proximity b) instance weight.
![Page 35: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/35.jpg)
Kernel trick
![Page 36: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/36.jpg)
Demo
Demo by Udi Aharoni http://www.youtube.com/watch?v=3liCbRZPrZA
![Page 37: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/37.jpg)
Support Vector Machines
• Main idea:
– linear classifier, but in φ(x) kernel space.
– criterion for decision: max-margin
• Algorithm
– user specifies kernel function
– learn weights for instances via convex
optimization
– no actual computation in high-dim space
• Classification
– average of the instance labels, weighted
by a) proximity b) instance weight.
![Page 38: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/38.jpg)
Kernel trick
• Linear classifier is in in high-dimensional ϕ(x) space
• However, no computation directly on ϕ(x); compute
only kernel = ϕ(x)Tϕ(x)
e.g. for
ϕ(x1,x
2) = {1, √2x
1, √2x
2, x
12, √2 x
1x
2, x
2
2 }
k(x,x') = ϕ(x)Tϕ(x')
= ( (1, x1, x
2) (1, x'
1, x'
2)T )2 = <x,x'>2
• Efficient only if scalar product can be efficiently
computed. Holds for:
• k(x,x') : continuous, symmetric and positive definite
![Page 39: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/39.jpg)
Dual Representations
• Scalar product representations arise naturally in
many classes of problems
• e.g. Linear regression
• Setting gradient to zero:
where ΦT = matrix of ϕ(x
n), and
![Page 40: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/40.jpg)
Dual Representations
• Instead of parameter space w, use parameter
space a
• Writing w = ΦTa into J(w):
where gram matrix K = ΦΦT, with Knm
= ϕ(xn)Tϕ(x
m)
solving for a by setting dJ(a)/da to zero:
![Page 41: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/41.jpg)
Dual Representations
• Substituting back into original regression model:
• Thus, all computations are performed purely with
the kernel
![Page 42: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/42.jpg)
SVM
• Why so popular:
– Very good classification performance,
compares w best
– Fast (convex) and scaleable learning
– Fast inference (but slower training)
• Difficulties:
– No model (discriminative; black-box)
– Not as useful for discrete inputs.
– Art: how to specify kernel function
– Difficulties with multiple classes
![Page 43: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/43.jpg)
Choosing Kernels
Popular kernels that satisfy the Gram
matrix positive-definiteness criterion inlcude:
– Linear kernels: k(x,x') = <x,x'>
– Polynomials: k(x,x') = <x,x'>n
• for n=2 (quadratic):
Polynomial basis functions k(x,x') = for x' = 0
![Page 44: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/44.jpg)
Choosing Kernels
– Gaussian:
– Radial basis functions
– Sigmoid (logistic) function
![Page 45: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/45.jpg)
Gaussian / Sigmoid Bases
gaussian bases logistic bases
k(x,0)
image from [Bishop 2005]
![Page 46: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/46.jpg)
Constructing Kernels
If k1 and k2 are valid kernels, then so are:
![Page 47: Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions](https://reader030.vdocuments.net/reader030/viewer/2022041014/5ec4d2babb6adc6f665f095e/html5/thumbnails/47.jpg)
Reading
• Machine Learning
Bishop : sections 1.1 to 1.2.4, 1.5.1-2, 6.1, 6.2
Russell & Norvig: 18.9