reproducing kernel hilbert spaces9.520/fall14/slides/class03/class03_rkhspart1.pdfhilbert spaces...
TRANSCRIPT
![Page 1: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/1.jpg)
Reproducing Kernel Hilbert Spaces
Lorenzo Rosasco
9.520 Class 03
L. Rosasco RKHS
![Page 2: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/2.jpg)
About this class
Goal To introduce a particularly useful family ofhypothesis spaces called Reproducing KernelHilbert Spaces (RKHS) We will discuss severalperspectives on RKHS. In particular in this classwe investigate the fundamental definition of RKHSas Hilbert spaces with bounded, continuousevaluation functionals and the intimate connectionwith symmetric positive definite kernels.
L. Rosasco RKHS
![Page 3: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/3.jpg)
Plan
Part I: RKHS are Hilbert spaces with bounded, continuousevaluation functionals.Part II: Reproducing KernelsPart III: Mercer TheoremPart IV: Feature MapsPart V: Representer Theorem
L. Rosasco RKHS
![Page 4: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/4.jpg)
Regularization
The basic idea of regularization (originally introducedindependently of the learning problem) is to restorewell-posedness of ERM by constraining the hypothesis spaceH.
RegularizationA possible way to do this is considering regularized empiricalrisk minimization, that is we look for solutions minimizing a twoterm functional
ERR(f )︸ ︷︷ ︸empirical error
+λ R(f )︸︷︷︸regularizer
the regularization parameter λ trade-offs the two terms.
L. Rosasco RKHS
![Page 5: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/5.jpg)
Tikhonov Regularization
Tikhonov regularization amounts to minimize
1n
n∑i=1
V (f (xi), yi) + λR(f ) λ > 0 (1)
V (f (x), y) is the loss function, that is the price we paywhen we predict f (x) in place of yR(f ) is a regularizer– often R(f ) = ‖ · ‖H, the norm in thefunction space H
The regularizer should encode some notion of smoothness of f .
L. Rosasco RKHS
![Page 6: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/6.jpg)
The "Ingredients" of Tikhonov Regularization
The scheme we just described is very general and bychoosing different loss functions V (f (x), y) we can recoverdifferent algorithmsThe main point we want to discuss is how to choose anorm encoding some notion of smoothness/complexity ofthe solutionReproducing Kernel Hilbert Spaces allow us to do this in avery powerful way
L. Rosasco RKHS
![Page 7: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/7.jpg)
Different Views on RKHS
L. Rosasco RKHS
![Page 8: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/8.jpg)
Part I: Evaluation Functionals
L. Rosasco RKHS
![Page 9: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/9.jpg)
Some Functional Analysis
A function space F is a space whose elements are functionsf , for example f : Rd → R.A norm is a nonnegative function ‖ · ‖ such that ∀f ,g ∈ F andα ∈ R
1 ‖f‖ ≥ 0 and ‖f‖ = 0 iff f = 0;2 ‖f + g‖ ≤ ‖f‖+ ‖g‖;3 ‖αf‖ = |α| ‖f‖.
A norm can be defined via a inner product ‖f‖ =√〈f , f 〉.
A Hilbert space is a complete inner product space.
L. Rosasco RKHS
![Page 10: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/10.jpg)
Some Functional Analysis
A function space F is a space whose elements are functionsf , for example f : Rd → R.A norm is a nonnegative function ‖ · ‖ such that ∀f ,g ∈ F andα ∈ R
1 ‖f‖ ≥ 0 and ‖f‖ = 0 iff f = 0;2 ‖f + g‖ ≤ ‖f‖+ ‖g‖;3 ‖αf‖ = |α| ‖f‖.
A norm can be defined via a inner product ‖f‖ =√〈f , f 〉.
A Hilbert space is a complete inner product space.
L. Rosasco RKHS
![Page 11: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/11.jpg)
Some Functional Analysis
A function space F is a space whose elements are functionsf , for example f : Rd → R.A norm is a nonnegative function ‖ · ‖ such that ∀f ,g ∈ F andα ∈ R
1 ‖f‖ ≥ 0 and ‖f‖ = 0 iff f = 0;2 ‖f + g‖ ≤ ‖f‖+ ‖g‖;3 ‖αf‖ = |α| ‖f‖.
A norm can be defined via a inner product ‖f‖ =√〈f , f 〉.
A Hilbert space is a complete inner product space.
L. Rosasco RKHS
![Page 12: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/12.jpg)
Some Functional Analysis
A function space F is a space whose elements are functionsf , for example f : Rd → R.A norm is a nonnegative function ‖ · ‖ such that ∀f ,g ∈ F andα ∈ R
1 ‖f‖ ≥ 0 and ‖f‖ = 0 iff f = 0;2 ‖f + g‖ ≤ ‖f‖+ ‖g‖;3 ‖αf‖ = |α| ‖f‖.
A norm can be defined via a inner product ‖f‖ =√〈f , f 〉.
A Hilbert space is a complete inner product space.
L. Rosasco RKHS
![Page 13: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/13.jpg)
Examples
Continuous functions C[a,b] :a norm can be established by defining
‖f‖ = maxa≤x≤b
|f (x)|
(not a Hilbert space!)Square integrable functions L2[a,b]:it is a Hilbert space where the norm is induced by the dotproduct
〈f ,g〉 =
∫ b
af (x)g(x)dx
L. Rosasco RKHS
![Page 14: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/14.jpg)
Examples
Continuous functions C[a,b] :a norm can be established by defining
‖f‖ = maxa≤x≤b
|f (x)|
(not a Hilbert space!)Square integrable functions L2[a,b]:it is a Hilbert space where the norm is induced by the dotproduct
〈f ,g〉 =
∫ b
af (x)g(x)dx
L. Rosasco RKHS
![Page 15: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/15.jpg)
Hypothesis Space: Desiderata
Hilbert Space.Point-wise defined functions.
L. Rosasco RKHS
![Page 16: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/16.jpg)
Hypothesis Space: Desiderata
Hilbert Space.Point-wise defined functions.
L. Rosasco RKHS
![Page 17: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/17.jpg)
RKHS
An evaluation functional over the Hilbert space of functions H isa linear functional Ft : H → R that evaluates each function inthe space at the point t , or
Ft [f ] = f (t).
DefinitionA Hilbert space H is a reproducing kernel Hilbert space(RKHS) if the evaluation functionals are bounded andcontinuous, i.e. if there exists a M s.t.
|Ft [f ]| = |f (t)| ≤ M‖f‖H ∀f ∈ H
L. Rosasco RKHS
![Page 18: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/18.jpg)
RKHS
An evaluation functional over the Hilbert space of functions H isa linear functional Ft : H → R that evaluates each function inthe space at the point t , or
Ft [f ] = f (t).
DefinitionA Hilbert space H is a reproducing kernel Hilbert space(RKHS) if the evaluation functionals are bounded andcontinuous, i.e. if there exists a M s.t.
|Ft [f ]| = |f (t)| ≤ M‖f‖H ∀f ∈ H
L. Rosasco RKHS
![Page 19: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/19.jpg)
Evaluation functionals
Evaluation functionals are not always bounded.Consider L2[a,b]:
Each element of the space is an equivalence class offunctions with the same integral
∫|f (x)|2dx .
An integral remains the same if we change the function ina countable set of points.
L. Rosasco RKHS
![Page 20: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/20.jpg)
Norms in RKHS and Smoothness
Choosing different kernels one can show that the norm in thecorresponding RKHS encodes different notions of smoothness.
Band limited functions. Consider the set of functions
H := {f ∈ L2(R) | F (ω) ∈ [−a,a],a <∞}
with the usual L2 inner product. the function at every pointis given by the convolution with a sinc function sin(ax)/ax .The norm
‖f‖2H =
∫f (x)2dx =
∫ a
a|F (ω)|2dω
Where F (ω) = F{f}(ω) =∫∞−∞ f (t)e−iωt dt is the Fourier
tranform of f .
L. Rosasco RKHS
![Page 21: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/21.jpg)
Norms in RKHS and Smoothness
Choosing different kernels one can show that the norm in thecorresponding RKHS encodes different notions of smoothness.
Band limited functions. Consider the set of functions
H := {f ∈ L2(R) | F (ω) ∈ [−a,a],a <∞}
with the usual L2 inner product. the function at every pointis given by the convolution with a sinc function sin(ax)/ax .The norm
‖f‖2H =
∫f (x)2dx =
∫ a
a|F (ω)|2dω
Where F (ω) = F{f}(ω) =∫∞−∞ f (t)e−iωt dt is the Fourier
tranform of f .
L. Rosasco RKHS
![Page 22: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/22.jpg)
Norms in RKHS and Smoothness
Sobolev Space: consider f : [0,1]→ R withf (0) = f (1) = 0. The norm
‖f‖2H =
∫(f ′(x))2dx =
∫ω2|F (ω)|2dω
Gaussian Space: the norm can be written as
‖f‖2H =1
2πd
∫|F (ω)|2exp
σ2ω22 dω
L. Rosasco RKHS
![Page 23: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/23.jpg)
Norms in RKHS and Smoothness
Sobolev Space: consider f : [0,1]→ R withf (0) = f (1) = 0. The norm
‖f‖2H =
∫(f ′(x))2dx =
∫ω2|F (ω)|2dω
Gaussian Space: the norm can be written as
‖f‖2H =1
2πd
∫|F (ω)|2exp
σ2ω22 dω
L. Rosasco RKHS
![Page 24: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/24.jpg)
Norms in RKHS and Smoothness
Sobolev Space: consider f : [0,1]→ R withf (0) = f (1) = 0. The norm
‖f‖2H =
∫(f ′(x))2dx =
∫ω2|F (ω)|2dω
Gaussian Space: the norm can be written as
‖f‖2H =1
2πd
∫|F (ω)|2exp
σ2ω22 dω
L. Rosasco RKHS
![Page 25: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/25.jpg)
Linear RKHS
Our function space is 1-dimensional lines
f (x) = w x
where the RKHS norm is simply
‖f‖2H = 〈f , f 〉H = w2
so that our measure of complexity is the slope of the line.We want to separate two classes using lines and see how themagnitude of the slope corresponds to a measure of complexity.We will look at three examples and see that each examplerequires more "complicated functions, functions with greaterslopes, to separate the positive examples from negativeexamples.
L. Rosasco RKHS
![Page 26: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/26.jpg)
Linear case (cont.)
here are three datasets: a linear function should be used toseparate the classes. Notice that as the class distinctionbecomes finer, a larger slope is required to separate theclasses.
−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2
−1.5
−1
−0.5
0
0.5
1
1.5
2
x
f(x)
−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2
−1.5
−1
−0.5
0
0.5
1
1.5
2
x
f(X)
−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2
−1.5
−1
−0.5
0
0.5
1
1.5
2
x
f(x)
L. Rosasco RKHS
![Page 27: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/27.jpg)
Part II: Kernels
L. Rosasco RKHS
![Page 28: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/28.jpg)
Different Views on RKHS
L. Rosasco RKHS
![Page 29: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/29.jpg)
Representation of Continuous Functionals
Let H be a Hilbert space and g ∈ H, then
Φg(f ) = 〈f ,g〉 , f ∈ H
is a continuous linear functional.
Riesz representation theoremThe theorem states that every continuous linear functional Φcan be written uniquely in the form,
Φ(f ) = 〈f ,g〉
for some appropriate element g ∈ H.
L. Rosasco RKHS
![Page 30: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/30.jpg)
Reproducing kernel (rk)
If H is a RKHS, then for each t ∈ X there exists, by theRiesz representation theorem a function Kt in H (calledrepresenter) with the reproducing property
Ft [f ] = 〈Kt , f 〉H = f (t).
Since Kt is a function in H, by the reproducing property, foreach x ∈ X
Kt (x) = 〈Kt ,Kx〉H
The reproducing kernel (rk) of H is
K (t , x) := Kt (x)
L. Rosasco RKHS
![Page 31: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/31.jpg)
Reproducing kernel (rk)
If H is a RKHS, then for each t ∈ X there exists, by theRiesz representation theorem a function Kt in H (calledrepresenter) with the reproducing property
Ft [f ] = 〈Kt , f 〉H = f (t).
Since Kt is a function in H, by the reproducing property, foreach x ∈ X
Kt (x) = 〈Kt ,Kx〉H
The reproducing kernel (rk) of H is
K (t , x) := Kt (x)
L. Rosasco RKHS
![Page 32: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/32.jpg)
Positive definite kernels
Let X be some set, for example a subset of Rd or Rd itself. Akernel is a symmetric function K : X × X → R.
DefinitionA kernel K (t , s) is positive definite (pd) if
n∑i,j=1
cicjK (ti , tj) ≥ 0
for any n ∈ N and choice of t1, ..., tn ∈ X and c1, ..., cn ∈ R.
L. Rosasco RKHS
![Page 33: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/33.jpg)
RKHS and kernels
The following theorem relates pd kernels and RKHS
Theorema) For every RKHS there exist an associated reproducingkernel which is symmetric and positive definite
b) Conversely every symmetric, positive definite kernel K onX × X defines a unique RKHS on X with K as its reproducingkernel
L. Rosasco RKHS
![Page 34: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/34.jpg)
Sketch of proof
a) We must prove that the rk K (t , x) = 〈Kt ,Kx〉H is symmetricand pd.• Symmetry follows from the symmetry property of dot products
〈Kt ,Kx〉H = 〈Kx ,Kt〉H
• K is pd because
n∑i,j=1
cicjK (ti , tj) =n∑
i,j=1
cicj〈Kti ,Ktj 〉H = ||∑
cjKtj ||2H ≥ 0.
L. Rosasco RKHS
![Page 35: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/35.jpg)
Sketch of proof (cont.)
b) Conversely, given K one can construct the RKHS H as thecompletion of the space of functions spanned by the set{Kx |x ∈ X} with a inner product defined as follows.The dot product of two functions f and g in span{Kx |x ∈ X}
f (x) =s∑
i=1
αiKxi (x)
g(x) =s′∑
i=1
βiKx ′i(x)
is by definition
〈f ,g〉H =s∑
i=1
s′∑j=1
αiβjK (xi , x ′j ).
L. Rosasco RKHS
![Page 36: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/36.jpg)
Examples of pd kernels
Very common examples of symmetric pd kernels are• Linear kernel
K (x , x ′) = x · x ′
• Gaussian kernel
K (x , x ′) = e−‖x−x′‖2
σ2 , σ > 0
• Polynomial kernel
K (x , x ′) = (x · x ′ + 1)d , d ∈ N
For specific applications, designing an effective kernel is achallenging problem.
L. Rosasco RKHS
![Page 37: Reproducing Kernel Hilbert Spaces9.520/fall14/slides/class03/class03_rkhsPart1.pdfHilbert Spaces (RKHS) We will discuss several perspectives on RKHS. In particular in this class we](https://reader034.vdocuments.net/reader034/viewer/2022051605/600e97cd9878a6054952ae20/html5/thumbnails/37.jpg)
Examples of pd kernels
Kernel are a very general concept. We can have kernel onvectors, string, matrices, graphs, probabilities...Combinations of Kernels allow to do integrate differentkinds of data.Often times Kernel are views and designed to be similaritymeasure (in this case it make sense to have normalizedkernels)
d(x , x ′)2 =∥∥Kx − K ′x
∥∥2= 2(1− K (x , x ′)).
L. Rosasco RKHS