introduction to gaussian processes - emtiyaz khan · motivation 2 goals of this lecture –...
TRANSCRIPT
![Page 1: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/1.jpg)
Introduction to Gaussian Processes
Carlos Becker, Emtiyaz KhanDecember 2015
Pattern Recognition & Machine Learning Course, EPFL
![Page 2: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/2.jpg)
Motivation2
Goals of this lecture
– Understand what a Gaussian Process (GP) is.
– Learn how GPs can be used for regression.
More specific to GPs, you will learn:
– What a covariance matrix means from a GP point of view.
– How a GP defines a prior over functions, and its relationship to its covariance matrix and correlation terms.
– What “conditioning on the measurements” means, in a probabilistic sense as well as mathematically.
Note: GPs for classification are outside the scope of this lecture. But if you understand regression GPs it won't be too difficult to learn how classification GPs work. Please see Rasmusen and William's “Gaussian Processes for Machine Learning” book.
![Page 3: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/3.jpg)
Motivation: why Gaussian Processes?
![Page 4: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/4.jpg)
Motivation4
Say we want to estimate a scalar function
from training data
x1 x2 x3
y1
y2y3
![Page 5: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/5.jpg)
Motivation5
Say we want to estimate a scalar function
from training data
x1 x2 x3
f1
f2f3
x1 x2 x3
y1
y2y3
2nd Order Polynomial
![Page 6: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/6.jpg)
Motivation6
Say we want to estimate a scalar function
from training data
x1 x2 x3
f1
f2f3
x1 x2 x3
y1
y2y3
3rd Order Polynomial
![Page 7: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/7.jpg)
Motivation7
Say we want to estimate a scalar function
from training data
x1 x2 x3
f1
f2f3
x1 x2 x3
y1
y2y3
Radial Basis Function
![Page 8: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/8.jpg)
Motivation8
Gaussian Processes let us place a prior on the 'shape' of
And this prior is formulated probabilistically
![Page 9: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/9.jpg)
Let's get started!
![Page 10: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/10.jpg)
Just before.10
![Page 11: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/11.jpg)
Gaussian Processes11
Let's get back to finding , but now
– From a single observation
– We want to predict
x1
f1
![Page 12: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/12.jpg)
Gaussian Processes12
x1
f1
Now, in a probabilistic manner.
![Page 13: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/13.jpg)
Gaussian Processes13
x1
f1
Getting there.
![Page 14: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/14.jpg)
Gaussian Processes14
What about multiple training points?
![Page 15: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/15.jpg)
Gaussian Processes15
![Page 16: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/16.jpg)
Gaussian Processes16
![Page 17: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/17.jpg)
Gaussian Processes17
![Page 18: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/18.jpg)
Gaussian Processes18
![Page 19: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/19.jpg)
Gaussian Processes19
![Page 20: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/20.jpg)
Gaussian Processes20
![Page 21: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/21.jpg)
Gaussian Processes21
![Page 22: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/22.jpg)
Gaussian Processes22
![Page 23: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/23.jpg)
Gaussian Processes23
![Page 24: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/24.jpg)
Gaussian Processes24
![Page 25: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/25.jpg)
Exercise25
![Page 26: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/26.jpg)
Time to get to the Real GP
![Page 27: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/27.jpg)
Gaussian Processes27
![Page 28: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/28.jpg)
Gaussian Processes28
1) A prior over functions
![Page 29: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/29.jpg)
Gaussian Processes29
1) A prior over functions
![Page 30: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/30.jpg)
Gaussian Processes30
1) A prior over functions
![Page 31: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/31.jpg)
Gaussian Processes31
1) A prior over functions
![Page 32: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/32.jpg)
Gaussian Processes32
1) A prior over functions
![Page 33: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/33.jpg)
Gaussian Processes33
1) A prior over functions
![Page 34: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/34.jpg)
Gaussian Processes34
2) Incorporating noise-free measurements
![Page 35: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/35.jpg)
Gaussian Processes35
2) Incorporating noise-free measurements
![Page 36: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/36.jpg)
Gaussian Processes36
3) Incorporating noisy measurements (as in real life)
![Page 37: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/37.jpg)
Gaussian Processes37
3) Incorporating noisy measurements (as in real life)
![Page 38: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/38.jpg)
Gaussian Processes38
Demo Time
![Page 39: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/39.jpg)
Gaussian Processes39
A Real Example [Rasmussen, Williams, Gaussian Processes for Machine Learning]
?
![Page 40: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/40.jpg)
Gaussian Processes40
A Real Example [Rasmussen, Williams, Gaussian Processes for Machine Learning]
![Page 41: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/41.jpg)
Gaussian Processes41
A Real Example [Rasmussen, Williams, Gaussian Processes for Machine Learning]
Kernel Design:
11 parameters
![Page 42: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/42.jpg)
Gaussian Processes42
What about Classification?
![Page 43: Introduction to Gaussian Processes - Emtiyaz Khan · Motivation 2 Goals of this lecture – Understand what a Gaussian Process (GP) is. – Learn how GPs can be used for regression](https://reader034.vdocuments.net/reader034/viewer/2022042306/5ed29e549c816a0c43326367/html5/thumbnails/43.jpg)
Gaussian Processes44
Summary