uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · uncertainty,...
TRANSCRIPT
![Page 1: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/1.jpg)
Uncertainty, precision, prediction errors –
and their relevance to computational psychiatry
Christoph Mathys
Wellcome Trust Centre for Neuroimaging at UCL, London, UK
Max Planck UCL Centre for Computational Psychiatry and Ageing Research, London, UK
Black Dog Institute, University of New South Wales
March 6, 2015
![Page 2: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/2.jpg)
Imagine the following situation:
You’re on a boat, you’re lost in a storm and trying to get back to shore. A lighthouse has just appeared on the horizon, but you can only see it when you’re at the peak of a wave. Your GPS etc., has all been washed overboard, but what you can still do to get an idea of your position is to measure the angle between north and the lighthouse. These are your measurements (in degrees):
76, 73, 75, 72, 77
What number are you going to base your calculation on?
Right. The mean: 74.6. How do you calculate that?
2
Uncertainty: a shamelessly artificial example
![Page 3: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/3.jpg)
The usual way to calculate the mean 𝑥 of 𝑥1, 𝑥2, … , 𝑥𝑛 is to take
𝑥 =1
𝑛 𝑥𝑖
𝑛
𝑖=1
This requires you to remember all 𝑥𝑖 , which can become inefficient. Since the measurements arrive sequentially, we would like to update 𝑥 sequentially as the 𝑥𝑖 come in – without having to remember them.
It turns out that this is possible. After some algebra (see next slide), we get
𝑥 𝑛 = 𝑥 𝑛−1 +1
𝑛𝑥𝑛 − 𝑥 𝑛−1
3
Uncertainty: updates to the mean
![Page 4: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/4.jpg)
Proof of sequential update formula:
𝑥 𝑛 =1
𝑛 𝑥𝑖 =
𝑥𝑛
𝑛
𝑛
𝑖=1
+1
𝑛 𝑥𝑖 =
𝑥𝑛
𝑛
𝑛−1
𝑖=1
+𝑛 − 1
𝑛
1
𝑛 − 1 𝑥𝑖
𝑛−1
𝑖=1
=𝑥 𝑛−1
=
=𝑥𝑛
𝑛+
𝑛 − 1
𝑛𝑥 𝑛−1 = 𝑥 𝑛−1 +
𝑥𝑛
𝑛+
𝑛 − 1
𝑛𝑥 𝑛−1 −
𝑛
𝑛𝑥 𝑛−1 =
= 𝑥 𝑛−1 +1
𝑛𝑥𝑛 + 𝑛 − 1 − 𝑛 𝑥 𝑛−1 = 𝑥 𝑛−1 +
1
𝑛𝑥𝑛 − 𝑥 𝑛−1
q.e.d.
4
Uncertainty: updates to the mean
![Page 5: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/5.jpg)
The seqential updates in our example now look like this:
5
Uncertainty: updates to the mean
72 73 74 75 76 77
𝑥 1 = 76
𝑥 2 = 76 +1
273 − 76 = 74.5
𝑥 3 = 74.5 +1
375 − 74.5 = 74.6
𝑥 4 = 74.6 +1
472 − 74.6 = 74
𝑥 5 = 74 +1
577 − 74 = 74.6
![Page 6: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/6.jpg)
𝑥 𝑛 = 𝑥 𝑛−1 +1
𝑛𝑥𝑛 − 𝑥 𝑛−1
6
What are the building blocks of the updates we’ve just seen, and where does uncertainty enter?
prediction
prediction error
new input
weight (learning rate)
So where is uncertainty? The learning rate reflects uncertainty: the less we know, the higher the learning rate.
![Page 7: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/7.jpg)
• More specifically, does it generalize to Bayesian inference?
• «Bayesian inference» simply means inference on uncertain quantities according to the rules of probability theory (i.e., according to logic).
• Agents who use Bayesian inference will make better predictions (provided they have a good model of their environment), which will give them an evolutionary advantage.
• We may therefore assume that evolved biological agents use Bayesian inference, or a close approximation to it.
• So is Bayesian inference based on predictions that are updated using uncertainty-weighted prediction errors?
7
Is this a general pattern?
![Page 8: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/8.jpg)
• Think boat, lighthouse, etc., again, but now we’re doing Bayesian inference.
• Before we make the next observation, our belief about the true angle 𝜗 can be described by a Gaussian prior:
𝑝(𝜗) ∼ 𝒩(𝜇𝜗, 𝜋𝜗
−1)
• The likelihood of our observation is also Gaussian, with precision 𝜋𝜀:
𝑝 𝑥 𝜗 ∼ 𝒩 𝜗, 𝜋𝜀
−1
• Bayes’ rule now tells us that the posterior is Gaussian again:
𝑝 𝜗 𝑥 =𝑝 𝑥 𝜗 𝑝(𝜗)
𝑝 𝑥 𝜗′ 𝑝 𝜗′ d𝜗′∼ 𝒩 𝜇𝜗|𝑦, 𝜋𝜗|𝑦
−1
8
Updates in a simple Gaussian model
![Page 9: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/9.jpg)
• Here’s how the updates to the sufficent statistics 𝜇 and 𝜋 describing our belief look like:
𝜋𝜗|𝑥 = 𝜋𝜗 + 𝜋𝜀
𝜇𝜗|𝑥 = 𝜇𝜗 +𝜋𝜀
𝜋𝜗|𝑥(𝑥 − 𝜇𝜗)
• So it’s the same story all over again: the mean is updated by an uncertainty-weighted (more specifically: prediction-weighted) prediction error.
• The size of the update is proportional to the likelihood precision and inversely proportional to the posterior precision.
• This pattern is not specific to the univariate Gaussian case, but generalizes to Bayesian updates for all exponential families of likelihood distributions with conjugate priors (i.e., to all formal descriptions of inference you are ever likely to need).
9
Updates in a simple Gaussian model
prediction weight (learning rate)=how much we′re learning here
how much we already know
prediction error
![Page 10: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/10.jpg)
10
Does inference as we’ve described it adequately describe the situation of actual biological agents?
𝜆 𝑥
Sensory input
True hidden states
Inferred hidden states
Action
𝑢
𝑎
World Agent No, the dynamics are missing!
![Page 11: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/11.jpg)
• Up to now, we’ve only looked at inference on static quantities, but biological agents live in a continually changing world.
• In our example, the boat’s position changes and with it the angle to the lighthouse.
• How can we take into account that old information becomes obsolete? If we don’t, our learning rate becomes smaller and smaller because our eqations were derived under the assumption that we’re accumulating information about a stable quantity.
11
What about dynamics?
![Page 12: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/12.jpg)
• Keep it constant!
• So, taking the update equation for the mean of our observations as our point of departure...
𝑥 𝑛 = 𝑥 𝑛−1 +1
𝑛𝑥𝑛 − 𝑥 𝑛−1 ,
• ... we simply replace 1
𝑛 with a constant 𝛼:
𝜇𝑛 = 𝜇𝑛−1 + 𝛼 𝑥𝑛 − 𝜇𝑛−1 .
• This is called Rescorla-Wagner learning [although it wasn’t this line of reasoning that led Rescorla & Wagner (1972) to their formulation].
12
What’s the simplest way to keep the learning rate from going too low?
![Page 13: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/13.jpg)
• Partly: it implies a certain rate of forgetting because it amounts to taking
only the 𝑛 =1
𝛼 last data points into account. But...
• ... if the learning rate is supposed to reflect uncertainty in Bayesian inference, then how do we
• (a) know that 𝛼 reflects the right level of uncertainty at any one time, and
• (b) account for changes in uncertainty if 𝛼 is constant?
• What we really need is an adaptive learning that accurately reflects uncertainty.
13
Does a constant learning rate solve our problems?
![Page 14: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/14.jpg)
• This requires us to think a bit more about what kinds of uncertainty we are dealing with.
• A possible taxonomy of uncertainty is (cf. Yu & Dayan, 2003; Payzan-LeNestour & Bossaerts, 2011):
• (a) outcome uncertainty that remains unaccounted for by the model, called risk by economists (𝜋𝜀 in our Bayesian example); this uncertainty remains even when we know all parameters exactly,
• (b) informational or expected uncertainty about the value of model parameters (𝜋𝜗|𝑥
in the Bayesian example),
• (c) environmental or unexpected uncertainty owing to changes in model parameters (not accounted for in our Bayesian example, hence unexpected).
14
An adaptive learning rate that accurately reflects uncertainty
![Page 15: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/15.jpg)
• Various efforts have been made to come up with an adaptive learning rate:
– Kalman (1960)
– Sutton (1992)
– Nassar et al. (2010)
– Payzan-LeNestour & Bossaerts (2011)
– Mathys et al. (2011)
– Wilson et al. (2013)
• The Kalman filter is optimal for linear dynamical systems, but realistic data usually require non-linear models.
• Mathys et al. use a generic non-linear hierarchical Bayesian model that allows us to derive update equations that are optimal in the sense that they minimize surprise.
15
An adaptive learning rate that accurately reflects uncertainty
![Page 16: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/16.jpg)
16
The hierarchical Gaussian filter (HGF)
𝑝 𝑥𝑛(𝑘)
𝑥2(𝑘−1)
𝑥𝑛(𝑘)
~𝒩 𝑥𝑛(𝑘−1)
, 𝜗
𝑥1(𝑘)
~𝒩 𝑥1𝑘−1
, 𝑓1 𝑥2
𝑝 𝑥1(𝑘)
𝑥1(𝑘−1)
𝑥2(𝑘)
~𝒩 𝑥2𝑘−1
, 𝑓2 𝑥3
𝑝 𝑥2(𝑘)
𝑥2(𝑘−1)
𝑥𝑖(𝑘)
~𝒩 𝑥𝑖𝑘−1
, 𝑓𝑖 𝑥𝑖+1
𝑝 𝑥𝑖(𝑘)
𝑥𝑖(𝑘−1)
![Page 17: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/17.jpg)
• At the outcome level (i.e., at the very bottom of the hierarchy), we have
𝑢(𝑘) ~ 𝒩 𝑥1𝑘
, 𝜋 𝑢−1
• This gives us the following update for our belief on 𝑥1 (our quantity of interest):
𝜋1(𝑘)
= 𝜋 1(𝑘)
+ 𝜋 𝑢
𝜇1(𝑘)
= 𝜇1(𝑘−1)
+𝜋 𝑢
𝜋1(𝑘)
𝑢 𝑘 − 𝜇1(𝑘−1)
• The familiar structure again – but now with a learning rate that is responsive to all kinds of uncertainty, including environmental (unexpected) uncertainty.
17
The hierarchical Gaussian filter (HGF)
![Page 18: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/18.jpg)
Unpacking the learning rate, we see:
𝜋 𝑢
𝜋1(𝑘)
=𝜋 𝑢
𝜋 1(𝑘)
+ 𝜋 𝑢=
𝜋 𝑢1
𝜎1(𝑘−1)
+ exp 𝜅1𝜇2(𝑘−1)
+ 𝜔1
+ 𝜋 𝑢
18
The learning rate in the HGF
informational uncertainty
environmental uncertainty
outcome uncertainty
![Page 19: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/19.jpg)
19
The learning rate in the HGF
Andreea Diaconescu
![Page 20: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/20.jpg)
20
HGF: empirical evidence (Iglesias et al., 2013)
Model comparison:
![Page 21: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/21.jpg)
21
HGF: empirical evidence (Iglesias et al., 2013)
![Page 22: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/22.jpg)
22
HGF: empirical evidence (Iglesias et al., 2013)
![Page 23: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/23.jpg)
23
HGF: empirical evidence (Iglesias et al., 2013)
![Page 24: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/24.jpg)
24
How to estimate and compare models: the HGF Toolbox
• Available at
http://www.translationalneuromodeling.org/tapas
• Start with README, manual, and interactive demo
• Modular, extensible
• Matlab-based
![Page 25: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/25.jpg)
A failure of sensory or interoceptive attenuation may be at the root of many clinical phenomena.
25
Why is this important for computational psychiatry?
![Page 26: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/26.jpg)
26
Example: Abnormalities of smooth pursuit eye movements in schizophrenia
Thaker et al., 1999; Hong et al., 2005; Adams et al., 2013
![Page 27: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/27.jpg)
27
The neurobiology of the Bayesian brain: predictive coding
(1)
x
(1)
x
(1)
v( )s t
(1)
v
10 20 30 40 50 60 -5
0
5
10
15
20
prediction and error
10 20 30 40 50 60 -5
0
5
10
15
20
hidden states
Backward predictions
Forward prediction error
10 20 30 40 50 60 -10
-5
0
5
10
15
20
causal states
stimulus
0.2 0.4 0.6 0.8 2000
2500
3000
3500
4000
4500
5000
time (seconds)
Friston, 2008
![Page 28: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/28.jpg)
28
Example: Hierarchical message passing in the visual-oculomotor system
frontal eye fields geniculate
visual cortex
retinal input
pons
oculomotor signals
( ) ( ) ( ) ( ) ( 1)
( ) ( ) ( ) ( )
i i i i i
v v v v
i i i i
x x x
D
D
( )i
( )i
Errors (superficial pyramidal cells)
Expectations (deep pyramidal cells)
Top-down or backward predictions
Bottom-up or forward prediction error
proprioceptive input reflex arc
Perception
Action
(1)
a va s
( ) ( ) ( ) ( ) ( 1) ( ) ( ) ( )
( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )
( ( , ))
( ( , ))
i i i i i i i i
v v v v v x v
i i i i i i i i
x x x x x x v
g
f
Precision-weighted prediction errors
Adams et al., 2013
![Page 29: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/29.jpg)
29
Overweighting of sensory evidence
Adams et al., 2013
![Page 30: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/30.jpg)
30
Smooth pursuit of a partially occluded target with and without high-level precision
Adams et al., 2013
![Page 31: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/31.jpg)
31
Failure of input attenuation: further examples
• Force matching illusion (Shergill et al., 2003; 2005; Teufel et al., 2010;
Brown et al., 2013)
• Cornsweet effect (Brown & Friston, 2012)
• Hollow mask illusion (Dima et al., 2009; 2010)
https://www.youtube.com/watch?v=ORoTCBrCKIQ
https://www.youtube.com/watch?v=6YIPtJlCbIA
• Attenuation of interoceptive signals in autism (Lawson et al., 2014;
Quattrocki & Friston, 2014)
![Page 32: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/32.jpg)
32
Summary
• We have to make good predictions to avoid surprise and survive, that is
we have to use probabilistic (i.e., Bayesian) inference based on a good
model of our environment.
• Bayesian inference means updating beliefs by uncertainty- (i.e.,
precision-) weighted prediction errors.
• Precision-weighting has to take account of all forms of uncertainty.
• A breakdown in this may be the root of many psychopathological
phenomena.
![Page 33: Uncertainty, precision, prediction errors and their relevance to … · 2020-07-15 · Uncertainty, precision, prediction errors – and their relevance to computational psychiatry](https://reader033.vdocuments.net/reader033/viewer/2022050409/5f860e04b65a375be3591265/html5/thumbnails/33.jpg)
33
Thanks