you’re doing it wrong!web.mit.edu/edbert/betancourtinference.pdf · you’re doing it wrong! an...
TRANSCRIPT
You’re Doing It Wrong!An Introduction to Bayesian Inference
Michael BetancourtPhysics Diversity Summit
January 25, 2010
Introduction to Bayesian Inference
Letting the Data Speak
** *
**
*
*
*
*
* *
**
**
*
xi
x j
D, p(D|!) , !?
2
Introduction to Bayesian Inference
Letting the Data Speak
** *
**
*
*
*
*
* *
**
**
*
xi
x j
D, p(D|!) , !?
2
Introduction to Bayesian Inference
Letting the Data Speak
** *
**
*
*
*
*
* *
**
**
*
xi
x j
D, p(D|!) , !?
2
Introduction to Bayesian Inference
Letting the Data Speak
** *
**
*
*
*
*
* *
**
**
*
xi
x j
D, p(D|!) , !?
2
Introduction to Bayesian Inference
Estimators
!(D) , !! "
3
Introduction to Bayesian Inference
Estimators
!(D) , !! "
!(!"")2# = (!!#"")2 +!!!2#"!!#2"
3
Introduction to Bayesian Inference
Estimators
!(D) , !! "
!(!"")2# = (!!#"")2 +!!!2#"!!#2"
3
Introduction to Bayesian Inference
Estimators
!(D) , !! "
!(!"")2# = (!!#"")2 +!!!2#"!!#2"
3
Introduction to Bayesian Inference
Estimators: Gaussian Example
p(D|!) = "i
N (xi|µ,#)
4
Introduction to Bayesian Inference
Estimators: Gaussian Example
p(D|!) = "i
N (xi|µ,#)
µ(D) = x =1n!
ixi !2 (D) =
1n!1"
i(xi! x)2
4
Introduction to Bayesian Inference
Estimators: Gaussian Example
p(D|!) = "i
N (xi|µ,#)
µ(D) = x =1n!
ixi
!µ" = µ
!2 (D) =1
n!1"i
(xi! x)2
!!2" = !2
4
Introduction to Bayesian Inference
Estimators: The Dirty Details
!!" =Z
dD p(D|")!(D)
5
Introduction to Bayesian Inference
Estimators: The Dirty Details
!!" =Z
dD p(D|")!(D)
var(!) = !!2"D #!!"2D
5
Introduction to Bayesian Inference
Maximum Likelihood
! = argmax! p(D|!)
6
Introduction to Bayesian Inference
Maximum Likelihood: χ2
p(D|!) = "i
N (xi|µi,#i)
7
Introduction to Bayesian Inference
Maximum Likelihood: χ2
p(D|!) = "i
N (xi|µi,#i)
p(D|!) " exp
!!1
2#i
"xi!µi
$i
#2$
7
Introduction to Bayesian Inference
Maximum Likelihood: χ2
p(D|!) = "i
N (xi|µi,#i)
p(D|!) " exp
!!1
2#i
"xi!µi
$i
#2$
p(D|!) " exp!!1
2#2
"
7
Introduction to Bayesian Inference
Maximum Likelihood: χ2
p(D|!) = "i
N (xi|µi,#i)
p(D|!) " exp
!!1
2#i
"xi!µi
$i
#2$
p(D|!) " exp!!1
2#2
"
argmax! p(D|!)! argmin
! "2
7
Introduction to Bayesian Inference
Maximum Likelihood: Gaussian
L = !i
1!2"#2
exp
!"(xi"µ)2
2#2
"=
#2"#2$"m
2 exp
!"$i (xi"µ)2
2#2
"
8
Introduction to Bayesian Inference
Maximum Likelihood: Gaussian
L = !i
1!2"#2
exp
!"(xi"µ)2
2#2
"=
#2"#2$"m
2 exp
!"$i (xi"µ)2
2#2
"
!L!µ
=!2"#2"!m
2 exp
#!$i (xi!µ)2
2#2
$$i (xi!µ)
#2
8
Introduction to Bayesian Inference
Maximum Likelihood: Gaussian
0 = !i (xi!µML)"2
ML= !i xi!mµML
"2ML
L = !i
1!2"#2
exp
!"(xi"µ)2
2#2
"=
#2"#2$"m
2 exp
!"$i (xi"µ)2
2#2
"
!L!µ
=!2"#2"!m
2 exp
#!$i (xi!µ)2
2#2
$$i (xi!µ)
#2
8
Introduction to Bayesian Inference
Maximum Likelihood: Gaussian
0 = !i (xi!µML)"2
ML= !i xi!mµML
"2ML
µML = !i xi
m
L = !i
1!2"#2
exp
!"(xi"µ)2
2#2
"=
#2"#2$"m
2 exp
!"$i (xi"µ)2
2#2
"
!L!µ
=!2"#2"!m
2 exp
#!$i (xi!µ)2
2#2
$$i (xi!µ)
#2
8
Introduction to Bayesian Inference
Maximum Likelihood: Gaussian
L = !i
1!2"#2
exp
!"(xi"µ)2
2#2
"=
#2"#2$"m
2 exp
!"$i (xi"µ)2
2#2
"
9
Introduction to Bayesian Inference
Maximum Likelihood: Gaussian
!L!"
=!2#"2"!m
2 exp
#!$i (xi!µ)2
2"2
$1"
%!m+ $i (xi!µ)2
"2
&
L = !i
1!2"#2
exp
!"(xi"µ)2
2#2
"=
#2"#2$"m
2 exp
!"$i (xi"µ)2
2#2
"
9
Introduction to Bayesian Inference
Maximum Likelihood: Gaussian
!L!"
=!2#"2"!m
2 exp
#!$i (xi!µ)2
2"2
$1"
%!m+ $i (xi!µ)2
"2
&
0 = !m+ !i (xi!µML)2
"2ML
= !m+ !i (xi!"x#)2
"2ML
L = !i
1!2"#2
exp
!"(xi"µ)2
2#2
"=
#2"#2$"m
2 exp
!"$i (xi"µ)2
2#2
"
9
Introduction to Bayesian Inference
Maximum Likelihood: Gaussian
!L!"
=!2#"2"!m
2 exp
#!$i (xi!µ)2
2"2
$1"
%!m+ $i (xi!µ)2
"2
&
0 = !m+ !i (xi!µML)2
"2ML
= !m+ !i (xi!"x#)2
"2ML
!2ML = "i (xi!"x#)2
m
L = !i
1!2"#2
exp
!"(xi"µ)2
2#2
"=
#2"#2$"m
2 exp
!"$i (xi"µ)2
2#2
"
9
Introduction to Bayesian Inference
Multimodal Distributions
10
Introduction to Bayesian Inference
A New Approach
p(!|D) =p(D,!)
p(D)
11
Introduction to Bayesian Inference
A New Approach
p(!|D) =p(D,!)
p(D)
11
Introduction to Bayesian Inference
A New Approach
p(!|D) =p(D,!)
p(D)
p(!|D) =p(D|!) p(!)
p(D)
11
Introduction to Bayesian Inference
Probability As Frequency
P(xi) =N (xi)
N! !(xi)
"
12
Introduction to Bayesian Inference
Probability In Science
What is the probability that...
13
Introduction to Bayesian Inference
Probability In Science
‣ gravitational waves exist?
What is the probability that...
13
Introduction to Bayesian Inference
Probability In Science
‣ gravitational waves exist?
What is the probability that...
‣ the Higgs will be found at the LHC?
13
Introduction to Bayesian Inference
Probability In Science
‣ gravitational waves exist?
‣ dark matter is a composite particle?
What is the probability that...
‣ the Higgs will be found at the LHC?
13
Introduction to Bayesian Inference
Probability In Science
‣ any of us graduate?
‣ gravitational waves exist?
‣ dark matter is a composite particle?
What is the probability that...
‣ the Higgs will be found at the LHC?
13
Introduction to Bayesian Inference
The Cox Axioms
14
Introduction to Bayesian Inference
The Cox Axioms
One : P(A) > P(B) > P(C)! P(A) > P(C)
14
Introduction to Bayesian Inference
The Cox Axioms
Two : P!A|B
"= f (P(A|B))
One : P(A) > P(B) > P(C)! P(A) > P(C)
14
Introduction to Bayesian Inference
The Cox Axioms
Two : P!A|B
"= f (P(A|B))
Three : P(A1A2|B) = g(P(A1|B) ,P(A2|A1B))
One : P(A) > P(B) > P(C)! P(A) > P(C)
14
Introduction to Bayesian Inference
Bayes Rule
p(!|D) =p(D|!) p(!)
p(D)
15
Introduction to Bayesian Inference
Bayes Rule
p(!|D) =p(D|!) p(!)
p(D)
15
Introduction to Bayesian Inference
Bayes Rule
p(!|D) =p(D|!) p(!)
p(D)
15
Introduction to Bayesian Inference
Bayes Rule
p(!|D) =p(D|!) p(!)
p(D)
15
Introduction to Bayesian Inference
Bayes Rule
p(!|D) =p(D|!) p(!)
p(D)
15
Introduction to Bayesian Inference
Make Inferences, Not War
p(!) . . .
16
Introduction to Bayesian Inference
Make Inferences, Not War
‣ But frequentists don’t have to make assumptions!
17
Introduction to Bayesian Inference
Bayesian Example
p(x|!) = !exp [!!x]
18
Introduction to Bayesian Inference
Bayesian Example
p(!|x1, . . . ,xm) =p(x1, . . . ,xm|!) p(!)
p(x1, . . . ,xm)=
!m exp [!!"i xi] p(!)p(x1, . . . ,xm)
p(x|!) = !exp [!!x]
18
Introduction to Bayesian Inference
Bayesian Example
p(!|x1, . . . ,xm) =p(x1, . . . ,xm|!) p(!)
p(x1, . . . ,xm)=
!m exp [!!"i xi] p(!)p(x1, . . . ,xm)
p(x|!) = !exp [!!x]
p(!|x1, . . . ,xm) =!m exp [!!"i xi]1/!
p(x1, . . . ,xm)
18
Introduction to Bayesian Inference
Bayesian Example
p(!|x1, . . . ,xm) =p(x1, . . . ,xm|!) p(!)
p(x1, . . . ,xm)=
!m exp [!!"i xi] p(!)p(x1, . . . ,xm)
p(x|!) = !exp [!!x]
p(!|x1, . . . ,xm) =!m exp [!!"i xi]1/!
p(x1, . . . ,xm)
p(!|x1, . . . ,xm) =("i xi)m
(m!1)!!m!1 exp
!!!"
ixi
"
18
Introduction to Bayesian Inference
Bayesian Example
19
Introduction to Bayesian Inference
Bayesian Example
19
Introduction to Bayesian Inference
Bayesian Example
19
Introduction to Bayesian Inference
Bayesian Example
19
Introduction to Bayesian Inference
Bayesian Example
19
Introduction to Bayesian Inference
Bayesian Example
19
Introduction to Bayesian Inference
Bayesian Example
19
Introduction to Bayesian Inference
Marginalization
p(!|D,") =p(D|!,") p(!) p(")
p(D)
20
Introduction to Bayesian Inference
Marginalization
p(!|D,") =p(D|!,") p(!) p(")
p(D)
p(!|D) =Z
d" p(!|D,")
20
Introduction to Bayesian Inference
Model Comparison
21
Introduction to Bayesian Inference
References
‣ Information Theory, Inference, and Learning Algorithms,
MacKay, http://www.inference.phy.cam.ac.uk/mackay/itila/
‣ Data Analysis,
Sivia with Skilling, Oxford 2006
‣ Lectures on Probability, Entropy, and Statistical Physics,
Caticha, http://arxiv.org/abs/0808.0012v1
‣ Probability Theory: The Logic of Science,
Jaynes, Cambridge 2003
22
Introduction to Bayesian Inference
Backups
23
Introduction to Bayesian Inference
A Consistent Decision Theory
R(!) =Z
dD L(!,") p(D|!)
24
Introduction to Bayesian Inference
A Consistent Decision Theory
R(!) =Z
dD L(!,") p(D|!)
!R" =Z
d!g(!)R(!)
24
Introduction to Bayesian Inference
A Consistent Decision Theory
R(!) =Z
dD L(!,") p(D|!)
!R" =Z
d!g(!)R(!)
L(!,") = (!!")2" " =R
d!! p(D|!)g(!)Rd! p(D|!)g(!)
24
Introduction to Bayesian Inference
A Consistent Decision Theory
R(!) =Z
dD L(!,") p(D|!)
!R" =Z
d!g(!)R(!)
L(!,") = (!!")2" " =R
d!! p(D|!)g(!)Rd! p(D|!)g(!)
24
Introduction to Bayesian Inference
Backup: Priors
‣ What does it mean to be ignorant?
25
Introduction to Bayesian Inference
Choosing Priors
‣ Transformation Invariance
‣ Translation Invariant: Uniform Prior
‣ Scale Invariant: Jeffries Prior
‣ Maximum Entropy
‣ Marginalization
‣ Hierarchal Models
26
Introduction to Bayesian Inference
Unnormalizable Priors?
p(!) = 1/",!"/2 < ! < "/20 , else
lim!!"
p(#) =?
27