edinburgh, bayes-250
DESCRIPTION
Slides of my talk at Bayes-250 in Edinburgh, Sept. 6, 2011TRANSCRIPT
![Page 1: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/1.jpg)
ABC Methods for Bayesian Model Choice
ABC Methods for Bayesian Model Choice
Christian P. Robert
Universite Paris-Dauphine, IuF, & CRESThttp://www.ceremade.dauphine.fr/~xian
Bayes-250, Edinburgh, September 6, 2011
![Page 2: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/2.jpg)
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Approximate Bayesian computation
Approximate Bayesian computation
ABC for model choice
Gibbs random fields
Generic ABC model choice
![Page 3: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/3.jpg)
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Regular Bayesian computation issues
When faced with a non-standard posterior distribution
π(θ|y) ∝ π(θ)L(θ|y)
the standard solution is to use simulation (Monte Carlo) toproduce a sample
θ1, . . . , θT
from π(θ|y) (or approximately by Markov chain Monte Carlomethods)
[Robert & Casella, 2004]
![Page 4: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/4.jpg)
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Untractable likelihoods
Cases when the likelihood function f(y|θ) is unavailable and whenthe completion step
f(y|θ) =∫
Zf(y, z|θ) dz
is impossible or too costly because of the dimension of zc© MCMC cannot be implemented!
![Page 5: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/5.jpg)
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Untractable likelihoods
c© MCMC cannot be implemented!
![Page 6: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/6.jpg)
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
The ABC method
Bayesian setting: target is π(θ)f(x|θ)
When likelihood f(x|θ) not in closed form, likelihood-free rejectiontechnique:
ABC algorithm
For an observation y ∼ f(y|θ), under the prior π(θ), keep jointlysimulating
θ′ ∼ π(θ) , z ∼ f(z|θ′) ,
until the auxiliary variable z is equal to the observed value, z = y.
[Tavare et al., 1997]
![Page 7: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/7.jpg)
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
The ABC method
Bayesian setting: target is π(θ)f(x|θ)When likelihood f(x|θ) not in closed form, likelihood-free rejectiontechnique:
ABC algorithm
For an observation y ∼ f(y|θ), under the prior π(θ), keep jointlysimulating
θ′ ∼ π(θ) , z ∼ f(z|θ′) ,
until the auxiliary variable z is equal to the observed value, z = y.
[Tavare et al., 1997]
![Page 8: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/8.jpg)
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
The ABC method
Bayesian setting: target is π(θ)f(x|θ)When likelihood f(x|θ) not in closed form, likelihood-free rejectiontechnique:
ABC algorithm
For an observation y ∼ f(y|θ), under the prior π(θ), keep jointlysimulating
θ′ ∼ π(θ) , z ∼ f(z|θ′) ,
until the auxiliary variable z is equal to the observed value, z = y.
[Tavare et al., 1997]
![Page 9: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/9.jpg)
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
A as approximative
When y is a continuous random variable, equality z = y is replacedwith a tolerance condition,
%(y, z) ≤ ε
where % is a distance
Output distributed from
π(θ)Pθ{%(y, z) < ε} ∝ π(θ|%(y, z) < ε)
![Page 10: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/10.jpg)
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
A as approximative
When y is a continuous random variable, equality z = y is replacedwith a tolerance condition,
%(y, z) ≤ ε
where % is a distanceOutput distributed from
π(θ)Pθ{%(y, z) < ε} ∝ π(θ|%(y, z) < ε)
![Page 11: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/11.jpg)
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
ABC algorithm
Algorithm 1 Likelihood-free rejection sampler
for i = 1 to N dorepeat
generate θ′ from the prior distribution π(·)generate z from the likelihood f(·|θ′)
until ρ{η(z), η(y)} ≤ εset θi = θ′
end for
where η(y) defines a (maybe in-sufficient) statistic
![Page 12: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/12.jpg)
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Output
The likelihood-free algorithm samples from the marginal in z of:
πε(θ, z|y) =π(θ)f(z|θ)IAε,y(z)∫
Aε,y×Θ π(θ)f(z|θ)dzdθ,
where Aε,y = {z ∈ D|ρ(η(z), η(y)) < ε}.
The idea behind ABC is that the summary statistics coupled with asmall tolerance should provide a good approximation of theposterior distribution:
πε(θ|y) =∫πε(θ, z|y)dz ≈ π(θ|y) .
![Page 13: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/13.jpg)
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Output
The likelihood-free algorithm samples from the marginal in z of:
πε(θ, z|y) =π(θ)f(z|θ)IAε,y(z)∫
Aε,y×Θ π(θ)f(z|θ)dzdθ,
where Aε,y = {z ∈ D|ρ(η(z), η(y)) < ε}.
The idea behind ABC is that the summary statistics coupled with asmall tolerance should provide a good approximation of theposterior distribution:
πε(θ|y) =∫πε(θ, z|y)dz ≈ π(θ|y) .
![Page 14: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/14.jpg)
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
MA example
Consider the MA(q) model
xt = εt +
q∑i=1
ϑiεt−i
Simple prior: uniform prior over the identifiability zone, e.g.triangle for MA(2)
![Page 15: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/15.jpg)
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
MA example (2)
ABC algorithm thus made of
1. picking a new value (ϑ1, ϑ2) in the triangle
2. generating an iid sequence (εt)−q<t≤T
3. producing a simulated series (x′t)1≤t≤T
Distance: basic distance between the series
ρ((x′t)1≤t≤T , (xt)1≤t≤T ) =T∑t=1
(xt − x′t)2
or between summary statistics like the first q autocorrelations
τj =T∑
t=j+1
xtxt−j
![Page 16: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/16.jpg)
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
MA example (2)
ABC algorithm thus made of
1. picking a new value (ϑ1, ϑ2) in the triangle
2. generating an iid sequence (εt)−q<t≤T
3. producing a simulated series (x′t)1≤t≤T
Distance: basic distance between the series
ρ((x′t)1≤t≤T , (xt)1≤t≤T ) =T∑t=1
(xt − x′t)2
or between summary statistics like the first q autocorrelations
τj =
T∑t=j+1
xtxt−j
![Page 17: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/17.jpg)
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Comparison of distance impact
Evaluation of the tolerance on the ABC sample against bothdistances (ε = 100%, 10%, 1%, 0.1%) for an MA(2) model
![Page 18: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/18.jpg)
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Comparison of distance impact
0.0 0.2 0.4 0.6 0.8
01
23
4
θ1
−2.0 −1.0 0.0 0.5 1.0 1.5
0.00.5
1.01.5
θ2
Evaluation of the tolerance on the ABC sample against bothdistances (ε = 100%, 10%, 1%, 0.1%) for an MA(2) model
![Page 19: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/19.jpg)
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Comparison of distance impact
0.0 0.2 0.4 0.6 0.8
01
23
4
θ1
−2.0 −1.0 0.0 0.5 1.0 1.5
0.00.5
1.01.5
θ2
Evaluation of the tolerance on the ABC sample against bothdistances (ε = 100%, 10%, 1%, 0.1%) for an MA(2) model
![Page 20: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/20.jpg)
ABC Methods for Bayesian Model Choice
ABC for model choice
ABC for model choice
Approximate Bayesian computation
ABC for model choice
Gibbs random fields
Generic ABC model choice
![Page 21: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/21.jpg)
ABC Methods for Bayesian Model Choice
ABC for model choice
Bayesian model choice
PrincipleSeveral models M1,M2, . . . are considered simultaneously fordataset y and model index M central to inference.Use of a prior π(M = m), plus a prior distribution on theparameter conditional on the value m of the model index, πm(θm)Goal is to derive the posterior distribution of M, a challengingcomputational target when models are complex.
![Page 22: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/22.jpg)
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC for model choice
Algorithm 2 Likelihood-free model choice sampler (ABC-MC)
for t = 1 to T dorepeat
Generate m from the prior π(M = m)Generate θm from the prior πm(θm)Generate z from the model fm(z|θm)
until ρ{η(z), η(y)} < εSet m(t) = m and θ(t) = θm
end for
[Toni, Welch, Strelkowa, Ipsen & Stumpf, 2009]
![Page 23: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/23.jpg)
ABC Methods for Bayesian Model Choice
ABC for model choice
ABC estimates
Posterior probability π(M = m|y) approximated by the frequencyof acceptances from model m
1
T
T∑t=1
Im(t)=m .
Early issues with implementation:
I should tolerances ε be the same for all models?
I should summary statistics vary across models? incl. theirdimension?
I should the distance measure ρ vary across models?
Extension to a weighted polychotomous logistic regression estimateof π(M = m|y), with non-parametric kernel weights
[Cornuet et al., DIYABC, 2009]
![Page 24: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/24.jpg)
ABC Methods for Bayesian Model Choice
ABC for model choice
ABC estimates
Posterior probability π(M = m|y) approximated by the frequencyof acceptances from model m
1
T
T∑t=1
Im(t)=m .
Early issues with implementation:
I ε then needs to become part of the model
I Varying statistics incompatible with Bayesian model choiceproper
I ρ then part of the model
Extension to a weighted polychotomous logistic regression estimateof π(M = m|y), with non-parametric kernel weights
[Cornuet et al., DIYABC, 2009]
![Page 25: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/25.jpg)
ABC Methods for Bayesian Model Choice
ABC for model choice
The great ABC controversy
On-going controvery in phylogeographic genetics about the validityof using ABC for testing
Against: Templeton, 2008,2009, 2010a, 2010b, 2010c, &tcargues that nested hypothesescannot have higher probabilitiesthan nesting hypotheses (!)
![Page 26: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/26.jpg)
ABC Methods for Bayesian Model Choice
ABC for model choice
The great ABC controversy
On-going controvery in phylogeographic genetics about the validityof using ABC for testing
Against: Templeton, 2008,2009, 2010a, 2010b, 2010c, &tcargues that nested hypothesescannot have higher probabilitiesthan nesting hypotheses (!)
Replies: Fagundes et al., 2008,Beaumont et al., 2010, Berger etal., 2010, Csillery et al., 2010point out that the criticisms areaddressed at [Bayesian]model-based inference and havenothing to do with ABC...
![Page 27: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/27.jpg)
ABC Methods for Bayesian Model Choice
Gibbs random fields
Potts model
Potts model∑c∈C Vc(y) is of the form∑
c∈C
Vc(y) = θS(y) = θ∑l∼i
δyl=yi
where l∼i denotes a neighbourhood structure
In most realistic settings, summation
Zθ =∑x∈X
exp{θTS(x)}
involves too many terms to be manageable and numericalapproximations cannot always be trusted
[Cucala et al., 2009]
![Page 28: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/28.jpg)
ABC Methods for Bayesian Model Choice
Gibbs random fields
Potts model
Potts model∑c∈C Vc(y) is of the form∑
c∈C
Vc(y) = θS(y) = θ∑l∼i
δyl=yi
where l∼i denotes a neighbourhood structure
In most realistic settings, summation
Zθ =∑x∈X
exp{θTS(x)}
involves too many terms to be manageable and numericalapproximations cannot always be trusted
[Cucala et al., 2009]
![Page 29: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/29.jpg)
ABC Methods for Bayesian Model Choice
Gibbs random fields
Neighbourhood relations
SetupChoice to be made between M neighbourhood relations
im∼ i′ (0 ≤ m ≤M − 1)
withSm(x) =
∑im∼i′
I{xi=xi′}
driven by the posterior probabilities of the models.
![Page 30: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/30.jpg)
ABC Methods for Bayesian Model Choice
Gibbs random fields
Model index
Computational target:
P(M = m|x) ∝∫
Θm
fm(x|θm)πm(θm) dθm π(M = m)
If S(x) sufficient statistic for the joint parameters(M, θ0, . . . , θM−1),
P(M = m|x) = P(M = m|S(x)) .
![Page 31: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/31.jpg)
ABC Methods for Bayesian Model Choice
Gibbs random fields
Model index
Computational target:
P(M = m|x) ∝∫
Θm
fm(x|θm)πm(θm) dθm π(M = m)
If S(x) sufficient statistic for the joint parameters(M, θ0, . . . , θM−1),
P(M = m|x) = P(M = m|S(x)) .
![Page 32: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/32.jpg)
ABC Methods for Bayesian Model Choice
Gibbs random fields
Sufficient statistics in Gibbs random fields
Each model m has its own sufficient statistic Sm(·) andS(·) = (S0(·), . . . , SM−1(·)) is also (model-)sufficient.Explanation: For Gibbs random fields,
x|M = m ∼ fm(x|θm) = f1m(x|S(x))f2
m(S(x)|θm)
=1
n(S(x))f2m(S(x)|θm)
wheren(S(x)) = ] {x ∈ X : S(x) = S(x)}
c© S(x) is therefore also sufficient for the joint parameters
![Page 33: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/33.jpg)
ABC Methods for Bayesian Model Choice
Gibbs random fields
Sufficient statistics in Gibbs random fields
Each model m has its own sufficient statistic Sm(·) andS(·) = (S0(·), . . . , SM−1(·)) is also (model-)sufficient.
Explanation: For Gibbs random fields,
x|M = m ∼ fm(x|θm) = f1m(x|S(x))f2
m(S(x)|θm)
=1
n(S(x))f2m(S(x)|θm)
wheren(S(x)) = ] {x ∈ X : S(x) = S(x)}
c© S(x) is therefore also sufficient for the joint parameters
![Page 34: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/34.jpg)
ABC Methods for Bayesian Model Choice
Gibbs random fields
Sufficient statistics in Gibbs random fields
Each model m has its own sufficient statistic Sm(·) andS(·) = (S0(·), . . . , SM−1(·)) is also (model-)sufficient.Explanation: For Gibbs random fields,
x|M = m ∼ fm(x|θm) = f1m(x|S(x))f2
m(S(x)|θm)
=1
n(S(x))f2m(S(x)|θm)
wheren(S(x)) = ] {x ∈ X : S(x) = S(x)}
c© S(x) is therefore also sufficient for the joint parameters
![Page 35: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/35.jpg)
ABC Methods for Bayesian Model Choice
Gibbs random fields
Toy example
iid Bernoulli model versus two-state first-order Markov chain, i.e.
f0(x|θ0) = exp
(θ0
n∑i=1
I{xi=1}
)/{1 + exp(θ0)}n ,
versus
f1(x|θ1) =1
2exp
(θ1
n∑i=2
I{xi=xi−1}
)/{1 + exp(θ1)}n−1 ,
with priors θ0 ∼ U(−5, 5) and θ1 ∼ U(0, 6) (inspired by “phasetransition” boundaries).
![Page 36: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/36.jpg)
ABC Methods for Bayesian Model Choice
Gibbs random fields
Toy example (2)
−40−20
010
−5 0 5
BF01
BF01
−40−20
010
−10 −5 0 5 10
BF01
BF01
(left) Comparison of the true BFm0/m1(x0) with BFm0/m1
(x0)(in logs) over 2, 000 simulations and 4.106 proposals from theprior. (right) Same when using tolerance ε corresponding to the1% quantile on the distances.
![Page 37: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/37.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
Back to sufficiency
‘Sufficient statistics for individual models are unlikely tobe very informative for the model probability. This isalready well known and understood by the ABC-usercommunity.’
[Scott Sisson, Jan. 31, 2011, ’Og]
If η1(x) sufficient statistic for model m = 1 and parameter θ1 andη2(x) sufficient statistic for model m = 2 and parameter θ2,(η1(x), η2(x)) is not always sufficient for (m, θm)
c© Potential loss of information at the testing level
![Page 38: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/38.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
Back to sufficiency
‘Sufficient statistics for individual models are unlikely tobe very informative for the model probability. This isalready well known and understood by the ABC-usercommunity.’
[Scott Sisson, Jan. 31, 2011, ’Og]
If η1(x) sufficient statistic for model m = 1 and parameter θ1 andη2(x) sufficient statistic for model m = 2 and parameter θ2,(η1(x), η2(x)) is not always sufficient for (m, θm)
c© Potential loss of information at the testing level
![Page 39: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/39.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
Back to sufficiency
‘Sufficient statistics for individual models are unlikely tobe very informative for the model probability. This isalready well known and understood by the ABC-usercommunity.’
[Scott Sisson, Jan. 31, 2011, ’Og]
If η1(x) sufficient statistic for model m = 1 and parameter θ1 andη2(x) sufficient statistic for model m = 2 and parameter θ2,(η1(x), η2(x)) is not always sufficient for (m, θm)
c© Potential loss of information at the testing level
![Page 40: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/40.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
Limiting behaviour of B12 (T →∞)
ABC approximation
B12(y) =
∑Tt=1 Imt=1 Iρ{η(zt),η(y)}≤ε∑Tt=1 Imt=2 Iρ{η(zt),η(y)}≤ε
,
where the (mt, zt)’s are simulated from the (joint) prior
As T go to infinity, limit
Bε12(y) =
∫Iρ{η(z),η(y)}≤επ1(θ1)f1(z|θ1) dz dθ1∫Iρ{η(z),η(y)}≤επ2(θ2)f2(z|θ2) dz dθ2
=
∫Iρ{η,η(y)}≤επ1(θ1)f
η1 (η|θ1) dη dθ1∫
Iρ{η,η(y)}≤επ2(θ2)fη2 (η|θ2) dη dθ2
,
where fη1 (η|θ1) and fη2 (η|θ2) distributions of η(z)
![Page 41: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/41.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
Limiting behaviour of B12 (T →∞)
ABC approximation
B12(y) =
∑Tt=1 Imt=1 Iρ{η(zt),η(y)}≤ε∑Tt=1 Imt=2 Iρ{η(zt),η(y)}≤ε
,
where the (mt, zt)’s are simulated from the (joint) priorAs T go to infinity, limit
Bε12(y) =
∫Iρ{η(z),η(y)}≤επ1(θ1)f1(z|θ1) dz dθ1∫Iρ{η(z),η(y)}≤επ2(θ2)f2(z|θ2) dz dθ2
=
∫Iρ{η,η(y)}≤επ1(θ1)f
η1 (η|θ1) dη dθ1∫
Iρ{η,η(y)}≤επ2(θ2)fη2 (η|θ2) dη dθ2
,
where fη1 (η|θ1) and fη2 (η|θ2) distributions of η(z)
![Page 42: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/42.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
Limiting behaviour of B12 (ε→ 0)
When ε goes to zero,
Bη12(y) =
∫π1(θ1)f
η1 (η(y)|θ1) dθ1∫
π2(θ2)fη2 (η(y)|θ2) dθ2
Bayes factor based on the sole observation of η(y)
![Page 43: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/43.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
Limiting behaviour of B12 (ε→ 0)
When ε goes to zero,
Bη12(y) =
∫π1(θ1)f
η1 (η(y)|θ1) dθ1∫
π2(θ2)fη2 (η(y)|θ2) dθ2
Bayes factor based on the sole observation of η(y)
![Page 44: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/44.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
Limiting behaviour of B12 (under sufficiency)
If η(y) sufficient statistic in both models,
fi(y|θi) = gi(y)fηi (η(y)|θi)
Thus
B12(y) =
∫Θ1π(θ1)g1(y)f
η1 (η(y)|θ1) dθ1∫
Θ2π(θ2)g2(y)f
η2 (η(y)|θ2) dθ2
=g1(y)
∫π1(θ1)f
η1 (η(y)|θ1) dθ1
g2(y)∫π2(θ2)f
η2 (η(y)|θ2) dθ2
=g1(y)
g2(y)Bη
12(y) .
[Didelot, Everitt, Johansen & Lawson, 2011]
c© No discrepancy only when cross-model sufficiency
![Page 45: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/45.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
Limiting behaviour of B12 (under sufficiency)
If η(y) sufficient statistic in both models,
fi(y|θi) = gi(y)fηi (η(y)|θi)
Thus
B12(y) =
∫Θ1π(θ1)g1(y)f
η1 (η(y)|θ1) dθ1∫
Θ2π(θ2)g2(y)f
η2 (η(y)|θ2) dθ2
=g1(y)
∫π1(θ1)f
η1 (η(y)|θ1) dθ1
g2(y)∫π2(θ2)f
η2 (η(y)|θ2) dθ2
=g1(y)
g2(y)Bη
12(y) .
[Didelot, Everitt, Johansen & Lawson, 2011]
c© No discrepancy only when cross-model sufficiency
![Page 46: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/46.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
Poisson/geometric example
Samplex = (x1, . . . , xn)
from either a Poisson P(λ) or from a geometric G(p)Sum
S =
n∑i=1
xi = η(x)
sufficient statistic for either model but not simultaneously
Discrepancy ratio
g1(x)
g2(x)=S!n−S/
∏i xi!
1/(
n+S−1S
)
![Page 47: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/47.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
Poisson/geometric discrepancy
Range of B12(x) versus Bη12(x): The values produced have
nothing in common.
![Page 48: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/48.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
Formal recovery
Creating an encompassing exponential family
f(x|θ1, θ2, α1, α2) ∝ exp{θT1 η1(x)+ θT
1 η1(x)+α1t1(x)+α2t2(x)}
leads to a sufficient statistic (η1(x), η2(x), t1(x), t2(x))[Didelot, Everitt, Johansen & Lawson, 2011]
![Page 49: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/49.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
Formal recovery
Creating an encompassing exponential family
f(x|θ1, θ2, α1, α2) ∝ exp{θT1 η1(x)+ θT
1 η1(x)+α1t1(x)+α2t2(x)}
leads to a sufficient statistic (η1(x), η2(x), t1(x), t2(x))[Didelot, Everitt, Johansen & Lawson, 2011]
In the Poisson/geometric case, if∏i xi! is added to S, no
discrepancy
![Page 50: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/50.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
Formal recovery
Creating an encompassing exponential family
f(x|θ1, θ2, α1, α2) ∝ exp{θT1 η1(x)+ θT
1 η1(x)+α1t1(x)+α2t2(x)}
leads to a sufficient statistic (η1(x), η2(x), t1(x), t2(x))[Didelot, Everitt, Johansen & Lawson, 2011]
Only applies in genuine sufficiency settings...
c© Inability to evaluate loss brought by summary statistics
![Page 51: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/51.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
The Pitman–Koopman lemma
Efficient sufficiency is not such a common occurrence:
Lemma
A necessary and sufficient condition for the existence of a sufficientstatistic with fixed dimension whatever the sample size is that thesampling distribution belongs to an exponential family.
[Pitman, 1933; Koopman, 1933]
Provision of fixed support (consider U(0, θ) counterexample)
![Page 52: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/52.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
The Pitman–Koopman lemma
Efficient sufficiency is not such a common occurrence:
Lemma
A necessary and sufficient condition for the existence of a sufficientstatistic with fixed dimension whatever the sample size is that thesampling distribution belongs to an exponential family.
[Pitman, 1933; Koopman, 1933]
Provision of fixed support (consider U(0, θ) counterexample)
![Page 53: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/53.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
Meaning of the ABC-Bayes factor
‘This is also why focus on model discrimination typically(...) proceeds by (...) accepting that the Bayes Factorthat one obtains is only derived from the summarystatistics and may in no way correspond to that of thefull model.’
[Scott Sisson, Jan. 31, 2011, ’Og]
In the Poisson/geometric case, if E[yi] = θ0 > 0,
limn→∞
Bη12(y) =
(θ0 + 1)2
θ0e−θ0
![Page 54: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/54.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
Meaning of the ABC-Bayes factor
‘This is also why focus on model discrimination typically(...) proceeds by (...) accepting that the Bayes Factorthat one obtains is only derived from the summarystatistics and may in no way correspond to that of thefull model.’
[Scott Sisson, Jan. 31, 2011, ’Og]
In the Poisson/geometric case, if E[yi] = θ0 > 0,
limn→∞
Bη12(y) =
(θ0 + 1)2
θ0e−θ0
![Page 55: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/55.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
MA example
1 2
0.0
0.1
0.2
0.3
0.4
0.5
0.6
1 2
0.0
0.1
0.2
0.3
0.4
0.5
0.6
1 2
0.0
0.1
0.2
0.3
0.4
0.5
0.6
1 2
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Evolution [against ε] of ABC Bayes factor, in terms of frequencies ofvisits to models MA(1) (left) and MA(2) (right) when ε equal to10, 1, .1, .01% quantiles on insufficient autocovariance distances. Sampleof 50 points from a MA(2) with θ1 = 0.6, θ2 = 0.2. True Bayes factorequal to 17.71.
![Page 56: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/56.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
MA example
1 2
0.0
0.2
0.4
0.6
1 2
0.0
0.2
0.4
0.6
1 2
0.0
0.2
0.4
0.6
1 2
0.0
0.2
0.4
0.6
0.8
Evolution [against ε] of ABC Bayes factor, in terms of frequencies ofvisits to models MA(1) (left) and MA(2) (right) when ε equal to10, 1, .1, .01% quantiles on insufficient autocovariance distances. Sampleof 50 points from a MA(1) model with θ1 = 0.6. True Bayes factor B21
equal to .004.
![Page 57: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/57.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
Further comments
‘There should be the possibility that for the same model,but different (non-minimal) [summary] statistics (sodifferent η’s: η1 and η∗1) the ratio of evidences may nolonger be equal to one.’
[Michael Stumpf, Jan. 28, 2011, ’Og]
Using different summary statistics [on different models] mayindicate the loss of information brought by each set but agreementdoes not lead to trustworthy approximations.
![Page 58: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/58.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
A population genetics evaluation
Population genetics example with
I 3 populations
I 2 scenari
I 15 individuals
I 5 loci
I single mutation parameter
I 24 summary statistics
I 2 million ABC proposal
I importance [tree] sampling alternative
![Page 59: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/59.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
A population genetics evaluation
Population genetics example with
I 3 populations
I 2 scenari
I 15 individuals
I 5 loci
I single mutation parameter
I 24 summary statistics
I 2 million ABC proposal
I importance [tree] sampling alternative
![Page 60: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/60.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
Stability of importance sampling
●
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
●
0.0
0.2
0.4
0.6
0.8
1.0
●
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
![Page 61: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/61.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
Comparison with ABC
Use of 24 summary statistics and DIY-ABC logistic correction
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
importance sampling
AB
C d
irect
and
logi
stic
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
![Page 62: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/62.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
Comparison with ABC
Use of 15 summary statistics and DIY-ABC logistic correction
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
● ●
●
●
●●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
−4 −2 0 2 4 6
−4
−2
02
46
importance sampling
AB
C d
irect
![Page 63: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/63.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
Comparison with ABC
Use of 15 summary statistics and DIY-ABC logistic correction
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
● ●
●
●
●●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
−4 −2 0 2 4 6
−4
−2
02
46
importance sampling
AB
C d
irect
and
logi
stic
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
![Page 64: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/64.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
Comparison with ABC
Use of 15 summary statistics and DIY-ABC logistic correction
●●
●●
●●●
●
●●●●●
●●●
●●
●●
●
●●
●●
●●●
●
●
●
●●●●
●
●●
●●●●
●
●
●
●
●
●●●●
●●
●●●●●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
0 20 40 60 80 100
−2
−1
01
2
index
log−
ratio
●
●
●
●●●●
●●●●●●
●
●●
●●
●
●
●
●
●
●
●
●●●
●
●
●●●●●
●
●●●●●
●●●●
●
●
●●
●
●
●
●
●●●●
●
●
●●●●
●
●
●
●
●●
●●
●
●●
●
●
●●●●●
●
●
●
●
●
●●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●●●●
●●●●●●
●
●●
●●
●
●
●
●
●
●
●
●●●
●
●
●●●●●
●
●●●●●
●●●●
●
●
●●
●
●
●
●
●●●●
●
●
●●●●
●
●
●
●
●●
●●
●
●●
●
●
●●●●●
●
●
●
●
●
●●
●
●
●●●●
●
●
●
●
●
●●●
●●
●●●
●
●●●●●
●●●
●●
●●
●
●●
●●
●●●
●
●
●
●●●●
●
●●
●●●●
●
●
●
●
●
●●●●
●●
●●●●●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
![Page 65: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/65.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
A second population genetics experiment
I three populations, two divergent 100 gen. ago
I two scenarios [third pop. recent admixture between first twopop. / diverging from pop. 1 5 gen. ago]
I In scenario 1, admixture rate 0.7 from pop. 1
I 100 datasets with 100 diploid individuals per population, 50independent microsatellite loci.
I Effective population size of 1000 and mutation rates of0.0005.
I 6 parameters: admixture rate (U [0.1, 0.9]), three effectivepopulation sizes (U [200, 2000]), the time of admixture/seconddivergence (U [1, 10]) and time of first divergence (U [50, 500]).
![Page 66: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/66.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
Results
IS algorithm performed with 100 coalescent trees per particle and50,000 particles [12 calendar days using 376 processors]Using ten times as many loci and seven times as many individualsdegrades the confidence in the importance sampling approximationbecause of an increased variability in the likelihood.
●●
●
●●●●●
●
●● ● ●●
●
●
●
●●●●●●
●
●
●
●
●
●●●●● ● ●●●
●●
●
●●●●●
●
●●●●●● ●
●● ●●●●●●● ●●
●
●●●●●●●●●●●●●● ●●●●●●●●●●●●
●●●●●
●
●
●●
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Importance Sampling estimates ofP(M=1|y) with 50,000 particles
Impo
rtan
ce S
ampl
ing
estim
ates
of
P(M
=1|
y) w
ith 1
,000
par
ticle
s
![Page 67: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/67.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
Results
Blurs potential divergence between ABC and genuine posteriorprobabilities because both are overwhelmingly close to one, due tothe high information content of the data.
● ●
●
●●●●●
●
●●● ●●
●
●
●
●●●● ● ●
●
●
●●
●
● ●● ●● ●●●● ●●
●
● ●●●●
●
●●●
●●●●●●●●●●●●●
●●
●
●●●●●●●●●●● ●●● ●●● ●●●●●●●● ●●●● ●●
●
●
●●
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
ABC estimates of P(M=1|y)
Impo
rtan
ce S
ampl
ing
estim
ates
of P
(M=
1|y)
with
50,
000
part
icle
s
![Page 68: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/68.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
The only safe cases???
Besides specific models like Gibbs random fields,
using distances over the data itself escapes the discrepancy...[Toni & Stumpf, 2010;Sousa et al., 2009]
...and so does the use of more informal model fitting measures[Ratmann, Andrieu, Richardson and Wiujf, 2009]
![Page 69: Edinburgh, Bayes-250](https://reader034.vdocuments.net/reader034/viewer/2022052618/554e7ecdb4c905f66a8b5373/html5/thumbnails/69.jpg)
ABC Methods for Bayesian Model Choice
Generic ABC model choice
The only safe cases???
Besides specific models like Gibbs random fields,
using distances over the data itself escapes the discrepancy...[Toni & Stumpf, 2010;Sousa et al., 2009]
...and so does the use of more informal model fitting measures[Ratmann, Andrieu, Richardson and Wiujf, 2009]