bayesian modeling and probabilistic image …kazu/tutorial...1 , 1 arg max ( , (), (),)., t t q t t...
TRANSCRIPT
14 January, 201014 January, 2010 Hokkaido University GCOE Tutorial (SapporoHokkaido University GCOE Tutorial (Sapporo)) 11
ベイジアンモデリングと確率的画像処理ベイジアンモデリングと確率的画像処理Bayesian Modeling andBayesian Modeling and
Probabilistic Image ProcessingProbabilistic Image Processing((付録:付録:AppendixAppendix))
東北大学東北大学 大学院情報科学研究科大学院情報科学研究科
田中田中 和之和之
http://www.smapip.is.tohoku.ac.jp/~kazu/http://www.smapip.is.tohoku.ac.jp/~kazu/
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 2
ガウス分布(正規分布)
( ) ( ) ⎟⎠⎞
⎜⎝⎛ −−= 2
22 21exp
21 μ
σπσρ xx
[ ] μ=XE [ ] 2V σ=X
∫∞+
∞−=⎟
⎠⎞
⎜⎝⎛− πξξ 2
21exp 2 d
平均と分散はガウス積分の公式 (Gaussian Integral Formula) から導かれる
平均μ,分散σ2 のガウス分布 (Gaussian Distribution) の確率密度関数
( )+∞<<∞− xp(x)
μ x0
)0( >σ
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 3
多次元ガウス分布
( )( )
( ) ⎟⎟⎠
⎞⎜⎜⎝
⎛⎟⎟⎠
⎞⎜⎜⎝
⎛−−
−−−= −
Y
XYX y
xyxyx
μμ
μμπ
ρ 1
2,
21exp
det2
1, CC
( )∫ ∫ ∫∞+
∞−
∞+
∞−
∞+
∞−− =⎟
⎠⎞
⎜⎝⎛− CC det2
21exp 1T ddxx πξ
rrrL
d 次元ガウス積分の公式から導かれる
行列 C を正定値の実対称行列として,2次元ガウス分布(Two-Dimensional Gaussian Distribution) の確率密度関数
( )+∞<<∞−+∞<<∞− yx ,
C=⎟⎟⎠
⎞⎜⎜⎝
⎛]V[],Cov[
],Cov[]V[YXY
YXX
一般の次元への拡張も同様
において行列 C が共分散行列になる.
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 4
Probabilistic Image Processing by Bayesian Network
BayesFormulas
Probabilistic image processing is formulated by assuming the degradation process and the prior process of original image and by using the Bayes formula.
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 5
Degraded Image
Statistical Estimation of Hyperparameters
∫ ===== zdzXxXyYyY rrrrrrrrr},|Pr{},|Pr{},|Pr{ γασσα
( )},|Pr{max arg)ˆ,ˆ(
,σασα
σαyY rr
==
xr g
Marginalized with respect to X
}|Pr{ αxX rr= },|Pr{ σxXyY rrrr
== yrOriginal Image
Marginal Likelihood
},|Pr{ σαyY rr=
Hyperparameters α, σ are determined so as to maximize the marginal likelihood Pr{Y=y|α,σ} with respect to α, σ.
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 6
Gaussian Graphical Model(Gauss Markov Random Fields)
( )
( ) ( )
⎟⎠⎞
⎜⎝⎛ −−−=
⎟⎟
⎠
⎞
⎜⎜
⎝
⎛−−−−∝ ∑∑
∈∈
xxyx
xxyx
yxP
Ejiji
Viii
rrrr
rr
CT22
},{
222
21
21exp
21
21exp
,,|
ασ
ασ
σα
( ) ( )( ) ( ) ⎟
⎠⎞
⎜⎝⎛
+−
+= yyyP
Vrrr
CIC
CI
C2
T2 2
1expdet2
det,ασ
αασπ
ασα
( ) yxdyxPxx rrrrrr 12)|(ˆ −+== ∫ CI ασ
Multidimensional Gauss Integral Formulas
( )( )
( )σασασα
,max argˆ,ˆ,
gP r=
Maximum Likelihood Estimation EM Algorithm
),( +∞−∞∈ix
⎪⎩
⎪⎨
⎧∈−∈=
=otherwise,0
},{,1,4
EjiVji
ji C
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 7
Average of Gaussian Graphical Model
0)))((()(21exp)( ||21
2T vL
rrrrrrL =⎟
⎠⎞
⎜⎝⎛ −−−−−∫ ∫ ∫
∞+
∞−
∞+
∞−
∞+
∞− Vdzdzdzzzz μγασμμ CI
( )
( )
y
dzdzdzyzyz
dzdzdzyzyzz
dzdzdzyYzXzXx
V
V
V
r
Lrrrr
L
Lrrrrr
L
Lrrrrr
Lrr
CII
CIICI
CII
CIICI
CII
2
||2122
T
22
||2122
T
22
||21
21exp
2
1exp
},,|Pr{,ˆ
ασ
ασασ
ασσ
ασασ
ασσ
σασα
+=
⎟⎟
⎠
⎞
⎜⎜
⎝
⎛⎟⎠⎞
⎜⎝⎛
+−+⎟
⎠⎞
⎜⎝⎛
+−−
⎟⎟
⎠
⎞
⎜⎜
⎝
⎛⎟⎠⎞
⎜⎝⎛
+−+⎟
⎠⎞
⎜⎝⎛
+−−
=
====
∫ ∫ ∫
∫ ∫ ∫
∫ ∫ ∫
∞+
∞−
∞+
∞−
∞+
∞−
∞+
∞−
∞+
∞−
∞+
∞−
+∞
∞−
+∞
∞−
+∞
∞−
Gaussian Integral formula
Average of the posterior probability can be calculated by using the multi-dimensional Gauss integral Formula
⎪⎩
⎪⎨
⎧∈−∈=
=otherwise,0
},{,1,4
EjiVji
ji C
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 8
Marginal Likelihood of Gaussian Graphical Model
)()2(),,(
}|Pr{},|Pr{},|Pr{PR
2/||2POS
σπσσα
ασσαZ
yZzdzXzXyYyY V
rrrrrrrrrr
====== ∫
AA
det)2()( )(
21exp
||||21
TV
Vdzdzdzzz πμμ =⎟⎠⎞
⎜⎝⎛ −−−∫ ∫ ∫
∞+
∞−
∞+
∞−
∞+
∞−L
rrrrL
( )
( )
⎟⎠⎞
⎜⎝⎛
+−
+=
⎟⎟
⎠
⎞
⎜⎜
⎝
⎛⎟⎠⎞
⎜⎝⎛
+−+⎟
⎠⎞
⎜⎝⎛
+−−×
⎟⎠⎞
⎜⎝⎛
+−=
⎟⎟
⎠
⎞
⎜⎜
⎝
⎛
+−⎟
⎠⎞
⎜⎝⎛
+−+⎟
⎠⎞
⎜⎝⎛
+−−=
∫ ∫ ∫
∫ ∫ ∫
∞+
∞−
∞+
∞−
∞+
∞−
∞+
∞−
∞+
∞−
∞+
∞−
yy
dzdzdzyzyz
yy
dzdzdzyyyzyz
yZ
V
V
V
rr
Lrrrr
L
rr
Lrrrrrr
L
r
CIC
CI
CIICI
CII
CIC
CIC
CIICI
CII
2T
2
||2
||2122
T
22
2T
||212T
22
T
22
POS
21exp
)det()2(
2
1exp
21exp
21
21exp
),,(
ασα
ασπσ
ασασ
ασσ
ασα
ασα
ασασ
ασσ
σα
Gaussian Integral formula
CC
det)2(
21exp)( ||
||||21
TV
VVPR dzdzdzzzZ
απαα =⎟
⎠⎞
⎜⎝⎛ −≡ ∫ ∫ ∫
∞+
∞−
∞+
∞−
∞+
∞−L
rrL
⎪⎩
⎪⎨
⎧∈−∈=
=otherwise,0
},{,1,4
EjiVji
ji C
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 9
Marginal Likelihood in Gaussian Graphical Model
( )( ) ( ) ⎟
⎠⎞
⎜⎝⎛
+−
+== T
22 21exp
det2
det},|Pr{ yyyYV
rrrr
CIC
CI
Cασ
αασπ
ασα
1
2T
2
2
||1Tr
||1
−
⎟⎟⎠
⎞⎜⎜⎝
⎛
++
+= yy
VVrr
CIC
CIC
ασασσα ( )
T22
242
2
2
||1Tr
||1 yy
VVrr
CI
CCI
I
ασ
σαασ
σσ+
++
=
Extremum Conditions for α and σ
( ) ( )( ) ( ) ( ) ( )
1
2T
2
2
11||1
111Tr
||1
−
⎟⎟⎠
⎞⎜⎜⎝
⎛
−−++
−−+
−← y
tty
Vttt
Vt rr
CIC
CIC
σασα
σα
( ) ( )( ) ( )
( ) ( )( ) ( )( )
ytt
ttyVtt
tV
t rr22
242T
2
2
11
11||
111
1Tr||
1
CI
CCI
I
−−+
−−+
−−+
−←
σα
σα
σα
σσ
( )),|Pr{max arg)ˆ,ˆ(
,σασα
σαyYrr
==
( ) ( )( )
( )( ) ( )( ).,,maxarg
1,1
,ttQ
ttσασα
σα
σα←
++
Iterated AlgorithmEM Algorithm
0},|Pr{ ,0},|Pr{ ==∂∂
==∂∂ σα
σσα
αyYyY
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 10
Bayesian Image Analysis by Gaussian Graphical Model
0
0.0002
0.0004
0.0006
0.0008
0.001
0 20 40 60 80 100( )tσ
( )tαyr f̂
r
ytttx rr 12 ))()(()(ˆ −+= CI σα
Iteration Procedure of EM algorithm in Gaussian Graphical Model
EM
f̂r
yr( )
),|(max arg)ˆ,ˆ(,
σασασα
gP=
40=σ
( ) ( )( )( )
( ) ( )( ).,,,maxarg1,1,
yttQtt rσασασασα
←++
0007130ˆ624.37ˆ
.==
ασ
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 11
扱いやすい確率モデルのグラフ表現
扱いやすい確率モデルの数理構造
⎟⎟⎠
⎞⎜⎜⎝
⎛⎟⎟⎠
⎞⎜⎜⎝
⎛⎟⎟⎠
⎞⎜⎜⎝
⎛= ∑∑∑
∑ ∑ ∑
===
= = =
FT,FT,FT,
FT, FT, FT,
),(),(),(
),(),(),(
CBA
A B C
DChDBgDAf
DChDBgDAf
A
B CD∑ ∑ ∑
= = =FT, FT, FT,A B C
扱いやすくない確率モデルの数理構造
∑ ∑ ∑= = =FT, FT, FT,
),(),(),(A B C
AChCBgBAf
A
B C
∑ ∑ ∑= = =FT, FT, FT,A B C
木構造をもつグラフ表現
閉路を含むグラフ表現
別々に和を計算できる
別々に和を計算することが難しい
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 12
転送行列法=確率伝搬法(1)
1次元鎖
{ } ( )∏−
=++==
1
111, ,1Pr
N
iiiii xxW
ZxX rr
( ) ( )∑∑ ∑ ∏−
⎟⎟⎠
⎞⎜⎜⎝
⎛≡
−
=++→−
1 2 1
1
111,1 ,
x x x
k
iiiiikkk
k
xxWxL L
( )kkk xL →−1
( )kkk xR →+1
k
k( ) ( )∑ ∑ ∑ ∏
+ + −⎟⎟⎠
⎞⎜⎜⎝
⎛≡
−
=++→+
1 2 1
111,1 ,
k k Nx x x
N
kiiiiikkk xxWxR L
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 13
転送行列法=確率伝搬法(2)
漸化式
( ) ( ) ( )∑ ++→−++→ =kx
kkkkkkkkkk xxWxLxL 11,111 ,
( )kkk xL →−1
k
( )11 ++→ kkk xL
1+k
{ } { }
( ) ( )( ) ( )∑
∑∑ ∑ ∑ ∑ ∑
→+→−
→+→−=
===− + + −
m
m m m N
xmmmmmm
mmmmmm
x x x x x xmm
xRxLxRxL
xXxX
11
11
1 2 1 1 2 1
PrPr rrLL
( )kkk xR →+1k
( )11 −−→ kkk xR1−k
( ) ( ) ( )kkkx
kkkkkkk xRxxWxRk
→+−−−−→ ∑= 11,111 ,
パスはひとつ
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 14
Belief Propagation for Tree Graphical Model
3
2 1
5
4
3
2 1
5
4
13→M
14→M
15→M
44 344 2144 344 2144 344 21)(
5115
)(
4114
)(
31132112
5115411431132112
115
5
114
4
113
3
3 4 5
),(),(),(),(
),(),(),(),(
xM
x
xM
x
xM
x
x x x
xxWxxWxxWxxW
xxWxxWxxWxxW
→→→
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛= ∑∑∑
∑∑∑
∑∑∑3 4 5x x x
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 15
Belief Propagation for Tree Graphical Model
3
2 1
5
4
The message from node 1 to node 2 can be expressed in terms of all the messages incoming to node 1 except the own message.
3
2 1
5
4
13→M14→M
15→M∑=
1x
121→M
2
Summation over all the nodes
except node 2
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 16
確率伝搬法 (Belief Propagation)
ab
1
cd
2
3 4
56
( )( ) ( ) ( ) ( ) ( )22211211
21
,,,,,,,,,,
xWxWxxWxWxWxxP
CBA dcbadcba
=
a∈3xc∈5x
b∈4xd∈6x
閉路のないグラフ上の確率モデル
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 17
確率伝搬法 (Belief Propagation)
1
2 ( )2112 , xxW
b41
( )1, xWB b
1
a3
( )1, xWA a
d
26
( )2, xWC c
c5
2
( )2, xWD d
( )( ) ( )( ) ( ) ( )222112
11
21
,,,,,
,,,,,
xWxWxxWxWxW
xxP
C
BA
dcba
dcba
×=
ab
1
cd
2
3 4
56
閉路のないグラフ上の確率モデル
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 18
閉路のないグラフ上の確率伝搬法
{ } ( )∏−
=++==
1
111, ,1Pr
N
iiiii xxW
ZxX rr
( ) ( )
( ) ( ) ( ) ( )∑
∑∑ ∑ ∏
++→−→−→−
=++++→
=
⎟⎟⎠
⎞⎜⎜⎝
⎛=
k
k
xkkkkkkkkkkkkk
x x x
k
iiiiikkk
xxWxMxMxM
xxWxM
11,321
111,11
,
,1 2
L
閉路が無いことが重要!!
同じノードは2度通らない
1X
2X 3X
1−kX
kX
2−kX
3−kX
1+kX
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 19
確率伝搬法
B:すべてのリンクの集合
( ) ( )
( ) ( ) ( ) ( )( ) ( ) ( ) ( )∑
∑∑ ∑
→→→→
→→→→≅
≡
1
2 3
115114113112
115114113112
32111 ,,,,
z
x x xL
zMzMzMzMxMxMxMxM
xxxxPxPL
LL
21
3
4
5
閉路のあるグラフ上の確率モデル閉路のあるグラフ上の確率モデル
( ) ( ) ( )∏∈
Φ==Bij
jiijL xxxxxPxP ,,,, 21 Lr
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 20
閉路のあるグラフ上の確率モデル
( )MMrrr
Ψ=メッセージに対する固定点方程式
閉路のあるグラフ上でも局所的な構造だけに着目してアルゴリムを構成することは可能.ただし,得られる結果は厳密ではなく近似アルゴリズム
( )( ) ( ) ( ) ( )
( ) ( ) ( ) ( )∑∑
∑
→→→
→→→
→ =
1 2
1
1151141132112
1151141132112
221 ,
,
z z
z
zMzMzMzzW
zMzMzMxzW
xM
21
3
4
5
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 21
固定点方程式と反復法
固定点方程式 ( )** MMrrr
Ψ=反復法
( )( )( )
M
rrr
rrr
rrr
23
12
01
MM
MM
MM
Ψ←
Ψ←
Ψ←
繰り返し出力を入力に入れることにより,固定点方程式の解が数値的に得られる.
0M1M
1M
0
xy =
)(xy Ψ=
y
x*M
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 22
Probabilistic Image Processing by Bayesian NetworkThe systems have double loop structures of an inner loop and an outer loop.Inner loop is the belief propagation in Bayesian network and outer loop is a statistical learning method to estimate hyperparameters α and σ.
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 23
確率伝搬法の情報論的解釈
[ ] ( )( ) 0ln)( ≥⎟⎟
⎠
⎞⎜⎜⎝
⎛≡∑ x
xxx P
QQPQD ( ) ⎟⎠
⎞⎜⎝
⎛=≥ ∑
xxx 1)( ,0 QQQ
( ) ( )
ZQF
ZQQxxWQPQD
QF
Nijjiij
ln][
lnln)(,ln)(]|[
][
+=
++= ∑∑∑∈ 4444444 34444444 21
xxxxx
( ) ( ) [ ] 0=⇒= PQDPQ xx
( ) ZPFQQFQ
ln][1][min −==⎭⎬⎫
⎩⎨⎧
=∑x
x
( )∑∏∈
≡x Nij
jiij xxWZ ,
( ) ( )∏∈
=Nij
jiijL xxWZ
xxxP ,1,,, 21 L
Free Energy
Kullback-Leibler Divergence
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 24
確率伝搬法の情報論的解釈
[ ] [ ] ( )ZQFPQD ln+=[ ] ( ) ( )
{ }( ) ( )
( ) ( ) ( )xx
xxx
xxx
x
xx
xx
QQxxWxxQ
QQxxWQ
QQxxWQQF
Eij x xjiijjiij
Eij x xjiij
xx
Eijjiij
i j
i j ji
ln)(,ln,
ln)(,ln)(
ln)(,ln)(
,\
∑∑ ∑∑
∑∑ ∑∑ ∑
∑∑∑
+=
+⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
+≡
∈
∈
∈
[ ] ( )( ) 0ln)( ≥⎟⎟
⎠
⎞⎜⎜⎝
⎛≡∑ x
xxx P
QQPQD
Free EnergyKL Divergence
( ) ( )∏∈
=Eij
jiij xxWZ
P ,1x
{ }∑≡
ji xx
jiij
Q
xxQ
,)(
),(
\xx
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 25
確率伝搬法の情報論的解釈
[ ] [ ] ( )ZQFPQD ln+=[ ] ( ) ( )
( )
( ) ( )
( ) ( )
( ) ( ) ( ) ( ) ( ) ( )∑ ∑∑∑∑
∑∑
∑∑∑
∑
∑∑∑
∈
∈
∈
∈
⎟⎟
⎠
⎞
⎜⎜
⎝
⎛−−+
+
≈
+
=
Eijjjiiijij
Viii
Eijijij
Eijijij
QQQQQQ
WQ
WQQF
ξξξ ζ
ξ
ξ ζ
ξ ζ
ξξξξζξζξ
ξξ
ζξζξ
ζξζξ
lnln,ln,
ln
,ln,
ln)(
,ln,
xxx
Bethe Free Energy
Free EnergyKL Divergence( ) ( )∏
∈=
Eijjiij xxW
ZP ,1x
{ }∑≡
ji xxjiij QxxQ
,)(),(
\xx
∑≡ix
ii QxQ\x
x)()(
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 26
確率伝搬法の情報論的解釈
[ ] FPQDQQ γγ
minargminarg ≅
( ) ( )∑=ς
ςξξ ,iji QQ
[ ] { }[ ] ZQQFPQD iji ln,Bethe +≅
[ ]{ }
{ }[ ]ijiQQQ
QQFPQDiji
,minargminarg Bethe,
⇒
( ) ( ) 1, == ∑∑∑ξ ςξ
ςξξ iji QQ
{ }[ ] ( ) ( ) ( ) ( )
( ) ( ) ( ) ( ) ( ) ( )∑ ∑∑∑∑
∑∑∑ ∑∑
∈
∈∈
⎟⎟
⎠
⎞
⎜⎜
⎝
⎛−−+
+≡
Eijjjiiijij
Viii
Eijijijiji
QQQQQQ
QQWQQQF
ξξξ ς
ξξ ς
ξξξξςξςξ
ξξςξςξ
lnln,ln,
ln,ln,,Bethe
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 27
確率伝搬法の情報論的解釈
{ }[ ] { }[ ]
( ) ( ) ( )
( ) ( )∑ ∑∑∑ ∑
∑ ∑ ∑ ∑
∈∈
∈ ∂∈
⎟⎟
⎠
⎞
⎜⎜
⎝
⎛−−⎟
⎟
⎠
⎞
⎜⎜
⎝
⎛−−
⎟⎟
⎠
⎞
⎜⎜
⎝
⎛−−
≡
Eijijij
Viii
Vi ijijiji
ijiiji
QQFQQL
1,1
,
,,
,
BetheBethe
ξ ζξ
ξ ς
ζξνξν
ζξξξλ
{ }{ }[ ] ( ) ( ) ( ) ( )
⎪⎭
⎪⎬⎫
⎪⎩
⎪⎨⎧
=== ∑∑∑∑ 1, ,,,minarg Bethe, ξ ςξς
ςξξςξξ ijiijiijiQQ
QQQQQQFiji
Lagrange Multipliers to ensure the constraints
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 28
確率伝搬法の情報論的解釈
{ }[ ] { }[ ] ( ) ( ) ( )
( ) ( )
( ) ( ) ( ) ( )
( ) ( ) ( ) ( ) ( ) ( )
( ) ( ) ( ) ( ) ( )∑ ∑∑∑ ∑∑ ∑ ∑ ∑
∑ ∑∑∑∑
∑∑∑ ∑∑
∑ ∑∑∑ ∑
∑ ∑ ∑ ∑
∈∈∈ ∂∈
∈
∈∈
∈∈
∈ ∂∈
⎟⎟
⎠
⎞
⎜⎜
⎝
⎛−−⎟
⎟
⎠
⎞
⎜⎜
⎝
⎛−−⎟
⎟
⎠
⎞
⎜⎜
⎝
⎛−−
⎟⎟
⎠
⎞
⎜⎜
⎝
⎛−−+
+=
⎟⎟
⎠
⎞
⎜⎜
⎝
⎛−−⎟
⎟
⎠
⎞
⎜⎜
⎝
⎛−−
⎟⎟
⎠
⎞
⎜⎜
⎝
⎛−−≡
Eijijij
Viii
Vi ijijiji
Eijjjiiijij
Viii
Eijijij
Eijijij
Viii
Vi ijijijiijiiji
QQQQ
QQQQQQ
QQWQ
QQQQFQQL
1,1,
lnln,ln,
ln,ln,
1,1
,,,
,
,BetheBethe
ξ ζξξ ζ
ξξξ ζ
ξξ ζ
ξ ζξ
ξ ς
ζξνξνζξξξλ
ξξξξζξζξ
ξξζξζξ
ζξνξν
ζξξξλ
( ) { }[ ] 0,Bethe =∂
∂iji
ii
QQLxQ
Extremum Condition
( ) { }[ ] 0,, Bethe =
∂∂
ijijiij
QQLxxQ
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 29
確率伝搬法の情報論的解釈
144 2
5
13→M
14→M
15→M
12→M
33
( ) ( ) ( )( ) ( )115114
11311211xMxMxMxMxQ
→→
→→
×
∝( ) ( ) ( ) ( )
( )( ) ( ) ( )228227226
2112
1151141132112,
,
xMxMxMxxW
xMxMxMxxQ
→→→
→→→
×
×
∝
ExtremumCondition( ) { }[ ] 0,Bethe =
∂∂
ijiii
QQLxQ ( ) { }[ ] 0,
, Bethe =∂
∂iji
jiij
QQLxxQ
26→M144
5
13→M
14→M
15→M
12Φ33
2
6
27→M
88
77
28→M
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 30
確率伝搬法の情報論的解釈
144 2
5
13→M
14→M
15→M
12→M
33
( ) ( )∑=2
211211 ,x
xxQxQ
( ) ( ) ( )( ) ( )115114
11311211xMxMxMxMxQ
→→
→→×
∝( ) ( ) ( ) ( )
( )( ) ( ) ( )228227226
2112
1151141132112,
,
xMxMxMxxW
xMxMxMxxQ
→→→
→→→
×
×
∝
( )( ) ( )
( ) ( )228227
2262112
112
2
,
xMxM
xMxxW
xM
x
→→
→
→
×
∝ ∑
Message Update Rule
26→M144
5
13→M
14→M
15→M
12Φ33
2
6
27→M
88
77
28→M
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 31
確率伝搬法の情報論的解釈
( )( ) ( ) ( ) ( )
( ) ( ) ( ) ( )∑∑
∑
→→→
→→→
→ =
1 2
1
1151141132112
1151141132112
221 ,
,
x x
x
xMxMxMxxW
xMxMxMxxW
xM
1
33
44 2
5
13→M
14→M
15→M
21→M
144
5
33
2
6
88
77
∑1x
211 7
6
88
=
Message Passing Rule of Belief Propagation
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 32
統計物理学による確率伝搬法の一般化
一般化された確率伝搬法一般化された確率伝搬法J. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy approximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005).
クラスター変分法という統計物理学の手法クラスター変分法という統計物理学の手法がその一般化の鍵がその一般化の鍵R. Kikuchi: A theory of cooperative phenomena, Phys. Rev., 81(1951).T. Morita: Cluster variation method of cooperative phenomena andits generalization I, J. Phys. Soc. Jpn, 12 (1957).
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 33
確率伝搬法の統計物理学的位置付け
木構造を持つグラフィカルモデルではベーテ近似は転送行列法と等価である.
閉路を持つグラフィカルモデル上のベイジアンネットでの確率伝搬法はベーテ近似またはその拡張版であるクラスター変分法に等価である(Yedidia, Weiss and Freeman, NIPS2000).
ベーテ近似転送行列法||(木構造)
もともとの確率伝搬法
クラスター変分法(菊池近似)
一般化された確率伝搬法
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 34
統計的性能の解析的評価
ydxXyYxyhV
x rrrrrrrrr },|Pr{),,(1),|(MSE2
σσασα ==−= ∫
),,( σαyh rr
gyr
Additive White Gaussian Noise
},|Pr{ σxXyY rrrr==xr
},,|Pr{ σαyYxX rrrr==
Posterior Probability
Restored Image
Original Image Degraded Image
},|Pr{ σxXyY rrrr==
Additive White Gaussian Noise
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 35
統計的性能の解析的評価
( ) { }
( ) { } ydxXyYxyV
ydxXyYxyhV
x
rrrrrrr
rrrrrrrrr
Pr1
Pr,,1),|MSE(
212
2
==−+=
==−≡
∫
∫
−CI ασ
σασα
( )
( ) y
xdyxPxyhr
rrrrrr
12
)|(,,−
+=
= ∫CI ασ
σα
{ } ( )
)2
1exp(
21exp,Pr
22
22
yx
yxxXyYVi
ii
rr
rrrr
−−=
⎟⎠⎞
⎜⎝⎛ −−∝== ∏
∈
σ
σσ
⎪⎩
⎪⎨
⎧∈−∈=
=otherwise,0
},{,1,4
EjiVji
ji C
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 36
Statistical Performance Estimation for Gaussian Markov Random Fields
T22
242
22
2
22
||
22
242T
22
||
22
2T
22
||
22
2T
22
||
22T
22
||
2
2
2
T
2
2
2
22
||2
2
2
2
22
||2
22
22
||2
2
2
)(||1
)(Tr1
21exp
21
)(1
21exp
21)(
)(1
21exp
21
)()(1
21exp
21)(
)()(1
21exp
21)()(1
21exp
21)(1
21exp
21)(1
21exp
211
},|Pr{),,(1),|(MSE
xI
xVV
ydxyxxV
ydxyxyxV
ydxyxxyV
ydxyxyxyV
ydxyxxyxxyV
ydxyxxyV
ydxyxxxyV
ydyxxyV
ydxXyYxyhV
x
V
V
V
V
V
V
V
V
rr
rrrrr
rrrrrr
rrrrrr
rrrrrrr
rrrrrrrrr
rrrrrr
rrrrrrr
rrrrr
rrrrrrrrr
CC
CII
CIC
CIC
CIC
CII
CIC
CII
CIC
CII
CIC
CII
CII
CII
CII
ασσα
ασσ
σσπασσα
σσπασασ
σσπασασ
σσπασ
σσπασασ
ασασασ
ασ
σσπασασ
ασ
σσπασασ
σσπασ
σσασα
++⎟⎟
⎠
⎞⎜⎜⎝
⎛
+=
⎟⎠⎞
⎜⎝⎛ −−⎟⎟
⎠
⎞⎜⎜⎝
⎛+
+
⎟⎠⎞
⎜⎝⎛ −−⎟⎟
⎠
⎞⎜⎜⎝
⎛−
+−
⎟⎠⎞
⎜⎝⎛ −−⎟⎟
⎠
⎞⎜⎜⎝
⎛+
−−
⎟⎠⎞
⎜⎝⎛ −−⎟⎟
⎠
⎞⎜⎜⎝
⎛−
+−=
⎟⎠⎞
⎜⎝⎛ −−⎟⎟
⎠
⎞⎜⎜⎝
⎛⎟⎟⎠
⎞⎜⎜⎝
⎛
++−
+⎟⎟⎠
⎞⎜⎜⎝
⎛
++−
+=
⎟⎠⎞
⎜⎝⎛ −−⎟⎟
⎠
⎞⎜⎜⎝
⎛+
+−+
=
⎟⎠⎞
⎜⎝⎛ −−⎟⎟
⎠
⎞⎜⎜⎝
⎛−
++−
+=
⎟⎠⎞
⎜⎝⎛ −−⎟⎟
⎠
⎞⎜⎜⎝
⎛×−
+=
==×−=
∫
∫
∫
∫
∫
∫
∫
∫
∫
= 0
⎪⎩
⎪⎨
⎧∈−∈=
=otherwise,0
},{,1,4
EjiVji
ji C
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 37
Statistical Performance Estimation for Gaussian Markov Random Fields
xI
xVV
ydyxxyV
ydxXyYxyhV
x
V
rr
rrrrr
rrrrrrrrr
22
242T
22
2
22
||
2
2
2
2
)(||1
)(Tr1
21exp
211
},|Pr{),,(1),|(MSE
CC
CII
CII
ασσα
ασσ
σσπασ
σσασα
++⎟⎟
⎠
⎞⎜⎜⎝
⎛
+=
⎟⎠⎞
⎜⎝⎛ −−⎟⎟
⎠
⎞⎜⎜⎝
⎛×−
+=
==×−=
∫
∫
⎪⎩
⎪⎨
⎧∈−∈=
=otherwise,0
},{,1,4
EjiVji
ji C
0
200
400
600
0 0.001 0.002 0.003α
σ=40
0
200
400
600
0 0.001 0.002 0.003
σ=40
α
),|(MSE xrσα ),|(MSE xrσα
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 38
Statistical Performance Estimation for Binary Markov Random Fields
ydxXyYxyhV
x rrrrrrrrr },|Pr{),,(1),|(MSE2
σσασα ==−= ∫
⎟⎟
⎠
⎞
⎜⎜
⎝
⎛+∝
==
∑∑∈∈ Eji
jiVi
ii sssy
yYsX
},{2
1exp
},,|Pr{
ασ
σαrrrr
∫ ∫ ∫ ∑ ∑ ∑ ∑∑∏∞+
∞−
∞+
∞−
∞+
∞−±= ±= ±= ∈== ⎟⎟
⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
⎟⎟
⎠
⎞
⎜⎜
⎝
⎛+
⎟⎟
⎠
⎞
⎜⎜
⎝
⎛⎟⎠⎞
⎜⎝⎛ −− ||21
1 1 1 },{
||
12
||
1
22
1 2 ||
1explog)(2
1exp Vs s s Eji
ji
V
iii
V
iii dydydysssyxy
V
LLL ασσ
It can be reduced to the calculation of the average of free energy with respect to locally non-uniform external fields y1,y2,…,y|V|.
Free Energy of Ising Model with Random External Fields
1±=isLight intensities of the original image can be regarded as spin states of ferromagnetic system.
== >
== >
Eji ∈},{
Eji ∈},{
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo) 39
0
0.1
0.2
0.3
0.4
0 0.2 0.4 0.6 0.8 1
Statistical Performance Estimation for Markov Random Fields
ydxXyYxyhV
x rrrrrrrrr },|Pr{),,(1),|(MSE2
σσασα ==−= ∫
0
200
400
600
0 0.001 0.002 0.003α
σ=40
),|(MSE xrσα
σ=1
α
),|(MSE xrσα
Multi-dimensional Gauss Integral Formulas
Spin Glass Theory in Statistical MechanicsLoopy Belief Propagation