Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 1
Physical FluctuomaticsApplied Stochastic Process
6th Graphical model and physical model
Kazuyuki TanakaGraduate School of Information Sciences, Tohoku University
[email protected]://www.smapip.is.tohoku.ac.jp/~kazu/
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 2
Textbooks
Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 5.
ReferencesH. Nishimori: Statistical Physics of Spin Glasses and Information Processing, ---An Introduction, Oxford University Press, 2001. H. Nishimori, G. Ortiz: Elements of Phase Transitions and Critical Phenomena, Oxford University Press, 2011.M. Mezard, A. Montanari: Information, Physics, and Computation, Oxford University Press, 2010.
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 3
Main InterestsInformation Processing:
DataPhysics:
Material, Natural Phenomena
System of a lot of elements with mutual relationCommon Concept between Information Sciences and Physics
MaterialMolecule
Materials are constructed from a lot of molecules.
Molecules have interactions of each other.
0 ,1 •101101110001
•010011101110101000111110000110000101000000111010101110101010Bit
Data
Data is constructed from many bits
A sequence is formed by deciding the arrangement of bits.
A lot of elements have mutual relation of each otherSome physical concepts
in Physical models are useful for the design of computational models in probabilistic information processing.
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 4
Why is a physical viewpoint effective in probabilistic information processing?
Matrials are constructed from a lot of molecules.(1023 molecules exist in 1 mol.)
Molecules have intermolecular forces of each other
1 2
,,, 21x x x
N
N
xxxf
Theoretical physicists always have to treat such multiple summation.
Development of Approximate MethodsProbabilistic information processing is also usually reduced to multiple summations or integrations.
Application of physical approximate methods to probabilistic information processing
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 5
Probabilistic Model for Ferromagnetic MaterialsProbabilistic Model for
Ferromagnetic Materials
p p
p p
)1,1()1,1()1.1()1.1( PPPP
pPP )1.1()1,1(
11 a
1
12 a
1
11
1 1
p
PP
2
1
)1.1()1,1(
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 6
Probabilistic Model for Ferromagnetic MaterialsProbabilistic Model for
Ferromagnetic Materials
Prior probability prefers to the configuration with the least number of red lines.
> >=
Lines Red of #Lines Blue of # )2
1()( ppaP
p p
11 a 112 a 111 1 1
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 7
More is different in Probabilistic Model for Ferromagnetic Materials
Disordered State
Ordered State
Sampling by Markov Chain Monte Carlo method
p p
Small p Large p
p p
More is different.
p2
1p
2
1
Critical Point(Large fluctuation)
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 8
Fundamental Probabilistic Models for Magnetic Materials
Since h is positive, the probablity of up spin is larger than the one of down spin .
1
)exp(
)exp()(
a
ha
haaP
1a
+1 1
he he
)tanh()(1
haaPma
h : External Field
)(tanh1)()( 2
1
2 haPmaaVa
Variance
Average
0h
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 9
Fundamental Probabilistic Models for Magnetic Materials
Since J is positive, (a1,a2)=(+1,+1) and (1,1) have the largest probability .
1 121
2121
1 2
)exp(
)exp(),(
a a
aJa
aJaaaP
11 a
0),(1 1
2111
1 2
a a
aaPam
J : Interaction
1),()(1 1
212
111
1 2
a a
aaPmaaVVariance
Average
0J
Je Je
+1 +1 1 1
+1 +1 11
12 aJe Je
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 10
Fundamental Probabilistic Models for Magnetic Materials
a
aEZ
))(exp(
Eji
jiVi
i aaJahaE},{
)(
Translational Symmetry
),( EVJ
J
h h
)(exp1
)( aEZ
aP
),,,( 21 Naaaa
E : Set of All the neighbouring Pairs of Nodes
1ia 1ia
N
i ai aPa
Nm
1
)(1
Problem: Compute
)'()()'()( aPaPaEaE
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 11
Fundamental Probabilistic Models for Magnetic Materials
Eji
jiVi
i aaJahaE},{
)(
N
i ai
NhaPa
Nm
10)(
1limlim
)(exp1
)( aEZ
aP
),,,( 21 Naaaa
1ia
Problem: Compute
Translational Symmetry
),( EV
J
J
h h
Spontaneous Magnetization
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 12
Mean Field Approximation for Ising Model
)},{( 0))(( Ejimama ji We assume that the probability for configurations satisfying
Vi
iaJmhaE )4()(
2mmamaaa ijji
Eji
jiVi
i aaJahaE},{
iJm
Jm
JmJm
h
are large.
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 13
Mean Field Approximation for Ising Model
)4tanh()(1
1
JmhaPaN
mN
i ai
Vi
ii aPaEZ
aP )())(exp(1
)(
Fixed Point Equation of m)(mm
We assume that all random variables ai are independent of each other, approximately.
Vi
iaJmhaE )4()(
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 14
Fixed Point Equation and Iterative Method
•Fixed Point Equation ** MM
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 15
Fixed Point Equation and Iterative Method
•Fixed Point Equation ** MM •Iterative Method
0
xy
)(xy
y
x*M
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 16
Fixed Point Equation and Iterative Method
•Fixed Point Equation ** MM •Iterative Method
0M0
xy
)(xy
y
x*M
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 17
Fixed Point Equation and Iterative Method
•Fixed Point Equation ** MM •Iterative Method
01 MM
0M
1M
0
xy
)(xy
y
x*M
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 18
Fixed Point Equation and Iterative Method
•Fixed Point Equation ** MM •Iterative Method
12
01
MM
MM
0M1M
1M
0
xy
)(xy
y
x*M
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 19
Fixed Point Equation and Iterative Method
•Fixed Point Equation ** MM •Iterative Method
12
01
MM
MM
0M1M
1M
0
xy
)(xy
y
x*M
2M
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 20
Fixed Point Equation and Iterative Method
•Fixed Point Equation ** MM •Iterative Method
23
12
01
MM
MM
MM
0M1M
1M
0
xy
)(xy
y
x*M
2M
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 21
Marginal Probability Distribution in Mean Field Approximation
))4exp((1
)()(
1 2 1 1
ii
a a a a aii
aJmhZ
aPaP
i i N
i
JmJm
JmJm
h
1
)(
iaiii aPam
))4tanh(( mJhm Jm: Mean Field
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 22
Advanced Mean Field Method
))4exp((1
)( ii
ii ahZ
aP
)))(3exp((1
),( jijii
jiij aJaaahZ
aaP
h
h
h
1
),()(
jajiijii aaPaP
))3tanh()(tanh(arctanh hJ
Bethe Approximation
Kikuchi Method (Cluster Variation Meth)
: Effective Field
Fixed Point Equation for
J
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 23
Average of Ising Model on Square Grid Graph
(a) Mean Field Approximation(b) Bethe Approximation(c) Kikuchi Method (Cluster Variation Method)(d) Exact Solution ( L. Onsager , C.N.Yang )
J/1
a
iNh
aPa
)(limlim
0
Ejiji
Vii aaJah
ZaP
},{
exp1 ),( EVJ
J
h h
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 24
Model Representation in Statistical Physics
),,,(},,,Pr{ 212211 NNN aaaPaAaAaA
a
aEZ
))(exp(
)(}Pr{ aPaA
))(exp(1
)( aEZ
aP
),,,( 21 NAAAA
Gibbs Distribution Partition Function
)))(exp(ln(ln a
aEZF
Free Energy
Energy Function
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 25
Gibbs Distribution and Free Energy
Gibbs Distribution
ZPFaQQFaQ
ln][}1)(|][{min
))(exp(1
)( aEZ
aP
)(ln)()()(][ aQaQaQaEQFaa
Variational Principle of Free Energy Functional Variational Principle of Free Energy Functional FF[[QQ] under Normalization Condition for ] under Normalization Condition for QQ((aa))
Free Energy Functional of Trial Probability Distribution Q(a)
a
aEZ
))(exp(lnlnFree Energy
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 26
Explicit Derivation of Variantional Principle for Minimization of Free Energy Functional
ZPFaQQFaQ
ln][}1)(|][{min
)(
)(exp
)(exp)(ˆ aP
aE
aEaQ
a
1)()())(ln)((1)(
aaa
aQaQaQaEaQQFQL
01)(ln)(
)(
aQaEaQ
QL
1)(exp)(ˆ aEaQ
Normalization Condition
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 27
Kullback-Leibler Divergence and Free Energy
0)(
)(ln)(
aP
aQaQPQD
a
a
aQaQ
1)( ,0)(
ZQF
ZaQaQaEaQPQD
QF
aa
ln][
ln)(ln)()()(]|[
][
0)()( PQDaPaQ
))(exp(1
)( aEZ
aP
}1)(|]|[{minarg}1)(|][{minarg aQaQ
aQPQDaQQF
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 28
Interpretation of Mean Field Approximation as Information Theory
Vi
ii aQaQ )()(
)(
)(ln)(
aP
aQaQPQD
a
))(exp(1
aEZ
aP
and
Marginal Probability Distributions Qi(ai) are determined so as to minimize D[Q|P]
1 2 1 1 2
)()()(\ a a a a a aaa
ii
i i i Ni
aQaQaQ
Minimization of Kullback-Leibler Divergence between
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 29
Interpretation of Mean Field Approximation as Information Theory
Eji
jiVi
i aaJahaE},{
)(
Vi a
iiVi a
i
i
aPaV
aPaV
m1
)(||
1)(
||
1
)(exp1
)( aEZ
aP
),,,( ||21 Vaaaa
1ia
Problem: Compute
Translational Symmetry
),( EV
J
J
h h
Magnetization
1 2 1 1 2
)()()(\ a a a a a aaa
ii
i i i Ni
aPaPaP
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 30
Kullback-Leibler Divergence in Mean Field Approximation for Ising Model
Vi
ii aQaQ )()(
ZViQFPQD i ln|MF
)(
)(ln)(
aP
aQaQPQD
a
Viii
Ejiji
Viii
QQQQJ
QhViQF
1},{ 11
1MF
ln))()()((
)(}]|[{
1 2 1 1 2
)(
)()(\
a a a a a a
aaii
i i i N
i
aQ
aQaQ
Eji
jiVi
i aaJahaE},{
)(
)(exp1
)( aEZ
aP
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 31
Minimization of Kullback-Leibler Divergence and Mean Field Equation
)( ))(ˆ(exp1ˆ
1
ViQJhZ
Qij
ji
i
} ,1)(|]|[{minarg)}(ˆ{}{
ViQPQDQ iQ
ii
Fixed Point Equations for {Qi|iV}
Variation
1 1
))(ˆ(exp
ij
ji QJhZ
i
Set of all the neighbouring nodes of the node i
Ejiji },{
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 32
Orthogonal Functional Representation of Marginal Probability Distribution of Ising Model
iiii amaQ2
1
2
1)(
1
)(
iaiii
aii aQaaQam
),,,( 21 Naaaa
1ia
ia
iiia
iia
iii
aii
ai
aii
iiii
maQadddacaaQa
aQccdacaQ
adacaQ
iii
iii
2
1)(
2
12)()(
2
1)(
2
12)()(
1)( )(
111
111
2
1 2 1 1
)()()(\ a a a a aaa
ii
i i Ni
aQaQaQ
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 33
Conventional Mean Field Equation in Ising Model
)4tanh( Jmhm
iiiiiaa
ii maamaQaPaPi
2
1
2
1
2
1
2
1)(ˆ)()(
\
maPaN
N
i ai
1
)(1
Fixed Point Equation
mmmm N 21
Eji
jiVi
i aaJahaE},{
)(
))4exp((1
))(ˆ(exp1
)(ˆ1
ii
iij
ji
ii aJmhZ
aQJhZ
aQ
VJ
J
Translational Symmetry
h h
)( 4|| Vii
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 34
Interpretation of Bethe Approximation (1)
Eji
jiVi
i aaJahaE},{
)(
)(exp1
)( aEZ
aP
),,,( ||21 Vaaaa
1ia
Translational Symmetry
),( EV
J
J
h h
1 2 1 1 2
)()()(\ a a a a a aaa
ii
i i i Ni
aPaPaP
1 2 1 1 2 1 1 2
)()(),(},\{ a a a a a a a a aaaa
jiij
i i i j j j Nji
aPaPaaP
Eji
jiij aaZ
aP},{
),(1
)(
jijijiij aJaha
jha
iaa
||
1
||
1exp),(
a Eji
jiij aaZ
},{
),(
Compute
and
Interpretation of Bethe Approximation (2)
ZQFPQD ln
aQaQaaaaQ
aQaQaaaQ
aQaQaaaQQF
aEji a ajiijjiij
aEji a ajiij
aaa
aEjijiij
a
i j
i j ji
ln)(,ln,
ln)(,ln)(
ln)(,ln)(
},{
},{ ,\
},{
0ln)(
aP
aQaQPQD
a
Free EnergyKL Divergence
Eji
jiij aaZ
P},{
,1
x
ji aaa
jiij
aQ
aaQ
,
)(
),(
\
35Physics Fluctuomatics / Applied
Stochastic Process (Tohoku University)
Interpretation of Bethe Approximation (3)
ZQFPQD ln
Ejijjiiijij
Viii
Ejiijij
a
Ejiijij
QQQQQQ
Q
aQaQ
QQF
},{
},{
},{
lnln,ln,
ln
,ln,
ln)(
,ln,
Bethe Free
Energy
Free EnergyKL Divergence
Eji
jiij aaZ
aP},{
,1
ji aaajiij aQaaQ
,
)(),(\
iaaii aQaQ
\
)()(
36Physics Fluctuomatics / Applied
Stochastic Process (Tohoku University)
Interpretation of Bethe Approximation (4)
FPQDQQ
minargminarg
,iji QQ
ZQQFPQD iji ln,Bethe
ijiQQQ
QQFPQDiji
,minargminarg Bethe,
1,
iji QQ
Ejijjiiijij
Viii
Ejiijijiji
QQQQQQ
QQQQQF
},{
},{Bethe
lnln,ln,
ln,ln,,
37Physics Fluctuomatics / Applied
Stochastic Process (Tohoku University)
Interpretation of Bethe Approximation (5)
Ejiijij
Viii
Vi ijijijii
ijiiji
QQFQQL
},{
},{,
BetheBethe
1,1
,
,,
1, ,,,minarg Bethe,
ijiijiijiQQ
QQQQQQFiji
Lagrange Multipliers to ensure the constraints
38Physics Fluctuomatics / Applied
Stochastic Process (Tohoku University)
Interpretation of Bethe Approximation (6)
Ejiijij
Viii
Vi ijijijji
Ejijjiiijij
Viii
Ejiijij
Ejiijij
Viii
Vi ijijijiiijiiji
QQQQ
QQQQQQ
QQQ
QQQQFQQL
},{},{,
},{
},{
},{
},{,BetheBethe
1,1,
lnln,ln,
ln,ln,
1,1
,,,
0,Bethe
iji
ii
QQLxQ
• Extremum Condition
0,, Bethe
ijijiij
QQLxxQ
39Physics Fluctuomatics / Applied
Stochastic Process (Tohoku University)
Interpretation of Bethe Approximation (7)
ik
ikiiii ai
aQ )(1||
1exp },{, )()(exp,, 2}2,1{,21}2,1{,121122112 aaaaaaQ
Extremum
Condition 0,Bethe
iji
ii
QQLxQ 0,
, Bethe
iji
jiij
QQLxxQ
40Physics Fluctuomatics / Applied
Stochastic Process (Tohoku University)
115114
1131121
11
1
aMaM
aMaMZ
aQ
2282272262112
11511411312
2112
,
1,
aMaMaMaa
aMaMaMZ
aaQ
)()(exp\
},{, ijik
ikijii aMa
Interpretation of Bethe Approximation (8)
144 2
5
13M
14M
15M
12M
33
26M
144
5
13M
14M
15M
1233
2
6
27M
88
77
28M
115114
1131121
11
1
aMaM
aMaMZ
aQ
2282272262112
11511411312
2112
,
1,
aMaMaMaa
aMaMaMZ
aaQ
Extremum
Condition 0,Bethe
iji
ii
QQLxQ 0,
, Bethe
iji
jiij
QQLxxQ
41Physics Fluctuomatics / Applied
Stochastic Process (Tohoku University)
Interpretation of Bethe Approximation (9)
144 2
5
13M
14M
15M
12M
33
4412W
1
5
13M
14M
15M
33
26M
2
6
27M
88
77
28M
,121 QQ
115114
1131121
11
1
aMaM
aMaMZ
aQ
2282272262112
11511411312
2112
,
1,
aMaMaMaa
aMaMaMZ
aaQ
1514
1312
21
,
MM
M
M
Message Update Rule
42Physics Fluctuomatics / Applied
Stochastic Process (Tohoku University)
Interpretation of Bethe Approximation (10)
15141312
15141312
21 ,
,
MMM
MMM
M
1
33
44 2
5
13M
14M
15M
21M
144
5
33
2
6
88
77
2a
144 2
5
33
=
Message Passing Rule of Belief Propagation
It corresponds to Bethe approximation in the statistical mechanics.
43Physics Fluctuomatics / Applied
Stochastic Process (Tohoku University)
Interpretation of Bethe Approximation (11)
1 1 \
1 \
,
,
jikikij
jikikij
ji M
M
M
44Physics Fluctuomatics / Applied
Stochastic Process (Tohoku University)
))tanh()(tanh(arctanh\
jik
ikji hJ
jijiM exp
))3tanh()(tanh(arctanh hJ
ji Translational Symmetry
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 45
Summary
Statistical Physics and Information TheoryProbabilistic Model of FerromagnetismMean Field TheoryGibbs Distribution and Free EnergyFree Energy and Kullback-Leibler DivergenceInterpretation of Mean Field Approximation as Information TheoryInterpretation of Bethe Approximation as Information Theory