physical fluctuomatics applied stochastic process 6th graphical model and physical model

Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 1

Physical FluctuomaticsApplied Stochastic Process

6th Graphical model and physical model

Kazuyuki TanakaGraduate School of Information Sciences, Tohoku University

[email protected]://www.smapip.is.tohoku.ac.jp/~kazu/


Textbooks

Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 5.

ReferencesH. Nishimori: Statistical Physics of Spin Glasses and Information Processing, ---An Introduction, Oxford University Press, 2001. H. Nishimori, G. Ortiz: Elements of Phase Transitions and Critical Phenomena, Oxford University Press, 2011.M. Mezard, A. Montanari: Information, Physics, and Computation, Oxford University Press, 2010.


Main InterestsInformation Processing:

DataPhysics:

Material, Natural Phenomena

System of a lot of elements with mutual relationCommon Concept between Information Sciences and Physics

MaterialMolecule

Materials are constructed from a lot of molecules.

Molecules have interactions of each other.

０ ,1 •101101110001

•010011101110101000111110000110000101000000111010101110101010Bit

Data

Data is constructed from many bits

A sequence is formed by deciding the arrangement of bits.

A lot of elements have mutual relation of each otherSome physical concepts

in Physical models are useful for the design of computational models in probabilistic information processing.


Why is a physical viewpoint effective in probabilistic information processing?

Matrials are constructed from a lot of molecules.(1023 molecules exist in 1 mol.)

Molecules have intermolecular forces of each other

1 2

,,, 21x x x

N

N

xxxf

Theoretical physicists always have to treat such multiple summation.

Development of Approximate MethodsProbabilistic information processing is also usually reduced to multiple summations or integrations.

Application of physical approximate methods to probabilistic information processing


Probabilistic Model for Ferromagnetic MaterialsProbabilistic Model for

Ferromagnetic Materials

p p

p p

)1,1()1,1()1.1()1.1( PPPP

pPP )1.1()1,1(

11 a

1

12 a

1

11

1 1

p

PP

2

1

)1.1()1,1(


Probabilistic Model for Ferromagnetic MaterialsProbabilistic Model for

Ferromagnetic Materials

Prior probability prefers to the configuration with the least number of red lines.

> >=

Lines Red of #Lines Blue of # )2

1()( ppaP

p p

11 a 112 a 111 1 1


More is different in Probabilistic Model for Ferromagnetic Materials

Disordered State

Ordered State

Sampling by Markov Chain Monte Carlo method

p p

Small p Large p

p p

More is different.

p2

1p

2

1

Critical Point(Large fluctuation)


Fundamental Probabilistic Models for Magnetic Materials

Since h is positive, the probablity of up spin is larger than the one of down spin ．

1

)exp(

)exp()(

a

ha

haaP

1a

+1 1

he he

)tanh()(1

haaPma

h ： External Field

)(tanh1)()( 2

1

2 haPmaaVa

Variance

Average

0h



Since J is positive, (a1,a2)=(+1,+1) and (1,1) have the largest probability ．

1 121

2121

1 2

)exp(

)exp(),(

a a

aJa

aJaaaP

11 a

0),(1 1

2111

1 2

a a

aaPam

J ： Interaction

1),()(1 1

212

111

1 2

a a

aaPmaaVVariance

Average

0J

Je Je

+1 +1 1 1

+1 +1 11

12 aJe Je



a

aEZ

))(exp(

Eji

jiVi

i aaJahaE},{

)(

Translational Symmetry

),( EVJ

J

h h

)(exp1

)( aEZ

aP

),,,( 21 Naaaa

E ： Set of All the neighbouring Pairs of Nodes

1ia 1ia

N

i ai aPa

Nm

1

)(1

Problem: Compute

)'()()'()( aPaPaEaE



Eji

jiVi

i aaJahaE},{

)(

N

i ai

NhaPa

Nm

10)(

1limlim

)(exp1

)( aEZ

aP

),,,( 21 Naaaa

1ia

Problem: Compute


),( EV

J

J

h h

Spontaneous Magnetization


Mean Field Approximation for Ising Model

)},{( 0))(( Ejimama ji We assume that the probability for configurations satisfying

Vi

iaJmhaE )4()(

2mmamaaa ijji

Eji

jiVi

i aaJahaE},{

iJm

Jm

JmJm

h

are large.


Mean Field Approximation for Ising Model

)4tanh()(1

1

JmhaPaN

mN

i ai

Vi

ii aPaEZ

aP )())(exp(1

)(

Fixed Point Equation of m)(mm

We assume that all random variables ai are independent of each other, approximately.

Vi

iaJmhaE )4()(


Fixed Point Equation and Iterative Method

•Fixed Point Equation ** MM



•Fixed Point Equation ** MM •Iterative Method

0

xy

)(xy

y

x*M




0M0

xy

)(xy

y

x*M




01 MM

0M

1M

0

xy

)(xy

y

x*M




12

01

MM

MM

0M1M

1M

0

xy

)(xy

y

x*M




12

01

MM

MM

0M1M

1M

0

xy

)(xy

y

x*M

2M




23

12

01

MM

MM

MM

0M1M

1M

0

xy

)(xy

y

x*M

2M


Marginal Probability Distribution in Mean Field Approximation

))4exp((1

)()(

1 2 1 1

ii

a a a a aii

aJmhZ

aPaP

i i N

i

JmJm

JmJm

h

1

)(

iaiii aPam

))4tanh(( mJhm Jm： Mean Field


Advanced Mean Field Method

))4exp((1

)( ii

ii ahZ

aP

)))(3exp((1

),( jijii

jiij aJaaahZ

aaP

h

h

h

1

),()(

jajiijii aaPaP

))3tanh()(tanh(arctanh hJ

Bethe Approximation

Kikuchi Method (Cluster Variation Meth)

： Effective Field

Fixed Point Equation for

J


Average of Ising Model on Square Grid Graph

(a) Mean Field Approximation(b) Bethe Approximation(c) Kikuchi Method (Cluster Variation Method)(d) Exact Solution （ L. Onsager ， C.N.Yang ）

J/1

a

iNh

aPa

)(limlim

0

Ejiji

Vii aaJah

ZaP

},{

exp1 ),( EVJ

J

h h


Model Representation in Statistical Physics

),,,(},,,Pr{ 212211 NNN aaaPaAaAaA

a

aEZ

))(exp(

)(}Pr{ aPaA

))(exp(1

)( aEZ

aP

),,,( 21 NAAAA

Gibbs Distribution Partition Function

)))(exp(ln(ln a

aEZF

Free Energy

Energy Function


Gibbs Distribution and Free Energy

Gibbs Distribution

ZPFaQQFaQ

ln][}1)(|][{min

))(exp(1

)( aEZ

aP

)(ln)()()(][ aQaQaQaEQFaa

Variational Principle of Free Energy Functional Variational Principle of Free Energy Functional FF[[QQ] under Normalization Condition for ] under Normalization Condition for QQ((aa))

Free Energy Functional of Trial Probability Distribution Q(a)

a

aEZ

))(exp(lnlnFree Energy


Explicit Derivation of Variantional Principle for Minimization of Free Energy Functional

ZPFaQQFaQ

ln][}1)(|][{min

)(

)(exp

)(exp)(ˆ aP

aE

aEaQ

a

1)()())(ln)((1)(

aaa

aQaQaQaEaQQFQL

01)(ln)(

)(

aQaEaQ

QL

1)(exp)(ˆ aEaQ

Normalization Condition


Kullback-Leibler Divergence and Free Energy

0)(

)(ln)(

aP

aQaQPQD

a

a

aQaQ

1)( ,0)(

ZQF

ZaQaQaEaQPQD

QF

aa

ln][

ln)(ln)()()(]|[

][

0)()( PQDaPaQ

))(exp(1

)( aEZ

aP

}1)(|]|[{minarg}1)(|][{minarg aQaQ

aQPQDaQQF


Interpretation of Mean Field Approximation as Information Theory

Vi

ii aQaQ )()(

)(

)(ln)(

aP

aQaQPQD

a

))(exp(1

aEZ

aP

and

Marginal Probability Distributions Qi(ai) are determined so as to minimize D[Q|P]

1 2 1 1 2

)()()(\ a a a a a aaa

ii

i i i Ni

aQaQaQ

Minimization of Kullback-Leibler Divergence between


Interpretation of Mean Field Approximation as Information Theory

Eji

jiVi

i aaJahaE},{

)(

Vi a

iiVi a

i

i

aPaV

aPaV

m1

)(||

1)(

||

1

)(exp1

)( aEZ

aP

),,,( ||21 Vaaaa

1ia

Problem: Compute


),( EV

J

J

h h

Magnetization

1 2 1 1 2

)()()(\ a a a a a aaa

ii

i i i Ni

aPaPaP


Kullback-Leibler Divergence in Mean Field Approximation for Ising Model

Vi

ii aQaQ )()(

ZViQFPQD i ln|MF

)(

)(ln)(

aP

aQaQPQD

a

Viii

Ejiji

Viii

QQQQJ

QhViQF

1},{ 11

1MF

ln))()()((

)(}]|[{

1 2 1 1 2

)(

)()(\

a a a a a a

aaii

i i i N

i

aQ

aQaQ

Eji

jiVi

i aaJahaE},{

)(

)(exp1

)( aEZ

aP


Minimization of Kullback-Leibler Divergence and Mean Field Equation

)( ))(ˆ(exp1ˆ

1

ViQJhZ

Qij

ji

i

} ,1)(|]|[{minarg)}(ˆ{}{

ViQPQDQ iQ

ii

Fixed Point Equations for {Qi|iV}

Variation

1 1

))(ˆ(exp

ij

ji QJhZ

i

Set of all the neighbouring nodes of the node i

Ejiji },{


Orthogonal Functional Representation of Marginal Probability Distribution of Ising Model

iiii amaQ2

1

2

1)(

1

)(

iaiii

aii aQaaQam

),,,( 21 Naaaa

1ia

ia

iiia

iia

iii

aii

ai

aii

iiii

maQadddacaaQa

aQccdacaQ

adacaQ

iii

iii

2

1)(

2

12)()(

2

1)(

2

12)()(

1)( )(

111

111

2

1 2 1 1

)()()(\ a a a a aaa

ii

i i Ni

aQaQaQ


Conventional Mean Field Equation in Ising Model

)4tanh( Jmhm

iiiiiaa

ii maamaQaPaPi

2

1

2

1

2

1

2

1)(ˆ)()(

\

maPaN

N

i ai

1

)(1

Fixed Point Equation

mmmm N 21

Eji

jiVi

i aaJahaE},{

)(

))4exp((1

))(ˆ(exp1

)(ˆ1

ii

iij

ji

ii aJmhZ

aQJhZ

aQ

VJ

J


h h

)( 4|| Vii


Interpretation of Bethe Approximation (1)

Eji

jiVi

i aaJahaE},{

)(

)(exp1

)( aEZ

aP

),,,( ||21 Vaaaa

1ia


),( EV

J

J

h h

1 2 1 1 2

)()()(\ a a a a a aaa

ii

i i i Ni

aPaPaP

1 2 1 1 2 1 1 2

)()(),(},\{ a a a a a a a a aaaa

jiij

i i i j j j Nji

aPaPaaP

Eji

jiij aaZ

aP},{

),(1

)(

jijijiij aJaha

jha

iaa

||

1

||

1exp),(

a Eji

jiij aaZ

},{

),(

Compute

and


ZQFPQD ln

aQaQaaaaQ

aQaQaaaQ

aQaQaaaQQF

aEji a ajiijjiij

aEji a ajiij

aaa

aEjijiij

a

i j

i j ji

ln)(,ln,

ln)(,ln)(

ln)(,ln)(

},{

},{ ,\

},{

0ln)(

aP

aQaQPQD

a

Free EnergyKL Divergence

Eji

jiij aaZ

P},{

,1

x

ji aaa

jiij

aQ

aaQ

,

)(

),(

\

35Physics Fluctuomatics / Applied

Stochastic Process (Tohoku University)


ZQFPQD ln

Ejijjiiijij

Viii

Ejiijij

a

Ejiijij

QQQQQQ

QQ

Q

aQaQ

QQF

},{

},{

},{

lnln,ln,

ln

,ln,

ln)(

,ln,

Bethe Free

Energy

Free EnergyKL Divergence

Eji

jiij aaZ

aP},{

,1

ji aaajiij aQaaQ

,

)(),(\

iaaii aQaQ

\

)()(




FPQDQQ

minargminarg

,iji QQ

ZQQFPQD iji ln,Bethe

ijiQQQ

QQFPQDiji

,minargminarg Bethe,

1,

iji QQ

Ejijjiiijij

Viii

Ejiijijiji

QQQQQQ

QQQQQF

},{

},{Bethe

lnln,ln,

ln,ln,,




Ejiijij

Viii

Vi ijijijii

ijiiji

QQ

QQ

QQFQQL

},{

},{,

BetheBethe

1,1

,

,,

1, ,,,minarg Bethe,

ijiijiijiQQ

QQQQQQFiji

Lagrange Multipliers to ensure the constraints




Ejiijij

Viii

Vi ijijijji

Ejijjiiijij

Viii

Ejiijij

Ejiijij

Viii

Vi ijijijiiijiiji

QQQQ

QQQQQQ

QQQ

QQ

QQQQFQQL

},{},{,

},{

},{

},{

},{,BetheBethe

1,1,

lnln,ln,

ln,ln,

1,1

,,,

0,Bethe

iji

ii

QQLxQ

• Extremum Condition

0,, Bethe

ijijiij

QQLxxQ




ik

ikiiii ai

aQ )(1||

1exp },{, )()(exp,, 2}2,1{,21}2,1{,121122112 aaaaaaQ

Extremum

Condition 0,Bethe

iji

ii

QQLxQ 0,

, Bethe

iji

jiij

QQLxxQ



115114

1131121

11

1

aMaM

aMaMZ

aQ

2282272262112

11511411312

2112

,

1,

aMaMaMaa

aMaMaMZ

aaQ

)()(exp\

},{, ijik

ikijii aMa


144 2

5

13M

14M

15M

12M

33

26M

144

5

13M

14M

15M

1233

2

6

27M

88

77

28M

115114

1131121

11

1

aMaM

aMaMZ

aQ

2282272262112

11511411312

2112

,

1,

aMaMaMaa

aMaMaMZ

aaQ

Extremum

Condition 0,Bethe

iji

ii

QQLxQ 0,

, Bethe

iji

jiij

QQLxxQ




144 2

5

13M

14M

15M

12M

33

4412W

1

5

13M

14M

15M

33

26M

2

6

27M

88

77

28M

,121 QQ

115114

1131121

11

1

aMaM

aMaMZ

aQ

2282272262112

11511411312

2112

,

1,

aMaMaMaa

aMaMaMZ

aaQ

1514

1312

21

,

MM

M

M

Message Update Rule




15141312

15141312

21 ,

,

MMM

MMM

M

1

33

44 2

5

13M

14M

15M

21M

144

5

33

2

6

88

77

2a

144 2

5

33

=

Message Passing Rule of Belief Propagation

It corresponds to Bethe approximation in the statistical mechanics.




1 1 \

1 \

,

,

jikikij

jikikij

ji M

M

M



))tanh()(tanh(arctanh\

jik

ikji hJ

jijiM exp

))3tanh()(tanh(arctanh hJ

ji Translational Symmetry


Summary

Statistical Physics and Information TheoryProbabilistic Model of FerromagnetismMean Field TheoryGibbs Distribution and Free EnergyFree Energy and Kullback-Leibler DivergenceInterpretation of Mean Field Approximation as Information TheoryInterpretation of Bethe Approximation as Information Theory

physical fluctuomatics applied stochastic process 6th graphical model and physical model

Documents

oxford university press

physical models

lot of molecules

othersome physical concepts

physical viewpoint effective

graphical model

physicsa lot of elements

design of computational