phase transitions in discrete structures

Phase transitions in discrete structures

Amin Coja-Oghlan

Goethe University Frankfurt

Overview

1 The physics approach. [following Mezard, Montanari ’09]I Basics.I Replica symmetry (“Belief Propagation”).I The 1RSB cavity method.

2 “Classical” rigorous work.I The second moment method.I “Quiet planting”.

3 A physics-inspired rigorous approach.I The Kauzmann transition.I The free entropy in the 1RSB phase.

4 Random k-SAT.I A rigorous Belief Propagation-based approach.

Amin Coja-Oghlan (Frankfurt) Random k-SAT 2 / 40

The Boltzmann distribution

Let X be a finite set of spins and let N be a “large” integer.

Consider a probability distribution on XN defined by

µ(x) =1

Z

M∏a=1

ψa(x), with Z =∑x∈XN

M∏a=1

ψa(x).

Assume that ψa depends only on the components ∂a ⊂ [N] of x .

Thus, ψa(x) = ψa(x∂a).

We call µ the Boltzmann distribution and Z the partition function.


The factor graph

Suppose we are given the Boltzmann distribution

µ(x) =1

Z

M∏a=1

ψa(x∂a).

We set up the bipartite factor graph.I Square vertices: the factors ψa, a = 1, . . . ,M.I Round vertices: the variables x1, . . . , xM .I Conncect each factor ψa with all variables xi , i ∈ ∂a.


The factor graph

Example: the 1-dimensional Ising model

There are N “sites” with spins in X = −1, 1.For any a ∈ 1, . . . ,N − 1 we define

ψa(x) = exp(βxaxa+1) (β > 0).

Hence, ∂a = a, a + 1.

This yields µ(x) ∝∏M

a=1 exp(βxaxa+1) = exp[β∑N−1

a=1 xaxa+1

].


The factor graph

Example: the Potts antiferromagnet

Let X = [K ] = 1, 2, . . . ,K.Let G = (V ,E ) be a graph on V = [N].

For an edge e joining vertices i , j in G let

ψe(x) = exp[−β · 1xi=xj

].

This gives rise to

µ(x) =∏e∈E

ψe(x) = exp [−β ·#monochromatic edges] .


The free entropy

Generally, the partition function Z scales exponentially with N.

Therefore, we are interested in the free entropy 1N ln Z .

In fact, we care mostly about the theormdynamic limit N →∞, i.e.,

limN→∞

1

Nln Z ,

the free entropy density.

I Does this limit exist?I If so, can we compute/approximate it?I Is the limit an analytic function, or are there phase transitions?

Example: the Ising model

1-dim: the free entropy density is analytic, i.e., no phase transition.

2-dim: there is a phase transition. [Onsager 1949]


The free entropy

Generally, the partition function Z scales exponentially with N.

Therefore, we are interested in the free entropy 1N ln Z .

In fact, we care mostly about the theormdynamic limit N →∞, i.e.,

limN→∞

1

Nln Z ,

the free entropy density.I Does this limit exist?I If so, can we compute/approximate it?I Is the limit an analytic function, or are there phase transitions?

Example: the Ising model

1-dim: the free entropy density is analytic, i.e., no phase transition.

2-dim: there is a phase transition. [Onsager 1949]


Disordered systems

Two levels of randomness

The Boltzmann distribution itself is random.

Hence, the free entropy density becomes

limN→∞

E

[1

Nln Z

].

Example: the Edwards-Anderson model

With x as in the 1-dim Ising model, we let

µ(x) ∝ exp

[β

N−1∑i=1

Jixixi+1

].

The Ji are independent Gaussian random variables.


Disordered systems

Example: the diluted mean-filed Potts antiferromagnet

Let X = 1, . . . ,K for some “small” K ≥ 2.

Consider a random graph G (N,M) on N vertices with M edges.

For any edge e of G (N,M) joining vertices i , j , define

ψe(x) = exp[−β · 1xi=xj

].

The Boltzmann distribution

µ(x) ∝∏e

ψe(x) = exp [−β ·#monochromatic edges]

is random as it depends on the graph G (N,M).

New parameter: the denisty α = M/N.


Disordered systems

Example: coloring random graphs

Let X = 1, . . . ,K for some “small” K ≥ 3.

For any edge e of G (N,M) joining vertices i , j , define

ψe(x) = exp[−∞ · 1xi=xj

]= 1xi 6=xj [“zero temperature”].

Thenµ(x) ∝

∏e

ψe(x) = 1no edge is monochromatic.

is the uniform distribution over proper K -colorings of G (N,M).


Disordered systems

Question [Erdos, Renyi 1960]

Is there a phase transition in this model (in terms of α = M/N)?


Disordered systems

Example: random k-SAT

Let X = 0, 1 and fix some “small” K ≥ 3.

Think of x1, . . . , xN as Boolean variables.

Let Φ be an expression of the form

(x1 ∨ x17 ∨ · · · ∨ x29)︸︷︷︸k-clause Φ1

∧ (x11 ∨ x2 ∨ · · · ∨ x1)︸︷︷︸k-clause Φ2

∧ · · ·

with M “clauses”, chosen uniformly at random.

For i = 1, . . . ,M let

ψi (x) = 1clause Φi evaluates to true.

Then µ(x) =∏M

i=1 ψi (x) is uniform over satisfying assignments.


Belief Propagation

Goals

To compute the free entropy on trees.I the Belief Propagation equations.I the Bethe free entropy.

The replica symmetric ansatz.I Belief Propagation as a distributional fixed point equation.I application to “diluted mean-field models”.



Let X = ±1, x ∈ XN and

µ(x) ∝ exp

[β

N−1∑i=1

xixi+1 + βBN∑i=1

xi

].

Goal: to compute the marginal µ(xi ) of xi .



For j ∈ [N] define two distributions ν→j , νj→ on X by

ν→j(xj) ∝∑

x1,...,xj−1∈Xexp

[β

j−1∑i=1

xixi+1 + βB

j−1∑i=1

xi

],

νj←(xj) ∝∑

xj+1,...,xN∈Xexp

β N−1∑i=j

xixi+1 + βBN∑

i=j+1

xi

.

Thenµ(xj) ∝ ν→j(xj) · exp(βBxj) · νj←(xj).

Moreover, we have the recurrence

ν→j+1(xj+1) =∑xj∈X

ν→j(xj) exp [βxjxj+1 + βBxj ] .



For j ∈ [N] define two distributions ν→j , νj→ on X by

ν→j(xj) ∝∑

x1,...,xj−1∈Xexp

[β

j−1∑i=1

xixi+1 + βB

j−1∑i=1

xi

],

νj←(xj) ∝∑

xj+1,...,xN∈Xexp

β N−1∑i=j

xixi+1 + βBN∑

i=j+1

xi

.Then

µ(xj) ∝ ν→j(xj) · exp(βBxj) · νj←(xj).

Moreover, we have the recurrence

ν→j+1(xj+1) =∑xj∈X

ν→j(xj) exp [βxjxj+1 + βBxj ] .



We have ν→j+1(xj+1) =∑

xj∈X ν→j(xj) exp [βxjxj+1 + βBxj ] .

Setting

u→j =1

2βln

ν→j(1)

ν→j(−1),

we can rephrase the above as

u→j+1 = f (u→j + B) with f (x) = β−1 atanh(tanh(β) tanh(βx)).

The function f has a unique fixed point u∗.

Hence, for j sufficiently far from the boundary, we find

µ(xj) ∼ tanh(β(2u∗ + B)).


Belief Propagation on trees

Suppose that the factor graph of µ(x) is a tree.

Generalising the above example, we define messages recursively by

ν(t+1)j→a (xj) ∝

∏b∈∂j\a

ν(t)b→j(xj),

ν(t)b→j(xj) ∝

∑x∂b\j

ψb(x∂b)∏

i∈∂b\j

ν(t)i→b(xi ).

I t counts the number of recursive steps,I ∂j is the neighborhood of j ,I the sum ranges over X ∂b\j , with xj is given upfront.

Define the Belief Propagation marginal of variable xj as

ν(t)(xj) ∝∏b∈∂j

ν(t−1)b→j (xj).



Theorem

Assume that the factor graph is a tree of diameter T .

1 The BP equations have a unique fixed point ν∗.

2 For t > T we have ν(t) = ν∗, regardless of the initialisation.

3 For all variables xj we have

ν∗(xj) = µ(xj),

i.e., the BP marginals coincide with the Boltzmann marginals.



Theorem (the Bethe free entropy)

Given the messages ν, define

F (ν) =M∑a=1

Fa(ν) +N∑i=1

Fi (ν)−∑(i ,a)

Fia(ν), where

Fa(ν) = ln∑x∂a

ψa(x∂a)∏i∈∂a

νi→a(xi ),

Fi (ν) = ln∑xi

∏b∈∂i

νb→i (xi ),

Fia(ν) = ln∑xi

νi→a(xi )νa→i (xi ).

If the factor graph is a tree, then ln Z = F (ν∗).



On trees, BP converges to its unique fixed point.

This fixed point yieldsI the correct marginals,I the free entropy.

Example: the 1-dim Ising model.


Belief Propagation on infinite trees

Question

Does this formalism survive the thermodynamic limit?

That is, does it extend to infinite trees?

Examples of infinite trees

The infinite regular tree.I Each variable/factor node has the same number of outgoing edges.I The factors ψa are identical.

Infinite random trees.I The numbers of outgoing edges is are independent Poisson variables.I The factors ψa might have some randomness, too.

Generally, we assume that sub-trees are identically distributed.



The distributional BP equations

We interpret the Belief Propagation equations

νj→a(xj) ∝∏

b∈∂j\a

νb→j(xj),

νb→j(xj) ∝∑x∂b\j

ψb(x∂b)∏

i∈∂b\j

νi→b(xi ).

as distributional fixed point equations.

That is, ifI ψb, |∂b|, |∂j | are i.i.d. from the distribution defining the tree,I the νi→b are i.i.d. from a distribution ν∗ over distributions,

then νj→a has distribution ν∗.

The Bethe free entropy is determined by ν∗.



Example: coloring regular trees

Consider the graph K -coloring problem on the d-ary tree.

By symmetry, all the marginals are identical, i.e.,

νj→a(xj) = 1/K for all j , xj .

Thus, ν∗ is the measure concentrated on the uniform distribution.



Example: random K -SAT

Each variable node has a Po(α) number of children.

The factors ψa are i.i.d. random clauses of length K .

The fixed point distribution ν∗I is not known to exist,I but is conjectured to be highly non-trivial and not discrete,I based on numerical evidence (“population dynamics”).

This mirrors the asymmetric combinatorics of the problem.


The replica symmetric ansatz

Diluted mean-field models

Think of the factor graphs in models such asI random graph K -coloring,I random K -SAT.

The factor graph is an Erdos-Renyi-like sparse random graph.

Thus, roughly speaking,I the vertices are bounded degree,I the shortest cycle has length Ω(ln N).

In other words, locally the graph looks like the infinite random tree!



The replica symmetric “solution”

Let ν∗ be the BP fixed point on the ∞ tree.

Let φ∗ be the resulting Bethe free entropy.

Then the replica symmetric prediction is that

φ∗ = limN→∞

E[ln Z ]

N,

with Z the partition function in, e.g., random K -SAT.

Hypotheses

The fixed point ν∗ exists and is only one “relevant” fixed point.

There is correlation decay.



Example: ferromagnetic Ising on sparse random graphs [DM ’08]

The replica symmetric solution is correct.

Paramagnetic/ferromagnetic phase transition.



Example: coloring random graphs

Let K = #colors, α = M/N density of the graph.

Then Z = #K -colorings of G (N,M).

According to the replica symmetric solution,

1

NE[ln Z ] ∼ 1

NlnE[Z ] = ln K + α ln(1− 1/K ).

The col/uncol transition occurs at α = (K − 12) ln K + oK (1).

Rigorous work: [ACO, Vilenchik ’13]I The r.s. solution is correct only for

α < (K − 1

2) ln K − ln 2 + oK (1).

I The col/uncol transition occurs at α ≤ (K − 12 ) ln K − 1

2 + oK (1).


Multiple Belief Propagation fixed points

Random factor graphs

In random graph K -coloring, the factor graph is similar to G (N,M).

Thus, it’s far from being a tree.

In effect, there can be multiple BP fixed points.

These mirror long range correlations.


Multiple Belief Propagation fixed pointsxi

w

The reconstruction problem

Suppose µ is defined by a factor graph G .

Pick a variable xi and let Gω be the depth-ω neighborhood.

What is the impact of the boundary on xi?



Correlation decay: Gibbs uniqueness

Let yω be a boundary configuration.

The worst-case influence of the boundary can be cast as

E[supyω|µ (xi |yω)− µ(xi )|].

Here the expectation is over the choice of µ.

Iflim

ω→∞lim

N→∞E[sup

yω|µ (xi |yω)− µ(xi )|] = 0,

we say there is Gibbs uniqueness.

Observe that this is a purely local property.



Theorem [Achlioptas, ACO ’08]

If the density satisfies

α = M/N >

(1

2+ ε

)K ln K ,

then in random graph K -coloring reconstruction is possible.

In fact, suppose we initialise BP with one random K -coloring.

Then for Ω(n) variables xi the BP marginals satisfy ν∗(xi ) = 1.

Thus, these variables are “locally frozen”.

Interpretation: µ decomposes into a large number of clusters.


The 1RSB cavity method

1-step replica symmetry breaking

In the above scenario, µ is a convex combination

µ =N∑i=1

wiµi

of probability measures µi (corresponding to “clusters”).

Here wi ∝ exp(Fi · N), with Fi the free entropy of µi .

Key idea: perform Belief Propagation on the level of the µi !

The parameter β is replaced by the Parisi parameter y .



Idea

A Boltzmann distribution on the µi .

This distribution is defined by

Py (µi ) ∝ w yi .

In terms of the complexity function

Σ(φ) =1

Nln # i ∈ [N ] : Fi ∼ φ ,

the partition function Ξ(y) of ξy satisfies

Ξ(y) = yφ+ Σ(φ), with φ s.t.∂Σ

∂φ= −y .



Belief Propagation one level up

What is the factor graph of Py?I The “old” messages νi→a, νa→i are the variables.I There are factor nodes enforcing the BP equations.I Additional factor nodes implement the Parisi parameter y .I (This is possible due to Bethe’s formula.)

On this factor graph, write the Belief Propagation equations.I 1RSB cavity equations.I Complication: the spins are continuous.

The BP fixed point for Py yields Σ(φ).I Analogous to the Bethe free entropy.

The 1RSB equations can be viewed as distributional equations.I the thermodynamic limit.

Setting y = 0 yields the Survey Propagation equations.



Table 1. Critical connectivities for the dynamical, con-densation and satisfiability transitions in k-SAT and q-COLSAT !d !c !s[11] COL ld[16] lc ls[14]k = 4 9.38 9.547 9.93 q = 4 9 10 10k = 5 19.16 20.80 21.12 q = 5 14 14 15k = 6 36.53 43.08 43.4 q = 6 18 19 20

unsatisfiable with high probability2 [10]. For k-SAT, it is known that!s(2) = 1. A conjecture based on the cavity method was put forwardin [6] for all k ! 3 that implied in particular the values presented inTa-ble 1 and !s(k) = 2k log 2" 1

2(1+log 2)+O(2!k) for large k [11].

Subsequently it was proved that !s(k) ! 2k log 2"O(k) confirmingthis asymptotic behavior [12]. An analogous conjecture for q-coloringwas proposed in [13] yielding, for regular random graphs [14], the val-ues reported inTable 1 and ls(q) = 2q log q"log q"1+o(1) for largeq (according to our convention, random graphs are whp uncolorable ifl ! ls(q)). It was proved in [15, 12] that ls(q) = 2q log q"O(log q).

Evenmore interesting and challenging are phase transitions in thestructure of the set S # X N of solutions of rCSP’s (‘structural’ phasetransitions). Assuming the existence of solutions, a convenient wayof describing S is to introduce the uniform measure over solutionsµ(x):

µ(x) =1

Z

MY

a=1

"a(xia(1), . . . , xia(k)) , [1]

where Z ! 1 is the number of solutions. Let us stress that, since Sdepends on the rCSP instance, µ( · ) is itself random.

We shall now introduce a few possible ‘global’ characterizationsof the measure µ( · ). Each one of these properties has its counter-part in the theory of Gibbs measures and we shall partially adopt thatterminology here [17].

In order to define the first of such characterizations, we let i $ [N ]be a uniformly random variable index, denote as x! the vector of vari-ables whose distance from i is at least #, and by µ(xi|x!) the marginaldistribution of xi given x!. Then we say that the measure [1] satisfiesthe uniqueness condition if, for any given i $ [N ],

E supx!,x!

!

X

xi"X

˛µ(xi|x!) " µ(xi|x#

!)˛% 0 . [2]

as # % & (here and below the limitN % & is understood to be takenbefore # % &). This expresses a ‘worst case’ correlation decay con-dition. Roughly speaking: the variable xi is (almost) independent ofthe far apart variables x! irrespective is the instance realization and thevariables distributionoutside the horizonof radius #. The threshold foruniqueness (above which uniqueness ceases to hold) was estimatedin [9] for random k-SAT, yielding !u(k) = (2 log k)/k[1 + o(1)](which is asymptotically close to the threshold for the pure literalheuristics) and in [18] for coloring implying lu(q) = q for q largeenough (a ‘numerical’ proof of the same statement exists for small q).Below such thresholds BP can be proved to return good estimates ofthe local marginals of the distribution [1].

Notice that the uniqueness threshold is far below the SAT-UNSATthreshold. Furthermore, several empirical studies [19, 20] pointed outthat BP (as well as many other heuristics [4, 5]) is effective up tomuchlarger values of the clause density. In a remarkable series of papers[21, 6], statistical physicists argued that a second structural phasetransition is more relevant than the uniqueness one. Following thisliterature, we shall refer to this as the ‘dynamic phase transition’ (DPT)and denote the corresponding threshold as !d(k) (or ld(q)). In orderto precise this notion, we provide here two alternative formulations

!d,+ !d !c !s

Fig. 2. Pictorial representation of the different phase transitions in the set ofsolutions of a rCSP. At!d,+ some clusters appear, but for!d,+ < ! < !d theycomprise only an exponentially small fraction of solutions. For!d < ! < !c thesolutions are split among about eN!" clusters of size eNs" . If !c < ! < !s

the set of solutions is dominated by a few large clusters (with strongly fluctuatingweights), and above !s the problem does not admit solutions any more.

corresponding to two distinct intuitions. According to the first one,above !d(k) the variables (x1, . . . , xN) become globally correlatedunder µ( · ). The criterion in [2] is replaced by one in which far apartvariables x! are themselves sampled from µ (‘extremality’ condition):

EX

x!

µ(x!)X

xi

|µ(xi|x!) " µ(xi)| % 0 . [3]

as # % &. The infimum value of ! (respectively l) such thatthis condition is no longer fulfilled is the threshold !d(k) (ld(k)).Of course this criterion is weaker than the uniqueness one (hence!d(k) ! !u(k)).

According to the second intuition, above !d(k), the measure[1] decomposes into a large number of disconnected ‘clusters’.This means that there exists a partition Ann=1...N of X N (de-pending on the instance) such that: (i) One cannot find n suchthat µ(An) % 1; (ii) Denoting by $"A the set of configurationsx $ X N\Awhose Hamming distance fromA is at mostN%, we haveµ($"An)/µ(An)(1 " µ(An)) % 0 exponentially fast inN for all nand % small enough. Notice that the measure µ can be decomposed as

µ( · ) =NX

n=1

wn µn( · ) , [4]

where wn ' µ(An) and µn( · ) ' µ( · |An). We shall always referto An as the ‘finer’ partition with these properties.

The above ideas are obviously related to the performance of algo-rithms. For instance, the correlation decay condition in [3] is likely tobe sufficient for approximate correctness of BP on random formulae.Also, the existence of partitions as above implies exponential slowingdown in a large class of MCMC sampling algorithms3.

Recently, some important rigorous results were obtained support-ing this picture [22, 23]. However, even at the heuristic level, severalcrucial questions remain open. The most important concern the dis-tribution of the weights wn: are they tightly concentrated (on anappropriate scale) or not? A (somewhat surprisingly) related ques-tion is: can the absence of decorrelation above !d(k) be detected byprobing a subset of variables bounded in N?

SP [6] can be thought as an inference algorithm for a modifiedgraphical model that gives unit weight to each cluster [24, 20], thustilting the original measure towards small clusters. The resulting per-formances will strongly depend on the distribution of the cluster sizeswn. Further, under the tilted measure, !d(k) is underestimated be-cause small clusters have a larger impact. The correct value was neverdetermined (but see [16] for coloring). The authors of [25] undertook

3One possible approach to the definition of a MCMC algorithm is to relax the constraints bysetting#a(· · · ) = " instead of 0 whenever the a-th constraint is violated. Glauber dynamicscan then be used to sample from the relaxed measure µ"( · ).

2 www.pnas.org — — Footline Author

The geometry of the Boltzmann distribution

The replica symmetric phase.

Dynamic 1RSB.

Static 1RSB (aka condensation, Kauzmann transition).



Example: for what d is the random d-regular graph K -colorable?

Setting y = 0, we consider the distribution

P0(µi ) =1

N, N = #clusters.

Suppose that each µi is characterized by frozen variables.I Encode µi as a map ζi : [N]→ 1, 2, . . . ,K , ∗.I ∗ = unfrozen (“joker color”).

This gives rise to SP messages:I Qi→j(xi ) =P0-probability that w/out j , i is frozen to color xi .I Qi→j(xi ) =P0-probability that w/out j , i is unfrozen.




We have the fixed point equations

Qi→j(xi ) =

∑x∂i\j∈N (xi )

∏k∈∂i\j Qk→i (xk)∑

x∂i\j∈D∏

k∈∂i\j Qk→i (xk)(xi ∈ [K ]).

I D contains all (c1, . . . , cd−1) such that [K ] 6⊂ c1, . . . , cd−1.I N (x) contains all (c1, . . . , cd−1) ∈ D with [K ] \ x ⊂ c1, . . . , cd−1.

Let Qi→j(∗) = 1−∑

xiQi→j(xi ).

Simplification: it might seem reasonable to assume thatI Qi→j(xi ) = a for some fixed 0 ≤ a ≤ 1/q and all xi ∈ [K ].I Qi→j(∗) = 1− Ka.I The above equation becomes

a =

∑q−1r=0 (−1)r

(K−1r

)(1− (r + 1)a)d−1∑q−1

r=0 (−1)r(

qr+1

)(1− (r + 1)a)d−1

.




We obtain the complexity ( 1N ln #clusters)

Σ = ln

[q−1∑r=0

(−1)r(

q

r + 1

)(1− (r + 1)a)d

]− d

2ln(1− Ka2)

with a the solution to

a =

∑q−1r=0 (−1)r

(K−1r

)(1− (r + 1)a)d−1∑q−1

r=0 (−1)r( qr+1

)(1− (r + 1)a)d−1

.

Now, K -colorability should occur iff Σ > 0.

Asymptotically, this yields a threshold of

dK−col = (2K − 1) ln K − 1 + oK (1).


Conclusion

The cavity method as a non-rigorous formalism.I free entropy,I phase transitions.

There are two variants:I the replica symmetric version (“Belief Propagation”).I the 1RSB version (“Survey Propagation”).

Can we develop a rigorous foundation?


phase transitions in discrete structures

Documents