from complex to simple: hierarchical free-energy landscape...

From Complex to Simple:Hierarchical Free-energy Landscape

Renormalized in Deep Neural Networks

Hajime Yoshino

Cybermedia Center & Department of Physics,Osaka Univ.

Deep Learning and Physics 2019@YITP

H. Yoshino, arXiv1910.09918 submitted to SciPost Phys

referee round deadline Nov. 26 th… comments are welcome!

[Q] Why deep neural networks works?

data size # of parameterse.g.

over fittingCommon sense

1. poor generalization

2. many local minimum learning is difficult (glassy dynamics)

Empirical observations on deep networks

1. generalization is not bad

2. learning is not too difficult

Why???

Why???

108106 ⌧

G. Carleo, et. al, arXiv:1903.10563v1

History: Spinglasses, Hopfield model, error correcting codes,....

• Edwards-Anderson model(1975)

Quenched random spin-spin interactions

H = ��

Jijsisj

A lot of energetic degeneracies due to frustration Ground state: disordered

Jij = 0 J2ij = J2

？

？

si = ±1 (i = 1, 2, . . . , N)

• Hopfield model (1982) : associative memory

Firing of neurons

si = ±1 Jij = 1pM

MX

µ=1

⇠µi ⇠µjJij

synaptic weight

⇠µi = ±1embedded patterns

Hebb rule

Jij < 0

Jij > 0

• error correcting code： a statistical inference problem send receiver tries to reconstruct siJij

Sourlas (1989)

… and back to p-spins… but with spin componentsand without quenched disorder

H. Yoshino, SciPost Phys. 4 (6), 040 (2018)

S0S1

S2

S3 S4

S0

S1

S2S3

S4

S1S2

S0

a) b) c)

M ! 1AAACcXichVHNSiNBEP4yrj+b9Se6F2Uvg0ERhFCjosaT6MWL4M9GBSMyM3Zi42R6mOkEY8gL7APoYQ/uCoLLPoYXX8CDjyB7zMJe9mBlEhERtRq6qr6qr/rrbifwZKSJ7hJGx4fOru6ej8lPvX39A6nBoa1IlUNX5FzlqXDHsSPhSV/ktNSe2AlCYZccT2w7R8vN+nZFhJFU/lddDcReyS76siBdWzOUXzXzWpl56Rd0dT+VpkyWrOysZb4MrAzFlkbb1lTqCnkcQMFFGSUI+NAce7AR8dqFBULA2B5qjIUcybguUEeSuWXuEtxhM3rEe5Gz3Tbqc96cGcXsY+5QzFQ8pQ4TY3RLv6hBN/Sb7un/q9Nq8ZSmmip7p8UVwf7At+HNf++ySuw1Dp9Yb6rWKGA+VitZfRAjzXu4LX7l5KyxubAxVhunC/rD+n/SHV3zDfzKX/dyXWx8R5K/4PGdzdeDramMNZ2ZWp9JLy61P6MHXzCKCX7xOSxiBWvI8bkBTnGOH4mGMWKYxmir1Ui0OZ/xzIzJByYljxU=

1999 20011987 1991 2008

From spin glass to structural glass

Kirkpatrick-Thirumalai Wolyness (1989)

Franz-Parisi (1995), Monasson (1995), Mezard-Parisi (1999)

Parisi-Zamponi (2010), Charbonneau-Kurchan-Parisi-Urbani-Zamponi (2014)

Yoshino-Mezard (2010), Yoshino-Zamoponi (2014), Rainone-Urbani-Yoshinno-Zamponi (2015)

d ! 1AAACcXichVHNSiNBEP4y6290Na4XxcuQoAgLoUZF1z3JevHoz0YFI2Fm7GjjZHqY6YTNBl/AB9DDHvwBQfExvOwL7CGPIB6z4MWDlUlEliVaDV1VX9VX/XW3E3gy0kT1hPGhq7unt68/OTD4cWg4NfJpM1Ll0BU5V3kq3HbsSHjSFzkttSe2g1DYJccTW87hcrO+VRFhJJX/XVcDsVuy931ZlK6tGcrvmXmtzLz0i7paSGUou0jW4rxl/h9YWYotg7atqtQ18tiDgosyShDwoTn2YCPitQMLhICxXdQYCzmScV3gCEnmlrlLcIfN6CHv+5zttFGf8+bMKGb/4A7FTMVTjmBikv7QDTXoN93SPT11nFaLpzTVVNk7La4ICsPHYxuP77JK7DUOXllvqtYo4kusVrL6IEaa93Bb/MrP08bG1/XJ2hRd0gPrv6A63fEN/Mpf92pNrP9Ckr/g5Z3NzsHmTNaazc6szWWWvrU/ow8TSGOaX3wBS1jBKnJ8boATnOE80TDGDdNIt1qNRJszin/M+PwMVVCPLA==

p-spin to RFOT

more on replicas

Jin-Yoshino (2017), Jin-Urbani-Zamponi-Yoshino (2018)

Soft Jammed Materials – Eric R. Weeks 23

3. Other soft materials Now that I’ve introduced colloids, let’s discuss other soft systems which resemble colloids to varying degrees. 3.1 Emulsions Emulsions are similar to colloids, but rather than solid particles in a liquid, they consist of liquid droplets of one liquid, mixed into a second immiscible liquid; for example, oil droplets mixed in water. Surfactant molecules are necessary to stabilize the droplets against coalescence which is when two droplets come together and form a single droplet. A cross-section of an emulsion is shown in Fig. 3.1, and a sketch showing a droplet with the surfactants is shown in Fig. 3.2. Mayonnaise is a common example of an emulsion, made with oil droplets in water, stabilized by egg yolks as the surfactant, with extra ingredients added for taste.

Fig. 3.1. Confocal microscope image of an emulsion. The droplets (dark) are dodecane, a transparent oil. The space between the droplets is filled with a mixture of water and glycerol, designed to match the index of refraction of the dodecane droplets. The droplets are outlined with a fluorescent surfactant. The hazy green patches are free surfactant in solution, or else the tops or bottoms of other droplets. (Picture taken by ER Weeks and C Hollinger.)

Fig. 3.2. Sketch of an emulsion droplet. Not to scale: typically the surfactants are tiny molecules, whereas the droplet is micron-sized. (Sketch by C Hollinger.)

10 µm

Glass order parameter with replicas

✏ab =@G[Q̂]

@Qab

��Q̂=Q̂SP

= 0

G[Q̂] = F [✏̂] +NX

ab

✏abQab

replica 1 replica 2

We are interested with lim✏ab!0+

limN!1

Qab

crystal

H[✏] =nX

a=1

H(sa)�X

a,b

✏ab

NX

i=1

Sai S

bi

replicas a=1,2,…nsymmetry breaking field

Qab = 0

Qab > 0

liquidglassspontaneous breaking of ergodicity

Explicit RSB : Parisi-Virasoro (1989)

��F [✏̂] = lnTrSe��H[✏̂]

Qab ⌘1

N

NX

i=1

DSai S

bi

E= � 1

N

@F [✏̂]

@✏abOverlap

Replica symmetry braking and ultra-metricity

overlap matrix

G. Parisi (1979)

first found in the SK model for spinglass

Qab =

breaking of ergodicity & permutation symmetry

Edwards-Anderson (EA) order parameter Self-overlap qEA = lim

b!aQab

AAACh3ichVFNTxRBEH0MoLCiLHgx8TJxg/G01LJmARMTwJh4ZIEFEpZMZoYGOsxMDzO9G3EyVw/+AQ6eJCHR4FX/gBf/gAd+gvEIiRcP1nwkxBilOul+/ape9etuJ/RkrInOB4zBoeEbN0dGK7fGbt8Zr05MrseqF7mi4ypPRZuOHQtPBqKjpfbEZhgJ23c8seEcPMvyG30RxVIFa/ooFNu+vRfIXenamimrah5aSTfyzeeL6dOuJ30rccyuVqadmm0rsZ3UqtaoPt+iBs2YBWi1StBsmo065VFDGcuq+h5d7EDBRQ8+BAJoxh5sxDy20AAhZG4bCXMRI5nnBVJUWNvjKsEVNrMHPO/xbqtkA95nPeNc/ZIrFCsVd0lhYoq+0Qe6oK90Rt/p1z+7JXmXzM0Rr06hFaE1/ube6s9rVT6vGvtXqv+61tjFXO5WsvswZ7J7uIW+/+r4YvXJylTykE7oB/t/R+f0hW8Q9C/d07ZYeYtK8QVZXD3432B9pt5o1pvtx7WFpfIzRnAfD/CIX3wWC3iBZXT43Nf4iE/4bIwa00bLmCtKjYFScxd/hLH4G9UUlos=

a b c

distance

overlap

Rammal-Toulouse-Virassoro (1986) Toulouse-Dehaene-Changeaux (1986) Q(a, b) = min(Q(a, c), Q(b, c))ultra-metricity

“Disorder-free” vector p-spin models on tree

“gap”

p = 3c = 4

H = �X

⌅V (r⌅) .

r⌅ = � �1pM

MX

µ=1

Sµ1(⌅)Sµ2(⌅) · · ·S

µp(⌅)

Locally tree like lattice with connectivity c

Hamiltonian

S0S1

S2

S3 S4

S0

S1

S2S3

S4

S1S2

S0

a) b) c)

N⌅ = Nc/p = NM↵/p < Np/p!# of factor nodes

c = ↵M

|Si2| = Mi = 1, 2, . . . , NSi = (S

1i , S

2i , . . . , S

Mi )

continuous or Ising Sµi = ±1

“Inter-mediate sparseness “: high connectivity but not “global coupling”


2. A Potts anti-ferromagnet on random graphsIt is immediate to realize that the q-coloring problem is equivalent to the question of determiningif the ground-state energy of a Potts anti-ferromagnet on a random graph is zero or not [8].Consider indeed a graph G = (V, E) defined by its vertices V = {1, . . . , N} and edges (i, j) ∈ Ewhich connect pairs of vertices i, j ∈ V; and the Hamiltonian

H({s}) =∑

(i,j)∈E

δ(si, sj) . (1)

With this choice there is no energy contribution for neighbors with different colors,but a positive contribution otherwise. The ground state energy (the energy at zerotemperature) is thus zero if and only if the graph is q-colorable. This transforms thecoloring problem into a well-defined statistical physics model. Usually, two types of randomgraphs are considered: in the c−regular ensemble all points are connected to exactlyc neighbors, while in the Erdős-Rényi case the connectivity has a Poisson distribution.

Figure 1. Example: a proper 3-coloring of a small graph.

3. Cavity method: Warnings, Beliefs and SurveysOver the last few years, a number of studies haveinvestigated CSPs following the adaptation of the so-calledcavity method [2] to random graphs [4, 9]. It is a powerfulheuristic tool —whose exactness is widely accepted buthas still to be rigorously demonstrated— equivalent to thereplica method of disordered systems [2]. Its main idealies in the fact that a large random graph is locally tree-like, and that an iterative procedure known in physics asthe Bethe-Peirls method can solve exactly any model ona tree (such models are often qualified as “mean field”in physics). Interestingly, it was realized [10] that anequivalent formalism has been developed independently incomputer science [11], where it is called Belief Propagation(BP, which conveniently enough, may also stands forBethe-Peirls). Defining ψi→jc (c = 1, .., q) as the probability that the spin i has color c in absenceof the spin j (the “belief” that the spin j has on the properties of the spin i), BP reads

ψi→jc =1

Zi→j0

∏

k∈N(i)\{j}

(

1 − ψk→ic)

(2)

where Zi→j0 is a normalization constant and the notation k ∈ N(i) \ {j} means the set ofneighbors of i except j. From a fixed point of these equations, the complete beliefs in presenceof all spins can be also computed. They give, for each vertex, the probability of each color fromwhich other quantities, as for instance the number of solutions, can be computed. A simplerformalism, called Warning Propagation (WP), restricts itself to frozen variables (i.e. to variablesfor which only one color can satisfy the constraints). However, WP does not allow to computethe number of solutions, only their existence, but is definitely simpler to handle.

It was soon realized, however, that these methods developed for trees could not beused straightforwardly on all random graphs because of a non-trivial phenomenon calledclustering [9, 12] (for which rigorous results are now available, see [13]). Indeed, while forgraphs with very low connectivities all solutions are “connected” —in the sense that it is easywith a local dynamics to move from one solution to another— they regroup into a large number

standard discrete coloring

“Vectorial” constraint satisfaction problems

antiferromagnetic Potts model

H =X

i,j

�qi,qj

“continuous” version

θ

“repulsive” vectorial spin model

H =X

i,j

V

✓� � Si · Sjp

M

◆

V (r) = lim✏!1

✏r2✓(�r)

↵ = c/MSciPost Phys. 4, 040 (2018)

0

2

4

6

8

10

12

14

-0.4 -0.2 0 0.2 0.4

�

�

Liquid q = 0

glass q �= 0

Figure 6: The phase diagram of the p = 2 body hardcore model within the replicasymmetric (RS) ansatz: q > 0 RS solution emerges continuously at the lower curvewhich represents ↵ = ↵c(�) given by Eq. (235). The value of the order parametersaturates q! 1 approaching the upper solid line ↵= ↵j(�) given by Eq. (237), whichis the jamming line within the RS ansatz. The lower curve coincides with the AT lineabove which the RS solution becomes unstable. The dotted line is the jamming lineobtained by the continuous RSB solution which is discussed later.

function G(q) for the hardcore model is given by Eq. (230). Expanding G(q) up to order O(q2)we find,

0= G(q) = 1� ↵↵c(�)⇥1� 2q+ (2� 2x0 � r0)q2

⇤+O(q4) (313)

where

x0 ⌘�p2

r0 ⌘ r(x0) =⇥0(x0)⇥(x0)

(314)

The above equation can be solved for q to find,

q =12✏� 1

4

Å1+ x0 +

r02

ã✏2 +O(✏3) ✏⌘ ↵

↵c(�)� 1 (315)

Thus we find that q 6= 0 solution emerges at the critical point ↵ = ↵c(�), where the q = 0solution becomes unstable, and the EA order parameter q grows continuously increasing ↵.

In Fig. 6 we show the phase diagram for the p = 2 hardcore model within the RS ansatz.The glass transition line ↵= ↵c(�) is given by Eq. (235). The jamming line ↵= ↵j(�) is givenby Eq. (237).

In Fig. 7, we display an example of a set of solutions of the RS saddle point equation givenby Eq. (135) with Eq. (230) for a↵= 4 with varying�. As shown in the panel a), the glass orderparameter q emerges continuously at the critical point �c ⇠ 0.51 (determined by ↵c(�c) = 4,see Fig. 6) and increases by decreasing � and saturates to q = 1 approaching the jammingpoint �j ⇠ �0.47065 (determined by ↵j(�) = 4, see Fig. 6) There we also show the behaviorof the pressure given by Eq. (238) which diverges approaching the jamming and evolution ofg(r) given by Eq. (239) which develops a diverging contact peak at r = 0 approaching thejamming.

58

connectivityJamming (SAT-UNSAT)

continuous RSB:hierarchical clustering of solutions


M ! 1AAACcXichVHNSiNBEP4yrj+b9Se6F2Uvg0ERhFCjosaT6MWL4M9GBSMyM3Zi42R6mOkEY8gL7APoYQ/uCoLLPoYXX8CDjyB7zMJe9mBlEhERtRq6qr6qr/rrbifwZKSJ7hJGx4fOru6ej8lPvX39A6nBoa1IlUNX5FzlqXDHsSPhSV/ktNSe2AlCYZccT2w7R8vN+nZFhJFU/lddDcReyS76siBdWzOUXzXzWpl56Rd0dT+VpkyWrOysZb4MrAzFlkbb1lTqCnkcQMFFGSUI+NAce7AR8dqFBULA2B5qjIUcybguUEeSuWXuEtxhM3rEe5Gz3Tbqc96cGcXsY+5QzFQ8pQ4TY3RLv6hBN/Sb7un/q9Nq8ZSmmip7p8UVwf7At+HNf++ySuw1Dp9Yb6rWKGA+VitZfRAjzXu4LX7l5KyxubAxVhunC/rD+n/SHV3zDfzKX/dyXWx8R5K/4PGdzdeDramMNZ2ZWp9JLy61P6MHXzCKCX7xOSxiBWvI8bkBTnGOH4mGMWKYxmir1Ui0OZ/xzIzJByYljxU=

SciPost Phys. 4, 040 (2018)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

xq(

x)

x

1�

q(x)

a) b)

10-5

10-4

10-3

10-2

10-1

100

10-4 10-3 10-2 10-1 100

6.76.716.72

6.7226.7246.726

power law

Figure 9: The q(x) function of the hardcore model with p = 2,� = 0 for which ↵c = 1.5708.. and ↵j = 6.732... a)↵ = 1.6,1.8, 2.0,2.2, 2.4,2.6, 2.8,3.0, 4.0,5.0, 6.0,6.5, 6.6,6.7 from the bottomto the top. b) The straight line represents the power law fit ax� with = 1.4157,the same exponent as that for the hardspheres [16].

at T = 0 (hardcore limit). We obtained the result using the scheme explained in sec. 8.4.3together with the inputs for the hardcore case shown in sec. 10.2.1. As can be seen in Fig. 9a), we found the continuous RSB solution with nonzero q(x) function emerges continuouslystarting from the AT line ↵= ↵c(�) as expected.

Using the scheme explained in sec. 10.2.4 we obtained the jamming line ↵ = ↵j(�) of thehardcore model and the result is displayed in Fig. 8 at the bottom. It is also shown in Fig. 6where we can see that the RS ansatz overestimates the jamming line. Quite interestingly wefind in Fig. 8 that the two Gardner planes ’G1’ and ’G2’ merges onto the the jamming line in thezero temperature limit. The geometry of the phase diagram is very different from that of thehardspheres [68] where there is only one Gardner line. We analyzed the criticality of jammingq(x1)! 1 of the hardcore model (✏!1), i. e. ↵! ↵�J (�) at T = 0. As shown in Fig. 9 b),we find power law behavior 1� q(x)/ x� with the expected exponent = 1.4157... Thisconfirms the scaling argument presented in sec. 10.3.1 establishing that the jamming of thepresent model belong to the same universality class as that of the hardspheres [16].

The liquid phase q = 0 at T = 0 can be regarded as an easy SAT region where the space ofthe solutions to satisfy the hard constraints (✏!1), i. e. the manifold of the ground statesare continuously connected. The glass phase at T = 0 with 0 < q(x1) < 1 can be regarded ashard SAT phase where the manifold of the ground states splits into clusters. The major differ-ence with respect to the case of usual discrete coloring [32] is that the transition is continuousand the clustering is hierarchical reflecting the continuous RSB. The region ↵ > ↵j(�) is theUNSAT region where the hard constraint cannot be satisfied.

11 Conclusions

In the present paper we developed a family of exactly solvable large M -component vectorialIsing/continuous spin systems with p-body interactions which exhibit glass transitions by theself-generated randomness. We also established a connection between the disorder-free model

61

1� q(x) = x�

= 1.415726...

Same jamming criticality as hard spheres

and perceptron (p=1)

Franz-Parisi (2016), Franz-Parisi-Sevlev-

Urbani-Zamponi (2017)

clustering = glass transition

“Design space” of a multilayer-perceptron network

input outputhidden

S0,1AAACcnichVHLSsNAFD2N7/qquhHdVEvFhciNCIor0Y1LtfYBbSlJnGowTWKSFjX0B/yBLtxYQVD8DDf+gAs/QVwquHHhbRoQLeodmDlz5p47Z+aqtqG7HtFTROrq7unt6x+IDg4Nj4zGxsYzrlV1NJHWLMNycqriCkM3RdrTPUPkbEcoFdUQWfVos3WerQnH1S1zzzu1RbGiHJh6WdcUj6miX1DL8VS95NOCXC/FErRIQcQ7gRyCBMLYtmI3KGAfFjRUUYGACY+xAQUujzxkEGzmivCZcxjpwblAHVHWVjlLcIbC7BHPB7zLh6zJ+1ZNN1CfcIbFSour1BFHkh7pll7pge7omT5+reYHVVpuTnlV21phl0bPJ1Pv/6oqvHo4/FL96dpDGauBW53d2wHTeofW1tfOGq+ptd2kP0dX9ML+m/RE9/wCs/amXe+I3QtEuQXyzw/vBJmlRZnxznJifSNsRj+mMYt5/vEVrGML20jzvcdo4BLNyJs0Jc1IYeekSKiZwLeQFj4B/C+O7g==AAACcnichVHLSsNAFD2N7/qquhHdVEvFhciNCIor0Y1LtfYBbSlJnGowTWKSFjX0B/yBLtxYQVD8DDf+gAs/QVwquHHhbRoQLeodmDlz5p47Z+aqtqG7HtFTROrq7unt6x+IDg4Nj4zGxsYzrlV1NJHWLMNycqriCkM3RdrTPUPkbEcoFdUQWfVos3WerQnH1S1zzzu1RbGiHJh6WdcUj6miX1DL8VS95NOCXC/FErRIQcQ7gRyCBMLYtmI3KGAfFjRUUYGACY+xAQUujzxkEGzmivCZcxjpwblAHVHWVjlLcIbC7BHPB7zLh6zJ+1ZNN1CfcIbFSour1BFHkh7pll7pge7omT5+reYHVVpuTnlV21phl0bPJ1Pv/6oqvHo4/FL96dpDGauBW53d2wHTeofW1tfOGq+ptd2kP0dX9ML+m/RE9/wCs/amXe+I3QtEuQXyzw/vBJmlRZnxznJifSNsRj+mMYt5/vEVrGML20jzvcdo4BLNyJs0Jc1IYeekSKiZwLeQFj4B/C+O7g==AAACcnichVHLSsNAFD2N7/qquhHdVEvFhciNCIor0Y1LtfYBbSlJnGowTWKSFjX0B/yBLtxYQVD8DDf+gAs/QVwquHHhbRoQLeodmDlz5p47Z+aqtqG7HtFTROrq7unt6x+IDg4Nj4zGxsYzrlV1NJHWLMNycqriCkM3RdrTPUPkbEcoFdUQWfVos3WerQnH1S1zzzu1RbGiHJh6WdcUj6miX1DL8VS95NOCXC/FErRIQcQ7gRyCBMLYtmI3KGAfFjRUUYGACY+xAQUujzxkEGzmivCZcxjpwblAHVHWVjlLcIbC7BHPB7zLh6zJ+1ZNN1CfcIbFSour1BFHkh7pll7pge7omT5+reYHVVpuTnlV21phl0bPJ1Pv/6oqvHo4/FL96dpDGauBW53d2wHTeofW1tfOGq+ptd2kP0dX9ML+m/RE9/wCs/amXe+I3QtEuQXyzw/vBJmlRZnxznJifSNsRj+mMYt5/vEVrGML20jzvcdo4BLNyJs0Jc1IYeekSKiZwLeQFj4B/C+O7g==AAACcnichVHLSsNAFD2N7/qquhHdVEvFhciNCIor0Y1LtfYBbSlJnGowTWKSFjX0B/yBLtxYQVD8DDf+gAs/QVwquHHhbRoQLeodmDlz5p47Z+aqtqG7HtFTROrq7unt6x+IDg4Nj4zGxsYzrlV1NJHWLMNycqriCkM3RdrTPUPkbEcoFdUQWfVos3WerQnH1S1zzzu1RbGiHJh6WdcUj6miX1DL8VS95NOCXC/FErRIQcQ7gRyCBMLYtmI3KGAfFjRUUYGACY+xAQUujzxkEGzmivCZcxjpwblAHVHWVjlLcIbC7BHPB7zLh6zJ+1ZNN1CfcIbFSour1BFHkh7pll7pge7omT5+reYHVVpuTnlV21phl0bPJ1Pv/6oqvHo4/FL96dpDGauBW53d2wHTeofW1tfOGq+ptd2kP0dX9ML+m/RE9/wCs/amXe+I3QtEuQXyzw/vBJmlRZnxznJifSNsRj+mMYt5/vEVrGML20jzvcdo4BLNyJs0Jc1IYeekSKiZwLeQFj4B/C+O7g==

S0,2(null)(null)(null)(null)




S2,1 S3,1

S0,N S1,N

Si,l = (S1i,l, S

2i,l, . . . , S

Mi,l ) S

µi,l = ±1

AAACvnichVFNS9xAGH5Mq9WtH9v2IvQSulgUlmWyFloKwmIvvRTUdV3B6JJkZ9fBJBOTyaIN+QP+gR56aqHQ0t/QUy/9Az149iS2Jwu99NB3s0FRUd+BmWee933eeWbGDlwRKcYOh7Q7d4dH7o2OFe6PT0xOFR88XItkHDq84UhXhuu2FXFX+LyhhHL5ehByy7Nd3rR3XvXzzR4PIyH9VbUf8E3P6vqiIxxLEdUqNhPT7uj1tJWIspsuzNYHYCsx0vIZrqZl021LFZ1Tb9I5XdfN3d3YautnrOnF6YIZeLrRKpZYhWWhXwVGDkrIY0kWP8NEGxIOYnjg8KEIu7AQ0diAAYaAuE0kxIWERJbnSFEgbUxVnCosYndo7tJuI2d92vd7Rpl6jyokKSV1SaFjhv1kX9gp+8G+smP279puSdal72afVnug5UFr6mC6/vdWlUerwva56kbXCh28yNwKch9kTP8ezkDfe/vutP5yZSZ5yj6yE/L/gR2y73QDv/fH+bTMV96jQF9gXH7wq2CtWjHmK9XlZ6XaYv4Zo3iMJ5ilF3+OGl5jCQ069xuO8Au/tZrW0TxNDkq1oVzzCBdC2/sPJVqrwQ==

L

N

state of neurons with respect to M different patterns

↵ =M

N

N,M ! 1

Shun’ichi Amari (1971)“Esemble of random perceptrons”

activation function

S1AAACaHichVFNLwNBGH66vuujiwPi0mgqTs1sSYiTcHGktSpBmt01mHS7u9ndNmHjD7g4Ik4kIuJnuPgDDn4CjpW4OHh3u4nQ4J3MzDPPvM87z8zojik8n7GnhNTW3tHZ1d2T7O3rH0jJg0Prnl1zDa4atmm7G7rmcVNYXPWFb/INx+VaVTd5Sa8shfulOnc9YVtr/oHDt6vaniV2haH5RKnFcqAcleUMy7Eo0q1AiUEGcazY8g22sAMbBmqogsOCT9iEBo/aJhQwOMRtIyDOJSSifY4jJElboyxOGRqxFRr3aLUZsxatw5pepDboFJO6S8o0suyR3bIGe2B37Jl9/ForiGqEXg5o1pta7pRTx6PF939VVZp97H+p/vTsYxdzkVdB3p2ICW9hNPX1w9NGcb6QDSbZFXsl/5fsid3TDaz6m3G9ygsXSNIHKD+fuxWs53PKdC6/OpNZWIy/ohvjmMAUvfcsFrCMFah0rsAJznCeeJFkaUQaa6ZKiVgzjG8hTXwCVvKLhw== J1

AAACaHichVHLLgRBFD3T3uMxjQViIzMZsZpUIyFWwkasZjBGwmTS3cqo6Fe6eyah4wdsLBErEhHxGTZ+wMInYEliY+F2TwsiuJWqunXuPbdO1dUcQ3g+Y/cJqam5pbWtvSPZ2dXdk5J7+1Y9u+bqvKjbhu2uaarHDWHxoi98g685LldNzeAlbWc+jJfq3PWEba34uw4vm2rVEltCV32CiouVQNmvyBklxyIbYT+cj1AGseVt+RIb2IQNHTWY4LDgk29AhUdjHQoYHMLKCAhzyRNRnGMfSeLWKItThkroDq1VOq3HqEXnsKYXsXW6xaDpEnMEWXbHrtgzu2XX7IG9/VoriGqEWnZp1xpc7lRSB4PLr/+yTNp9bH+y/tTsYwvTkVZB2p0ICV+hN/j1vaPn5ZmlbDDKztkT6T9j9+yGXmDVX/SLAl86RfJrA353VsdzykRuvDCZmZ2LW9GOYaQxRv89hVksII8i3StwiGOcJB4lWRqQhhqpUiLm9OObSel3RsqLfw==

J2AAACaHichVE9SwNBEH05v+NHohYqNiEhYhU2UVCsgjZipYmXBGIId+cal1zujrtLQEP+gI2lipWCiPgzbPwDFv4EtVSwsXDuElERdZbdnX0zb/btjmrpwnEZuw9IXd09vX39A8HBoeGRUHh0LOeYdVvjsmbqpl1QFYfrwuCyK1ydFyybKzVV53m1uuLF8w1uO8I0Nt09i5dqSsUQO0JTXILktXIz1SqHY8kE8y3CfjgfoRg6tm6GL7GFbZjQUEcNHAZc8nUocGgUkQSDRVgJTcJs8oQf52ghSNw6ZXHKUAit0lqhU7GDGnT2ajo+W6NbdJo2MSOIszt2xZ7ZLbtmD+zt11pNv4anZY92tc3lVjl0MJl9/ZdVo93F7ifrT80udrDoaxWk3fIR7xVam9/YP3rOLmXizRl2zp5I/xm7Zzf0AqPxol1s8Mwpgl8b8LuTSyWSc4nUxnwsvdxpRT+mEcUs/fcC0ljFOmS6V+AQxzgJPEphaUKaaqdKgQ5nHN9Mir4DSMuLgA==S2

AAACaHichVFNLwNBGH66vqo+ujggLqIhTs1sSYhTw8WxVUWCNLtrWpPuV3a3TWrTP+DiiDiRiIif4eIPOPgJOJK4OHi73UQQvJOZeeaZ93nnmRnNMYTnM/YQkzo6u7p74r2Jvv6BwaQ8NLzh2TVX50XdNmx3S1M9bgiLF33hG3zLcblqagbf1Korrf3NOnc9YVvrfsPhu6ZasURZ6KpPVLFQCjLNkpxiaRbG5E+gRCCFKHK2fIUd7MGGjhpMcFjwCRtQ4VHbhgIGh7hdBMS5hES4z9FEgrQ1yuKUoRJbpbFCq+2ItWjdqumFap1OMai7pJzENLtn1+yF3bEb9sjef60VhDVaXho0a20td0rJw7HC278qk2Yf+5+qPz37KGMx9CrIuxMyrVvobX394PilsLQ2HcywC/ZM/s/ZA7ulG1j1V/0yz9fOkKAPUL4/90+wkUkrc+lMfj6VXY6+Io4JTGGW3nsBWawihyKdK3CEE5zGniRZGpXG26lSLNKM4EtIUx9Y84uI S0AAACaHichVFNLwNBGH66vuujxQFxaTQVp+YtEuLUcHFEVSXVNLtrWhPb3c3utglN/4CLI+JEIiJ+hos/4OAn4Eji4uDtdhOhwTuZmWeeeZ93npnRbEO6HtFjSOno7Oru6e0L9w8MDkWiwyNbrlV1dJHVLcNytjXVFYY0RdaTniG2bUeoFc0QOW1/pbmfqwnHlZa56R3YolBRy6YsSV31mMpminVqFKNxSpIfsXaQCkAcQaxZ0WvsYBcWdFRRgYAJj7EBFS63PFIg2MwVUGfOYST9fYEGwqytcpbgDJXZfR7LvMoHrMnrZk3XV+t8isHdYWUMCXqgG3qle7qlJ/r4tVbdr9H0csCz1tIKuxg5Gs+8/6uq8Oxh70v1p2cPJSz6XiV7t32meQu9pa8dnrxmljYS9Wm6pBf2f0GPdMc3MGtv+tW62DhHmD8g9fO528HWbDI1l5xdn4+nl4Ov6MUkpjDD772ANFaxhiyfK3GMU5yFnpWoMqZMtFKVUKAZxbdQpj4BVPGLhg==

Sc

JNAAACaHichVFNLwNRFD0d3/XRwQKxaTTEqnktCbFq2IiFtGhJkGZmvNZL5ysz0yY0/QM2logViYj4GTb+gIWfgCWJjYU70xFEcF/ee/ede899572r2rpwPcbuI1JLa1t7R2dXtLunty8m9w8UXKvqaDyvWbrlbKiKy3Vh8rwnPJ1v2A5XDFXn62plwY+v17jjCstc8/Zsvm0oZVOUhKZ4BOWXivXlRlFOpJIssDj74XyEEggta8mX2MIOLGiowgCHCY98HQpcGptIgcEmbBt1whzyRBDnaCBK3CplccpQCK3QWqbTZoiadPZrugFbo1t0mg4x4xhnd+yKPbNbds0e2NuvtepBDV/LHu1qk8vtYuxgePX1X5ZBu4fdT9afmj2UMBtoFaTdDhD/FVqTX9s/el6dWxmvT7Bz9kT6z9g9u6EXmLUX7SLHV04R/dqA351COpmaSqZz04nMfNiKToxiDJP03zPIYBFZ5OlegUMc4yTyKMnSkDTSTJUiIWcQ30waeweA54uc

Perceptron（McCulloch-Pitts model)

S0(t+ 1) = sgn

1pN

NX

i=1

JiSi(t)

!

AAACx3ichVHLahRBFD1pX3F8ZNSN4KZxiMwgDNVRUIRA0I26kCTjJIF0bLo7NZMi/bKqZkgsauHWH3DhSkFE/Aw3+QEX+QQR3ESICxfe7mkQDeotqurec++5daoqKhKhNGP7U86x4ydOnpo+3Thz9tz5meaFiysqH8mY9+M8yeVaFCqeiIz3tdAJXyskD9Mo4avR9r0yvzrmUok8e6x3C76RhsNMDEQcaoKCZtQLDLNtfd3ruPPGl6mrhpl1/YQPdNsfyDA2njW+eiq1eWStr0ZpYMS8Z59Q6Pqa7+hKhJF805qHlLO2V65t3fGlGG7pTtBssS6rzD3qeLXTQm2LefMdfGwiR4wRUnBk0OQnCKForMMDQ0HYBgxhkjxR5TksGsQdURWnipDQbVqHFK3XaEZx2VNV7JhOSWhKYrqYZZ/Ye3bA9tgH9pn9+GsvU/UotezSHk24vAhmXlzuHf6XldKusfWL9U/NGgPcrrQK0l5USHmLeMIfP3t50LuzPGuusTfsC+l/zfbZR7pBNv4Wv13iy6/QoA/w/nzuo87KXNe70Z1butlauFt/xTSu4Cra9N63sID7WESfzt3DVxziu/PAyZ2xszMpdaZqziX8Zs7zn1BHsdo=

fixed point patterns µ = 1, 2, . . . ,M

Statistical mechanics on the “ensemble of fixed points”

Elisabeth Gardner (1957-1988)

Gardner volume

“Gap”e��V (h) = ✓(h)

“Hardcore” constraint

Si(t) = Sµi

AAACeXichVHLSgMxFD0dX7W+6mMhuFGLUl2UVAVFEEQ3LvuwKvgoM2PU4LyYSQt16A/4Ay7cqCBS/Qw3/oALP0FcKrjQhbfTAVFRb0hycnLPzUmiOYbwJGMPEaWpuaW1Ldoe6+js6u6J9/ateXbJ1XlBtw3b3dBUjxvC4gUppME3HJerpmbwde1wub6/XuauJ2xrVVYcvm2q+5bYE7oqiSrGB/JFX1STcmIhADv+llmqFuMJlmJBDP8E6RAkEEbGjl9hC7uwoaMEExwWJGEDKjxqm0iDwSFuGz5xLiER7HNUESNtibI4ZajEHtK4T6vNkLVoXa/pBWqdTjGou6Qcxhi7ZzX2zO7YDXtk77/W8oMadS8VmrWGljvFnuPB/Ou/KpNmiYNP1Z+eJfYwF3gV5N0JmPot9Ia+fHTynJ/Pjfnj7II9kf9z9sBu6QZW+UW/zPLcKWL0Aenvz/0TrE2l0tOpqexMYnEp/IoohjCKJL33LBaxggwKdO4RzlDDdeRNGVGSymQjVYmEmn58CWX6AxVkki8=

V =

Z NY

j=1

dJjp2⇡

e�Jj

2

2

MY

µ=1

e��V (rµ)

AAAC/HichZHNahRBEMdrRqNxjGaNF8HL4JIQwSw9G8EgBIJeRFDy4W4CmewyM1sbO5kve3oXY9M+gC+QgydFEQlefQEvvoCHPIJ4jOAlgdTMDorGJD30dPW/6ldV3e2nIc8kY7uGeebs0LnzwxesiyOXLo9Wrow1s6QnAmwESZiIFd/LMOQxNiSXIa6kAr3ID3HZ37yf+5f7KDKexE/kVoprkbce8y4PPElSu7LdnLVcHkvbclORdNpqY9bRLfVYu13hBapjuxKfy6KOEtjR6iGFaK2Vmz0TUtXdlGttYUtNDYD/h7dUnRCav8u4Ua8o9KiEfZSe3ZwUrdyjb+p2pcpqrBj2UcMpjSqUYz6pfAAXOpBAAD2IACEGSXYIHmT0rYIDDFLS1kCRJsjihR9Bg0Vsj6KQIjxSN+m/TrvVUo1pn+fMCjqgKiFNQaQN4+wb+8j22Fe2w76z/WNzqSJH3ssWrf6AxbQ9+ura0q9TqYhWCU//UCf2LKELM0WvnHpPCyU/RTDg+y+295buLo6rCfaW/aD+37Bd9oVOEPd/Bu8XcPE1WPQAzr/XfdRo1mvOdK2+cLs6d698imG4Djdgku77DszBA5iHBtU9MKrGLWPKfGm+M3fMT4NQ0yiZq/DXMD8fAg87xfA=

↵ =M

NAAACeXichVHLSsNAFD2Nr1ofrXUjuKkWRVyUqQqKIBTduFHaaqtgpUziVEPTJCRpUUN/wB9wIQgKYsXPcOMPuPATxGUFF7rwNg2IinoHZs6ce8+dMzOyqam2w9hjQOro7OruCfaG+voHBsORoWjeNqqWInKKoRnWtsxtoam6yDmqo4lt0xK8ImtiSy6vtPJbNWHZqqFvOkem2K3wfV0tqQp3iCpGogWumQd8qVCyuOKu1d31ejESZwnmRewnSPogDj/SRqSBAvZgQEEVFQjocAhr4LBp7CAJBpO4XbjEWYRULy9QR4i0VaoSVMGJLdO8T7sdn9Vp3+ppe+pDqjBIaVCXOmKYYA/shjXZPbtlT+z9126u16Xl5ohWua0VZjF8MrLx+q+qQquDg0/Vn64dlLDguVXJvekxrXsobX3t+LS5sZidcCfZJXsm/xfskd3RDfTai3KVEdkzhOgLkt8f/CfIzySSs4mZzFw8tex/RhCjGMcUvfg8UlhFGjk69xDnuEYj8CaNSVPSdLtUCviaYXwJafYDwveSBA==

rµ = Sµ0

NX

i=1

1pN

JiSµi

AAACunichVHLThRBFD00Ijg+GGVD4mbiBONqUo0mECMJkQ1xYYBxgITGTndTA5XpF9XVE7BSP+APuHAlCTHGT3Dpxh9wwdKlssTEjQtvPxKiRL2Vqjr31D333qry01BkirGTEWv00tjl8YkrjavXrt+YbN68tZ4luQx4L0jCRG76XsZDEfOeEirkm6nkXuSHfMMfLBXnG0MuM5HEz9RhyrcjbzcWfRF4iii32ZPPtRPlZqHVdTUzleNkeeRqsWCT/9Q4fekF2jbayfalIsI4ih+osraWfMfoJxRsTLcSF9httlmHlda6COwatFHbStJ8Cwc7SBAgRwSOGIpwCA8ZjS3YYEiJ24YmThIS5TmHQYO0OUVxivCIHdC6S95WzcbkFzmzUh1QlZCmJGULM+wze8fO2Cf2nn1lP/+aS5c5il4OafcrLU/dyZfT3R//VUW0K+ydq/7Zs0If82WvgnpPS6a4RVDphy9enXUfrs3ou+yInVL/b9gJ+0g3iIffg+NVvvYaDfoA+8/nvgjWZzv2/c7s6oP24uP6KyZwG3dwj957DotYxgp6VPcDvuAbTq1Hlm8Ja1CFWiO1Zgq/maV+AVavrvM=

N,M ! 1 with fixed ↵AAACi3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuoqPjp+CrElOQrxGTmpZVUqiiUZ5ZkKKRlVqSmKKjEJOYUZCSqxAsoG+gZgIECJsMQylBmgIKAfIEVDDEMKQz5DMkMpQy5DKkMeQwlQHYOQyJDMRBGMxgyGDAUAMViGaqBYkVAViZYPpWhloELqLcUqCoVqCIRKJoNJNOBvGioaB6QDzKzGKy7AqgiH6gzH2hKLYMCg6rBVYOVBp8NThisNnhp8AenadVgU0CuqQTSSRC9qQXx/F0Swd8J6soF0iUMGQhdeF1dwpDGYAF2bSbQ9QVgEZA/kiH6y6qmfw62ClKtVjNYZPAa6P6FBjcNDgN9kFf2JXlpYGrQbAYucBRYgoAxPMAxGWFGeobGesaBRsoOTtDI4GCQZlBi0ACGuDmDA4MHQwBDKNDeLoYNDDsZdjHxMhkzWTHZQJQyMUL1CDOgACZXAIDllq8=

Sµ⌅(1)

Sµ⌅(2) Sµ⌅

J1⌅

J2⌅

Sµ⌅(N)AAACfnichVFNLwNBGH66vqq+igtxEQ3hoKYl0Tg1XJykVEuiNLtryqb7ZT+a1KZx9wccnEgaEQk/wsUfcPATxLESFwdvt5sIgncyM8888z7vPDMjmapiO4w9hYS29o7OrnB3pKe3r38gOjiUtw3XknlONlTD2pZEm6uKznOO4qh827S4qEkq35LKK839rQq3bMXQN52qyXc18UBXSoosOkQVo6PZPa+gubWiV5BUUS7bR65o8em1mVoxGmNx5sf4T5AIQAxBZIzoFQrYhwEZLjRw6HAIqxBhU9tBAgwmcbvwiLMIKf4+Rw0R0rqUxSlDJLZM4wGtdgJWp3Wzpu2rZTpFpW6RchyT7JFdswZ7YDfsmb3/WsvzazS9VGmWWlpuFgdOR7Jv/6o0mh0cfqr+9OyghJTvVSHvps80byG39JXjs0Z2aWPSm2KX7IX8X7Andk830Cuvcn2db5wjQh+Q+P7cP0E+GU/Mx5PrC7H0cvAVYYxhAtP03otIYxUZ5OjcE9RxizsBwpQwK8y1UoVQoBnGlxBSH/TUk20=

Sµ⌅ = sgn

1pN

NX

i=1

J i⌅Sµ⌅(i)

!

AAAC8nichVFLaxRBEK4ZX3F9ZNWL4mXIEtlclp5EUIRA0It4kCTrJoF0dpjp9G6anVe6exZj039A8CzoKYqI6L/wkj+QQ36CeFzBiwdrZwfExEc13V31dX1VX3dHeSyUJuTIcU+dPnP23NT52oWLly5P169cXVNZIRnvsCzO5EYUKh6LlHe00DHfyCUPkyjm69Hgwfh8fcilEln6RO/lfCsJ+6noCRZqhIL6i3bX0KSwgaFRHLKB2i1Cye2ioTLxVD+1Ho15TzdpT4bM+NZQtSu1eWwtVUUSGLHo2y6GHtX8qS71GMm3rXl0rGLXCGv/2Kwp5iyVor+j54J6g7RIad5Jx6+cBlS2nNXfA4VtyIBBAQlwSEGjH0MICscm+EAgR2wLDGISPVGec7BQQ26BWRwzQkQHuPYx2qzQFONxTVWyGXaJcUpkejBLDskHMiIH5CP5Qn78tZYpa4y17OEeTbg8D6afX29//y8rwV3Dzi/WPzVr6MHdUqtA7XmJjG/BJvzhs5ej9r3VWXOLvCFfUf8+OSKf8Qbp8Bt7t8JXX0MNP8A//twnnbX5lr/Qml+53Vi6X33FFNyEGWjie9+BJXgIy9DBviPnhjPjNFztvnL33beTVNepONfgN3M//QSdqsPi

JN⌅AAACeXichVHNLgNRFP46/uun9bOQ2KAhYtHcthJiJWzEQqoUCTWZGReTzp+ZaRMmfQEvYGGDRKQ8ho0XsOgjiCWJBQtnpiOIlHNz7z33O+c797v3yJamOi5jtYjQ1NzS2tbeEe3s6u6JxXv71h2zZCs8r5iaaW/KksM11eB5V3U1vmnZXNJljW/IxQU/vlHmtqOaxpp7ZPGCLu0b6p6qSC5BYnxgSfS2ZU1Sis5hSbJ5ZcdbrojxRCrJAhtmv5zPUAKhZc34NbaxCxMKStDBYcAlX4MEh8YWUmCwCCvAI8wmTw3iHBVEiVuiLE4ZEqFFWvfptBWiBp39mk7AVugWjaZNzGGMsQdWZc/snt2yR/besJYX1PC1HNEu17ncEmMng6uv/7J02l0cfLH+1OxiDzOBVpW0WwHiv0Kp88vHp8+rs7kxb5xdsifSf8Fq7I5eYJRflKsVnjtD9HsDGjvr6WQqk0yvTCXm5sNWtGMIo5ig/57GHBaRRZ7uPcY5qriJvAkjwoQwWU8VIiGnHz9MyHwAAzCSow==





input outputhidden


S2,1 S3,1

S0,N S1,N

Usual strategy of learning

(1) define “loss function" E =NX

i=1

MX

µ=1

⇣SµL,i � (S⇤)

µL,i

⌘2e.g.

desired output

(2) try to minimize the loss functionvia back-propagation

e.g. SDG (stochastic gradient descent)

Too much long-ranged, highly convoluted, non-linear interaction! …hard to analyze

SµL,i(t+ 1) = sgn

0

@ 1pN

NX

j=1

JL,i,jsgn

1pN

NX

k=1

JL�1,j,k · · · sgn

1pN

NX

m=1

J1,l,mSµ0,m(t)

!!1

A

AAADinicjVHLbtNAFL2ueZRQaAobJDYWoSgVJhoHJF6qVAELhBBqG9JWqlvLnkxSJ34xM4koo/kB+AAWrEBCCLFiC0s2/ACLfgJiWSQ2LLhxLBCtCL2Wfe89c8/xmZkgi0IhCdkxJsxDh48cnTxWOj514uR0eebUikj7nLImTaOUrwW+YFGYsKYMZcTWMs78OIjYatC7PVxfHTAuwjR5KLczthH7nSRsh9SXCHkzxvnGpnLjvvbUfTvUVXnRmbPmlctjS3QSbbkRa8uq2+Y+VY5WrnjEpXqgtSv6sae6847exNZyJXssczeKs5ZW93I5u6v1QaV646QuOXbX7uEobaVSWKWDisZjRB07smOtf2+fYFeVcy4PO1t7k1eukBrJw9pfOEVRgSIW0/IbcKEFKVDoQwwMEpBYR+CDwGcdHCCQIbYBCjGOVZivM9BQQm4fpxhO+Ij28NvBbr1AE+yHmiJnU/xLhC9HpgWz5At5S3bJZ/KOfCU//6mlco2hl23MwYjLMm/66ZnGj/+yYswStv6wxnqW0IZrudcQvWc5MtwFHfEHT57vNm4sz6oL5BX5hv5fkh3yCXeQDL7T10ts+QWU8AKcvce9v1ip15zLtfrSlcrCreIqJuEsnIMqnvdVWIC7sAhNoMYz473xwfhoTpl187p5czQ6YRSc0/BXmHd+Ab3E+i4=

Gardner volume generalized for a multi-layer network

“Gap”

(c.f. ) internal representation (2-layer): R. Monasson and R. Zecchina (1995)

Hamiltonian with “short-ranged” interactions

trace over hidden variables





input outputhidden


S2,1 S3,1

S0,N S1,N

e��V (h) = ✓(h)

“Hardcore” constraint

V (S(0),S(L)) = eNMS(S(0),S(l)) =

0

@L�1Y

l=1

NY

i=1

X

Sµl,i=±1

1

A

0

@Z Y

⌅

NY

j=1

dJj

⌅p2⇡

e�

(Jj⌅)

2

2

1

A e��H

H =MX

µ=1

X

⌅V (rµ⌅)

Gaussian approx. or modified model

⇠µ⌫ : Gaussian with zero mean and variance 1

rµ⌅ =NX

i=1

1pN

J i⌅

1pM

MX

⌫=1

⇠µ⌫S⌫⌅(i)S⌫⌅

!

AAADQnichVHLahRBFL3dvmL7yEQ3gpvGITLZDNVRMAiBYDYiRJKMkwRSmaa7UzMppl+prh4Si/oBf8CFKwUR8TN04Q+4yMIPUFcygpssvFPToE583Karbp17z6lTVWEe80IScmTZp06fOXtu6rxz4eKly9O1mSsbRVaKiLWjLM7EVhgULOYpa0suY7aVCxYkYcw2w/7yqL45YKLgWfpIHuZsJwl6Ke/yKJAI+bWPoqNoUmpf0TAOon6xXwaC6UXHpUWZ+IoverqjHmqXdkUQKU8rWuwLiYimkh1I40AJtqvVgwmNjuJaOzRmXdmYYK8g28jTtDQbrPyqFsYlw84Dbqxhi9ZuqzPqnbDZ4HP6jwVNBe/tyTm/VidNYsI9mXhVUocqVrPaK6CwCxlEUEICDFKQmMcQQIHfNnhAIEdsBxRiAjNu6gw0OMgtsYthR4BoH8cerrYrNMX1SLMw7Ah3ifEXyHRhlnwgr8mQvCdvyCdy/FctZTRGXg5xDsdclvvTT661vv+XleAsYe8n65+eJXRhwXjl6D03yOgU0Zg/ePx02Lq7PqtukhfkC/p/To7IWzxBOvgWvVxj68/AwQfwJq/7ZLIx3/RuNefXbteX7lVPMQXX4QY08L7vwBLch1VoQ2QtW9wSVmG/sz/bX+3huNW2Ks5V+C3s4x+kr+ap

Sµ⌅(1)

Sµ⌅(2) Sµ⌅

J1⌅

J2⌅

Sµ⌅(N)AAACfnichVFNLwNBGH66vqq+igtxEQ3hoKYl0Tg1XJykVEuiNLtryqb7ZT+a1KZx9wccnEgaEQk/wsUfcPATxLESFwdvt5sIgncyM8888z7vPDMjmapiO4w9hYS29o7OrnB3pKe3r38gOjiUtw3XknlONlTD2pZEm6uKznOO4qh827S4qEkq35LKK839rQq3bMXQN52qyXc18UBXSoosOkQVo6PZPa+gubWiV5BUUS7bR65o8em1mVoxGmNx5sf4T5AIQAxBZIzoFQrYhwEZLjRw6HAIqxBhU9tBAgwmcbvwiLMIKf4+Rw0R0rqUxSlDJLZM4wGtdgJWp3Wzpu2rZTpFpW6RchyT7JFdswZ7YDfsmb3/WsvzazS9VGmWWlpuFgdOR7Jv/6o0mh0cfqr+9OyghJTvVSHvps80byG39JXjs0Z2aWPSm2KX7IX8X7Andk830Cuvcn2db5wjQh+Q+P7cP0E+GU/Mx5PrC7H0cvAVYYxhAtP03otIYxUZ5OjcE9RxizsBwpQwK8y1UoVQoBnGlxBSH/TUk20=

Sµ⌅ = sgn

1pN

NX

i=1

J i⌅Sµ⌅(i)

!

AAAC8nichVFLaxRBEK4ZX3F9ZNWL4mXIEtlclp5EUIRA0It4kCTrJoF0dpjp9G6anVe6exZj039A8CzoKYqI6L/wkj+QQ36CeFzBiwdrZwfExEc13V31dX1VX3dHeSyUJuTIcU+dPnP23NT52oWLly5P169cXVNZIRnvsCzO5EYUKh6LlHe00DHfyCUPkyjm69Hgwfh8fcilEln6RO/lfCsJ+6noCRZqhIL6i3bX0KSwgaFRHLKB2i1Cye2ioTLxVD+1Ho15TzdpT4bM+NZQtSu1eWwtVUUSGLHo2y6GHtX8qS71GMm3rXl0rGLXCGv/2Kwp5iyVor+j54J6g7RIad5Jx6+cBlS2nNXfA4VtyIBBAQlwSEGjH0MICscm+EAgR2wLDGISPVGec7BQQ26BWRwzQkQHuPYx2qzQFONxTVWyGXaJcUpkejBLDskHMiIH5CP5Qn78tZYpa4y17OEeTbg8D6afX29//y8rwV3Dzi/WPzVr6MHdUqtA7XmJjG/BJvzhs5ej9r3VWXOLvCFfUf8+OSKf8Qbp8Bt7t8JXX0MNP8A//twnnbX5lr/Qml+53Vi6X33FFNyEGWjie9+BJXgIy9DBviPnhjPjNFztvnL33beTVNepONfgN3M//QSdqsPi

JN⌅AAACeXichVHNLgNRFP46/uun9bOQ2KAhYtHcthJiJWzEQqoUCTWZGReTzp+ZaRMmfQEvYGGDRKQ8ho0XsOgjiCWJBQtnpiOIlHNz7z33O+c797v3yJamOi5jtYjQ1NzS2tbeEe3s6u6JxXv71h2zZCs8r5iaaW/KksM11eB5V3U1vmnZXNJljW/IxQU/vlHmtqOaxpp7ZPGCLu0b6p6qSC5BYnxgSfS2ZU1Sis5hSbJ5ZcdbrojxRCrJAhtmv5zPUAKhZc34NbaxCxMKStDBYcAlX4MEh8YWUmCwCCvAI8wmTw3iHBVEiVuiLE4ZEqFFWvfptBWiBp39mk7AVugWjaZNzGGMsQdWZc/snt2yR/besJYX1PC1HNEu17ncEmMng6uv/7J02l0cfLH+1OxiDzOBVpW0WwHiv0Kp88vHp8+rs7kxb5xdsifSf8Fq7I5eYJRflKsVnjtD9HsDGjvr6WQqk0yvTCXm5sNWtGMIo5ig/57GHBaRRZ7uPcY5qriJvAkjwoQwWU8VIiGnHz9MyHwAAzCSow==

rµ⌅ =NX

i=1

1pN

J i⌅Sµ⌅(i)S

µ⌅

AAAC8nichVHNThRBEK4Z/MH1hwUvEi8bNhi8bHrQBGNCQvRCPBBgXSBh2ElP04udnT+6ezZip1/AxLMJnoAYY/QtvPgCHngE43FJuHiwdnYSI6BWp7urv6rvq+ruMIuE0oQcO+7IpctXro5eq1y/cfPWWHV8Yk2luWS8xdIolRshVTwSCW9poSO+kUlO4zDi62H36SC+3uNSiTR5rvcyvhXTnUR0BKMaoaD6RraNH+c2MH4YUdZVuzmV3M7XfJXHgRHznm2bJet3JGXGs8ZXu1IjYH3NX+qivpF825pnZxTaRljbvEh8Rty3tQsjNqjWSYMUVjvveKVTh9KW0+oH8GEbUmCQQwwcEtDoR0BB4dgEDwhkiG2BQUyiJ4o4BwsV5OaYxTGDItrFdQdPmyWa4HmgqQo2wyoRTonMGkyTb+Qj6ZOv5BP5Tn7+VcsUGoNe9nAPh1yeBWOv7zRP/8uKcdfw4jfrnz1r6MCjoleBvWcFMrgFG/J7r972m49Xp809ckh+YP8H5Jh8wRskvRP2foWvvoMKfoB39rnPO2uzDe9BY3blYX3hSfkVo3AXpmAG33sOFmARlqGFdfvOpDPl1F3t7rsH7tEw1XVKzm34w9zPvwCO18S8

glass order parameters

same random input

same random output

machine 1J1⌅,i

J2⌅,i

Jn⌅,i

machine 2

machine n

S1⌅

S2⌅

Sn⌅

qab,⌅ =1

M

MX

µ=1

(Sµ⌅)a(Sµ⌅)

b Qab,⌅ =1

N

NX

i=1

Ja⌅,iJb⌅,i

Replicas: machines learning in parallel

Parisi's RSB ansatz

overlap matrix

Probability distribution of overlap between replicas P (Q) =

dx(Q)

dQ

Qab = Q(x)

n

n n ! 0

l1 2 L

Extension to multi-layers

1AAACZnichVFNLwNRFD0dX1VfRYTEptEQq8mtSqpWwsZSW0WCyMx4mJjOTGamjWr6B9gSCysSCfEzbPwBC/+AWJLYWLgzHUQE9yXv3XvuPfed+55qG7rrEd1HpKbmlta2aHuso7Oruyfe27fkWmVHE0XNMixnRVVcYeimKHq6Z4gV2xFKSTXEsro75+eXK8Jxdctc9Kq2WC8p26a+pWuKx1AutRFPpmQKLEFyhig7mWUnRD5SSYS2YMUvsYZNWNBQRgkCJjz2DShwea0iBYLN2DpqjDns6UFeoI4Yc8tcJbhCYXSX922OVkPU5Njv6QbsPa6wmGlxlzoSGKU7uqJnuqVreqS3X7vVgi6+miqfaoMr7I2eg6HC67+sEp8edr5Yf6r2sIWpQK3O6u0A8efQGvzK/slzYTo/Whujc3pi/Wd0Tzc8gVl50S5yIn+KGH8ByVnf0p8P/tNZmpBTaTmdm0zOzIafEcUwRjDOL57BDOaxgCLfK3CIIxxHHqRuaUAabJRKkZDTj28mJd4B41uKhg==

0AAACZnichVFNLwNRFD0dX1VfRYTEptEQq8mtSqpWwsZSW0WCyMx4mJjOTGamjWr6B9gSCysSCfEzbPwBC/+AWJLYWLgzHUQE9yXv3XvuPfed+55qG7rrEd1HpKbmlta2aHuso7Oruyfe27fkWmVHE0XNMixnRVVcYeimKHq6Z4gV2xFKSTXEsro75+eXK8Jxdctc9Kq2WC8p26a+pWuKx1CONuLJlEyBJUjOEGUns+yEyEcqidAWrPgl1rAJCxrKKEHAhMe+AQUur1WkQLAZW0eNMYc9PcgL1BFjbpmrBFcojO7yvs3RaoiaHPs93YC9xxUWMy3uUkcCo3RHV/RMt3RNj/T2a7da0MVXU+VTbXCFvdFzMFR4/ZdV4tPDzhfrT9UetjAVqNVZvR0g/hxag1/ZP3kuTOdHa2N0Tk+s/4zu6YYnMCsv2kVO5E8R4y8gOetb+vPBfzpLE3IqLadzk8mZ2fAzohjGCMb5xTOYwTwWUOR7BQ5xhOPIg9QtDUiDjVIpEnL68c2kxDvhWYqF

xAAACZnichVFNLwNRFD0d3/VVREhsGk3FanKrolg1bCwVRVIiM+OVienMZGbaoOkfYEssrEgkxM+w8Qcs/ANiSWJj4c50EBHcl7x377n33Hfue6pt6K5HdB+RGhqbmlta26LtHZ1d3bGe3mXXKjuayGuWYTmrquIKQzdF3tM9Q6zajlBKqiFW1J1ZP79SEY6rW+aSt2eL9ZKyZepFXVM8hnK7G7FESqbA4iRnxidpIsNOiHykEght3opdYg2bsKChjBIETHjsG1Dg8iogBYLN2DqqjDns6UFeoIYoc8tcJbhCYXSH9y2OCiFqcuz3dAP2LldYzLS4Sw1xJOmOruiZbumaHunt127VoIuvZo9Ptc4V9kb3weDi67+sEp8etr9Yf6r2UMRkoFZn9XaA+HNodX5l/+R5cXohWR2hc3pi/Wd0Tzc8gVl50S5yYuEUUf4Ckqd8S38++E9neUxOpeV0bjyRnQk/oxVDGMYov3gGWcxhHnm+V+AQRziOPEhdUr80UC+VIiGnD99Mir8Dd9qK0A==

Replicated Gardner volume

qab(0) = qab(L) = 1

Replicated free-energy

��F (S0,SL)visible

NM=

@nV n(S0,SL)visible

��n=0

NM= Sn[{Q̂(l), q̂(l)}]

AAADbnicjVFbaxNBFD6b9VLjpWkFEURcDJUUapi0giIUioL4oNI0Jq1k0jC7TpKhs5fuTiJ1nD/gH/DBJwUfxJ8hgi8++tCfIL4IFXzxwbMXLFq8zLIz33xzvu+cmeNGUiSKkF2rZB86fOTo1LHy8RMnT01XZmY7STiOPd72QhnGGy5LuBQBbyuhJN+IYs58V/J1d+tmer4+4XEiwuC+2ol4z2fDQAyExxRS/coHOoiZpy9Tlyvm0BBjUyt9q6apO3Bapq+JWfiJ75h5s6lp7DsTkQhMYoy+d9cs5y5U8oGqOzRisRJM9nVg9h07m7j9X1cai+FIOU/QYpkUOcqt1LBLtaYjppymqcn5hRxvp5iaXr9SJXWSDecgaBSgCsVYDSvvgMJDCMGDMfjAIQCFWAKDBL8uNIBAhFwPNHIxIpGdczBQRu0YozhGMGS3cB7irluwAe5TzyRTe5hF4h+j0oE58pG8JnvkPXlDPpHvf/TSmUdayw6ubq7lUX/66dnWt3+qfFwVjPZVf61ZwQCuZbUKrD3KmPQWXq6fPH6217q+NqcvkZfkM9b/guySt3iDYPLVe9Xka88z9wA1j7Lb+ln+AN9XI+9iVlMgWeROYBuzpO+K74nNa/zeqoOgs1hvLNUXm1eqKzeKNk7BObgINezVVViB27AKbfCsJeuB5Vpe6Yt9xj5vX8hDS1ahOQ2/DLv2A6c44+w=

V n (S0,SL) =nY

a=1

Y

⌅TrJa⌅

!0

@Y

⌅\outputTrSa⌅

1

AY

µ,⌅,ae��V (r

µ⌅,a)

rµ⌅,a = Sµ⌅,a

NX

i=1

1pN

J i⌅,aSµ⌅(i),a

AAAC/HichVHNahRBEK4Z/5L1J6teArksLpEIcemJgiIEgl6CB0mybhLIZIeeTm9sdv7S3bMYm84D5AVy8KQoIsGrL5BLXsBDHkE8RsglgrWzA6JrtJrurv6qvqqvu8MsEkoTcuS4585fuHhpZLRy+crVa2PV6zeWVZpLxlssjVK5GlLFI5HwlhY64quZ5DQOI74Sdp/04ys9LpVIk+d6O+PrMd1MREcwqhEKqnuybfw4t4Hxw4iyrtrKqeTT1M7WmmdEfJXHgRGznm2bZ9bvSMqMZ42vtqRGwPqav9SFMlMQrXk6VKNthLV/bTAl7mC8FlTrpEEKqw07XunUobSFtPoBfNiAFBjkEAOHBDT6EVBQONbAAwIZYutgEJPoiSLOwUIFuTlmccygiHZx3cTTWokmeO7XVAWbYZcIp0RmDSbJF/KRHJNDsk++ktMza5miRl/LNu7hgMuzYGx3vHnyX1aMu4YXv1j/1KyhAw8LrQK1ZwXSvwUb8Huv9o6bj5YmzW3ylnxD/W/IETnAGyS97+z9Il96DRX8AO/P5x52lmca3r3GzOL9+tzj8itGYAJuwRS+9wOYg3lYgBb2/eHUnWnnrrvjvnP33U+DVNcpOTfhN3M//wQX68gW

quenched random input/output

Sn[{q̂(l)}, {Q̂(l)}] = ↵�1LX

l=1

Sbondent [Q̂(l)] +L�1X

l=1

Sspinent [q̂(l)]

�LX

l=1

e12

Pab qab(l�1)Qab(l)qab(l)@ha(l)@hb(l)

nY

a=1

e��V (ha(l))��ha(l)=0

AAADz3ichVHLbhMxFL1peJTh0RY2SGwsqlYJkMjTVipFqlTBhgWLpqEPqU5HnqnTjOp5dMYpKsaILfwAC1YgIYT4DDYs2bDoJyCWRbBBghtnShVVlDsa+9zje46vbT+VYa4o3S8NlU+dPnN2+Jxz/sLFSyOjY5dX8qSbBWI5SGSSrfk8FzKMxbIKlRRraSZ45Eux6m/f662v7oosD5P4odpLRSviW3HYDgOukPLGSuEkaXo6NutMa9bhiuyYiqwyc+swb/TzljPPuEw7fEPXXENY3o08Ledds6EfGNLc0CyLiJ/Em8azUMQKPdFAW4OWc3NA0vMoRHkaxsdEtosWYXESdyNfZIQxZ5KwTp7yQNzQ00FkSG2wCUegXzvjgXaNnjL9Re6bHTtWZM2tNgpYPeSqLOWZCrn0dAepHmMGKFtkkMuSTSywW8XGYVK0VZ3gjjXmC8XJSuVQXzWEsCzc6ijy5Mh1nhpvdJzWqQ1yHLgFGIciFpPRd8BgExIIoAsRCIhBIZbAIcdvHVygkCLXAo1chii06wIMOKjtYpXACo7sNo5bmK0XbIx5zzO36gB3kfhnqCQwQb/Q9/SAfqIf6Ff6659e2nr0etnD2e9rReqNvLja/PlfVYSzgs6R6sSeFbThtu01xN5Ty/ROEfT1u49fHjTvLE3oSfqGfsP+X9N9+hFPEO9+D942xNIrcOwDzNkgfTA7U4A59+8DrEzV3en6VGNmfOFu8RTDcA2uQwXvexYW4D4swjIEpc+lH6XfQ1BulB+Vn5af9UuHSoXmCgxE+fkfl7MPyA==

N,M ! 1 ↵ =M

N

1st Glass transition

bond

continuous transition to full RSB glass phase

at 1 st & (L-1) th layer

spin

other layers remain in the liquid phase

spin

↵g ' 2.03AAACfnichVFNL0NBFD2er6qvYkNsGg2xUbclIVYNG0tUkag07z2jJt6X916bVCP2/oCFFUkjIuFH2PgDFv0JYkliY+H29SWC4E5m5syZe+6cmdEcQ3o+Ub1FaW1r7+iMdEW7e3r7+mMDgxueXXJ1kdNtw3a3NNUThrREzpe+IbYcV6imZohN7WCpsb9ZFq4nbWvdrzhix1SLltyTuuozVYiN5FXD2VcL1bxrxovHeU+a4jCeTtJMIZagJAUR/wlSIUggjBU7doU8dmFDRwkmBCz4jA2o8LhtIwWCw9wOqsy5jGSwL3CMKGtLnCU4Q2X2gMcir7ZD1uJ1o6YXqHU+xeDusjKOcXqka3qhB7qhJ3r/tVY1qNHwUuFZa2qFU+g/Hc6+/asyefax/6n607OPPcwHXiV7dwKmcQu9qS8fnb1kF9bGqxN0Sc/s/4LqdM83sMqvem1VrJ0jyh+Q+v7cP8FGOpmaSaZXZxOZxfArIhjFGCb5veeQwTJWkONzT1DDLe4UKBPKlDLdTFVaQs0QvoQy/wETcJJ/

SciPost Physics Submission

4.1 Liquid phase

For small ↵ we find the whole system is in the liquid phase where the glass order parametersare all zero: for i = 1, 2, . . . , k qi(l) = 0 (l = 1, 2, . . . , L�1) and Qi(l) = 0 (l = 1, 2, . . . , L).This simply means that the parameter space is so large compared with the number ofimposed constraits so that the system can very easily satisfy the constrains.

4.2 1st glass transition

Figure 4: Glass order parameters close to the 1st glass transition: a) spatial profile ofthe Edwards-Anderson (EA) order parameter for spins qEA(l) and bonds QEA(l) slightlybefore ↵ = 2.0 (empty symbols)/after ↵ = 3.125 (filled symbols) the 1st glass transition.The depth is L = 10 in this example. b) Variation of the EA order parameters qEA(1) =qk(1) and QEA(1) = Qk(1) at the 1st layer after passing the critical point of the 1stglass transition ↵g(1) ' 2.03. c),d) Glass order parameter function q(x, l) for spins andQ(x, l) for bonds at the 1st layer l = 1 at around the 1st glasstransition. Here 1/↵ =0.32, 0.34, 0.36, 0.38, 0.40, 0.42 from the top to the bottom. e),f) the overlap distributionfunction of spins P (q) = dx(q)/dq and bonds P (Q) = dx(Q)/dQ Eq. (25).

With increasing ↵ Eq. (3), the system becomes more constrained. We find a continuousglass transition at ↵g(1) ' 2.03 on the two layers l = 1, L � 1 just beside the quenchedinput/output boundaries as shown in Fig. 4 a). The rest of the system (l = 2, 3, . . . , L�2)remains in the liquid phase qEA(l) = QEA(l) = 0 during the 1st glass transition. As shownin Fig. 4 b) the Edwards-Anderson (EA) order parameters of the spins qEA(l) = qk(l) andbonds QEA(l) = Qk(l) at the 1st layer l = 1 grow continuously across the critical point↵g(1). Exactly the same happens at the last layer l = L � 1.

As shown in Fig. 4 c),d), the functions q(x, l) an Q(x, l) at the 1st layer l = 1 (andthe same for l = L � 1) are continuous functions of x with plateaus at qEA and QEA forsome range x1(↵) < x < 1 with x1(↵) decreasing with ↵. Thus the replica symmetry isfully broken much as in the SK model for spinglasses [10]. Correspondingly the overlapdistribution functions Eq. (25) P (q) = dx(q)/dq and P (Q) = dx(Q)/dQ shown in Fig. 4e),f), exhibit delta peaks at q = qEA, Q = QEA plus non-trivial continuous parts extending

9

bond

1� qAAACaHichVFNLwNBGH66vuujiwPiIhrioplViY+TcHHUVpEgze6a1qb7ZXfaoPEHJK4cnEgkxM9w8Qcc/AQcSVwcvLvdEGnwTjLzzPO+zzvPzGiuafiCsceY1NTc0trW3hHv7OruSci9fWu+U/F0ntcd0/E2NNXnpmHzvDCEyTdcj6uWZvJ1rbwU5Ner3PMNx14VBy7fttSSbRQNXRVE5ZTJvYKcZCkWxkgjUCKQRBQrjnyNLezAgY4KLHDYEIRNqPBpbEIBg0vcNmrEeYSMMM9xhDhpK1TFqUIltkxziXabEWvTPujph+p9qnBI6VCXI4xgjD2wG/bK7tkte2Ifv3arhV0CNwe0anUtdwuJ48Hc+78qi1aB3W/Vn64FipgN3Rrk3g2Z4B56XV89PHvNzWfHauPskr2Q/wv2yO7oBnb1Tb/K8Ow54uEXzAWR/nrwRrA2lVLSqXRmOrmwGH1GO4Yxigl68RksYBkryNO5JZzgFGexZ0mWBqSheqkUizT9+BHS6Ccx3Ysa

qEAAAACb3ichVHLSiNRED1pdYxRJ1EXCoI0E5RxEypG83DlA2GWvqIyKqG7vWpjv+zuBDXkB1wLLkRBQRjxM9z4Ay78BJmVODAbF1Z3mhmGYca6cG/VqTp1T92rOobu+USPMamlte1De7wj0dnV/TGZ6uld8eyqq4myZhu2u6YqnjB0S5R93TfEmuMKxVQNsaruzQb51ZpwPd22lv1DR2yayo6lb+ua4jP0db9S33BNeW66UUmlKUOhyZTJFydKxTw7BRqjfEnORqk0Ipu3U9+wgS3Y0FCFCQELPvsGFHi81pEFwWFsE3XGXPb0MC/QQIK5Va4SXKEwusf7DkfrEWpxHPT0QvYBV9jMtLlLAzKG6YFu6IXu6Zae6PWf3ephl0DNIZ9qkyucSvJ4YOnnuyyTTx+7v1n/Ve1jG8VQrc7qnRAJ5tCa/NrR6cvS5OJwfYSu6Dvrv6RHuuMJrNoP7XpBLJ4hEX5BKbDcrwf/21kZy2RzmdzCeHpqJvqMOAbxCZ/5xQuYwhfMo8z3WjjBOS5iz1K/NCTJzVIpFnH68IdJo2+CCI58

QEAAAACb3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKRQXGV8cU5Sq4OtbGCygb6JkbWpqZWygY6BkbG1mYGAMZ5gaGZmbmCoZ6BmCgzAAFAfkCKxhiGFIY8hmSGUoZchlSGfIYSoDsHIZEhmIgjGYwZDBgKACKxTJUA8WKgKxMsHwqQy0DF1BvKVBVKlBFIlA0G0imA3nRUNE8IB9kZjFYdwVQRT5QZz7QlFoGBQZVg6sGKw0+G5wwWG3w0uAPTtOqwaaAXFMJpJMgelML4vm7JIK/E9SVC6RLGDIQuvC6uoQhjcEC7NpMoOsLwCIgfyRD9JdVTf8cbBWkWq1msMjgNdD9Cw1uGhwG+iCv7Evy0sDUoNkMXOAosAQBY3iAYzLCjPQMjfWMA02UHZygkcHBIM2gxKABDHFzBgcGD4YAhlCgvXkMUxjmMsxj/MgkziTHpABRysQI1SPMgAKYNAFsg45y

qEAAAACb3ichVHLSiNRED1pdYxRJ1EXCoI0E5RxEypG83DlA2GWvqIyKqG7vWpjv+zuBDXkB1wLLkRBQRjxM9z4Ay78BJmVODAbF1Z3mhmGYca6cG/VqTp1T92rOobu+USPMamlte1De7wj0dnV/TGZ6uld8eyqq4myZhu2u6YqnjB0S5R93TfEmuMKxVQNsaruzQb51ZpwPd22lv1DR2yayo6lb+ua4jP0db9S33BNeW66UUmlKUOhyZTJFydKxTw7BRqjfEnORqk0Ipu3U9+wgS3Y0FCFCQELPvsGFHi81pEFwWFsE3XGXPb0MC/QQIK5Va4SXKEwusf7DkfrEWpxHPT0QvYBV9jMtLlLAzKG6YFu6IXu6Zae6PWf3ephl0DNIZ9qkyucSvJ4YOnnuyyTTx+7v1n/Ve1jG8VQrc7qnRAJ5tCa/NrR6cvS5OJwfYSu6Dvrv6RHuuMJrNoP7XpBLJ4hEX5BKbDcrwf/21kZy2RzmdzCeHpqJvqMOAbxCZ/5xQuYwhfMo8z3WjjBOS5iz1K/NCTJzVIpFnH68IdJo2+CCI58

QEAAAACb3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKRQXGV8cU5Sq4OtbGCygb6JkbWpqZWygY6BkbG1mYGAMZ5gaGZmbmCoZ6BmCgzAAFAfkCKxhiGFIY8hmSGUoZchlSGfIYSoDsHIZEhmIgjGYwZDBgKACKxTJUA8WKgKxMsHwqQy0DF1BvKVBVKlBFIlA0G0imA3nRUNE8IB9kZjFYdwVQRT5QZz7QlFoGBQZVg6sGKw0+G5wwWG3w0uAPTtOqwaaAXFMJpJMgelML4vm7JIK/E9SVC6RLGDIQuvC6uoQhjcEC7NpMoOsLwCIgfyRD9JdVTf8cbBWkWq1msMjgNdD9Cw1uGhwG+iCv7Evy0sDUoNkMXOAosAQBY3iAYzLCjPQMjfWMA02UHZygkcHBIM2gxKABDHFzBgcGD4YAhlCgvXkMUxjmMsxj/MgkziTHpABRysQI1SPMgAKYNAFsg45y

2nd Glass transition

at 2nd & (L-2) th layer

continuous transition to full RSB glass phase

which also induce 2nd glass transitions at 1st and L-th layer


to q = 0 and Q = 0.

4.3 2nd glasstransition

Figure 5: Glass order parameters close to the 2nd glass transition: a) spatial profile ofthe Edwards-Anderson (EA) order parameter for spins qEA(l) and bonds QEA(l) slightlybefore ↵ = 15.38 (empty symbols)/after ↵ = 25 (filled symbols) the 2nd glass tran-sition. The depth is L = 10 in this example. b) Variation of the EA order param-eters qEA(2) and QEA(2) at the 2nd layer after passing the critical point of the 2ndglass transition ↵g(2) ' 15.9. c),d) Glass order parameter function q(x, l) for spinsand Q(x, l) for bonds at the 1st layer l = 2 at around the 2nd glasstransition. Here1/↵ = 0.04, 0.045, 0.05, 0.055, 0.06, 0.065 from the top to the bottom. e),f) the overlapdistribution function of spins P (q) = dx(q)/dq and bonds P (Q) = dx(Q)/dQ Eq. (25).

Increasing ↵ further we find 2nd glass transition at ↵g(2) ' 15.9 by which two layersl = 2, L�2 become included in the glass phase while the rest of the system l = 3, 4, . . . , L�3still remains in the liquid phase as shown in Fig. 5 a). The glass phase has grown a stepfurther into the interior. The transition is again a continuous one as can be seen in Fig. 5b) where we display the EA order parameters qEA(l) = qk(l) and QEA(l) = Qk(l) at l = 2.Exactly the same happens on the other side at l = L � 2.

As shown in Fig. 5 c),d), the functions q(x, l) an Q(x, l) at the 2nd layer l = 2 (andthe same for l = L� 2) are continuous functions of x with plateaus at qEA(2) and QEA(2)in some range x2(↵) < x < 1 with x2(↵) decreasing with ↵. A marked di↵erence withrespect to the 1st glass transition which happend at l = 1 (and l = L � 1) is that theorder parameters becomes finite only in some range x2(↵) / x < 1. As the result it looksapproximately like a step function with the step located at x2(↵). As shown in Fig. 5 e),f),this amount to induce a delta peak not only at qEA(2) (QEA(2)) but also at q = Q = 0 inthe distribution of the overlaps.

Remarkably, the 2nd glass transition induce another continuous glass transition withinthe glass phase on the 1st layer l = 1 (and l = L � 1). As can be seen in Fig. 5 c),d), aninternal step like structure emerge continuously within the region where the glass order

10

bond

spin

↵g(2) ' 15.38AAAChHichVFNLwNBGH6sr6qv4iJx0GgIiTSzrYY4iHBxbEuRqDS7a9TEftndNmHTo4s/4OBEIiK98gtc/AEHP0EcSVwcvN1uIgjeycw888z7vPPMjGrrwvUYe2yRWtvaOzojXdHunt6+/tjA4LprVRyNFzRLt5xNVXG5Lkxe8ISn803b4Yqh6nxD3V9u7G9UueMKy1zzDm2+bShlU+wKTfGIKsVGi4pu7ykl3y86Rrxcq02mpoquMPhBXM4k03OlWIIlWRDxn0AOQQJhZK3YFYrYgQUNFRjgMOER1qHApbYFGQw2cdvwiXMIiWCfo4YoaSuUxSlDIXafxjKttkLWpHWjphuoNTpFp+6QMo5x9sCu2Qu7Z3X2xN5/reUHNRpeDmlWm1pul/pPhlff/lUZNHvY+1T96dnDLuYCr4K82wHTuIXW1FePTl9W5/Pj/gS7YM/k/5w9sju6gVl91S5zPH+GKH2A/P25f4L1VFJOJ1O5mcTiUvgVEYxgDJP03rNYxAqyKNC5x6jjBrdShzQtpaVMM1VqCTVD+BLSwgeefZRy

1� qAAACaHichVFNLwNBGH66vuujiwPiIhrioplViY+TcHHUVpEgze6a1qb7ZXfaoPEHJK4cnEgkxM9w8Qcc/AQcSVwcvLvdEGnwTjLzzPO+zzvPzGiuafiCsceY1NTc0trW3hHv7OruSci9fWu+U/F0ntcd0/E2NNXnpmHzvDCEyTdcj6uWZvJ1rbwU5Ner3PMNx14VBy7fttSSbRQNXRVE5ZTJvYKcZCkWxkgjUCKQRBQrjnyNLezAgY4KLHDYEIRNqPBpbEIBg0vcNmrEeYSMMM9xhDhpK1TFqUIltkxziXabEWvTPujph+p9qnBI6VCXI4xgjD2wG/bK7tkte2Ifv3arhV0CNwe0anUtdwuJ48Hc+78qi1aB3W/Vn64FipgN3Rrk3g2Z4B56XV89PHvNzWfHauPskr2Q/wv2yO7oBnb1Tb/K8Ow54uEXzAWR/nrwRrA2lVLSqXRmOrmwGH1GO4Yxigl68RksYBkryNO5JZzgFGexZ0mWBqSheqkUizT9+BHS6Ccx3Ysa

Growth of glass phase under larger constrains

qEA

l l

spin bond

1/↵ = 0.02, 0.01, 0.005, 0.001, 0.0005, 0.00025AAACknichVHLSsNAFD2Nr1ofrY+F4KZYFBdab6pFEQQfGxcufFUFlZLEaQ2mSUjSghZ/wK0LF64URMQvcKsbf8CFnyAuFdy48CYNiIp6h7lz7pl77tyZUW1Ddz2ix4hUV9/Q2BRtjrW0trXHEx2da65VdjSR0yzDcjZUxRWGboqcp3uG2LAdoZRUQ6yre3P+/npFOK5umavevi22S0rR1Au6pnhM5RPD8siWYti7yhSlKTPETvYdZQNfw2FAmWw+kQoQW/InkEOQQmiLVuISW9iBBQ1llCBgwmNsQIHLYxMyCDZz26gy5zDSg32BQ8RYW+YswRkKs3vsixxthqzJsV/TDdQan2LwdFiZRD890BW90D1d0xO9/1qrGtTwe9nnVa1phZ2PH/WsvP2rKvHqYfdT9WfPHgqYCHrVuXc7YPxbaDV95eDkZWVyub86QOf0zP2f0SPd8Q3Myqt2sSSWTxHjD5C/P/dPsJZJy6PpzNJYano2/IooetGHQX7vcUxjHovI8bnHuMEt7qRuaVKakeZqqVIk1HThi0kLHzQilCI=

0

0.2

0.4

0.6

0.8

1

0 5 10 15 20 0

0.2

0.4

0.6

0.8

1

2 4 6 8 10 12 14 16 18 20

⇠(↵) / ln↵AAAChHichVHLLgRBFD3ae7wGG4mFjgkhkUmNR4iFCBtLr0GiZdLdChU13ZXumgkmljZ+wMKKRERs+QIbP2DhE8SSxMbCnZ5OBMGtVNWpU/fcOlXlKClCzdhjlVFdU1tX39CYaGpuaW1LtneshH4hcHnW9aUfrDl2yKXweFYLLfmaCriddyRfdXZny/urRR6EwveW9b7iG3l72xNbwrU1Ublkj7UnBixbqh170LRU4Cvtm5b0zApn5pIplmZRmD9BJgYpxDHvJy9hYRM+XBSQB4cHTVjCRkhtHRkwKOI2UCIuICSifY5DJEhboCxOGTaxuzRu02o9Zj1al2uGkdqlUyT1gJQm+tgDu2Iv7J5dsyf2/mutUlSj7GWfZqei5SrXdty19PavKk+zxs6n6k/PGluYiLwK8q4ipnwLt6IvHpy8LE0u9pX62Tl7Jv9n7JHd0Q284qt7scAXT5GgD8h8f+6fYGU4nRlJDy+MpqZn4q9oQDd6MUDvPY5pzGEeWTr3CNe4wa1RZwwZI8ZYJdWoijWd+BLG1AeHR5Tq

“penetration depth”

QEAAAACb3ichVHLSsNAFD2Nr1pfVRcKghRLRTdlWgXFVVUEl221ttiWksRRg3mRpMVa+gOuBReioCAofoYbf8CFnyCuRMGNC2/SgKiod2DmzLn33DkzI5mqYjuMPQSEtvaOzq5gd6int69/IDw4tGEbVUvmOdlQDasgiTZXFZ3nHMVRecG0uKhJKs9Le8tuPl/jlq0Y+rpTN3lZE3d0ZVuRRYeozUylUbK0yMpisxKOsjjzIvITJHwQhR9pI3yFErZgQEYVGjh0OIRViLBpFJEAg0lcGQ3iLEKKl+doIkTaKlVxqhCJ3aN5h3ZFn9Vp7/a0PfU+VRikNKhLExHE2D27Zi/sjt2wR/b+a7eG18V1U6dVamm5WRk4HF17+1el0epg91P1p2sH25j33Crk3vQY9x5yS187OH5ZW8jGGpPsgj2R/3P2wG7pBnrtVb7M8OwJQvQFie8P/hNsJOOJmXgyMxtNLfmfEcQYJjBFLz6HFFaRRo7O1XGEU5wFnoURYVyItEqFgK8ZxpcQpj8Aq2GN6g==


Figure 8: Glass order parameter functions under stronger constraints ↵ = 4000. In thisexample L = 20 and the centeral layers at l = 8, 9 still remain in the liquid phase. a),b)3 dimensinoal plots of q(x, l) and Q(x, l). c),d) the same in 2 dimensional plots. e,f)the corresponding overlap distribution functions Eq. (25) at l = 1, 2 (for clarity others atl = 3, 4, . . . are not shown).

x means hierarchical organization of clusters at each layer l: clusters of replicas which areclose to each other are enclosed in a meta-cluster of replicas at lower x in which replicascan be a bit more separated from each other, and so on.

To understand meaning of the hierarchical clustering of replicas along the x-axis, it isuseful to consider two independent machines subjected to the same inputs/outputs whichcan be regarded as two replicas. Recalling that x has a probablistic meaning as given inEq. (25), we notice that the two macines at a given layer l belong to the same cluster atlevel x with the probability 1� x which is larger for smaller x. Therefore, even if the twomachines do not belong to the same cluster at a given level x with some overlap q, theymay still belong to a common cluster at some lower x with lower overlap q. The hierarhicalorganization of clusters imply the hierahircal free-energy landscape as shown in panel c):small valleys are group into meta-valleys. The free-energies of the vallyes within the samemeta-valley are correlated [14]. The strength of fluctuation 1 � q(x, l) at di↵erent levelx can be understood as e↵ective ’flattness’ of the free-energy landscape at correspondinglevels.

The most important feature of our result is that the glass order parameter functionq(x, l) evolves in space in such a way that it progressively become less complex and flatteras we go deeper into the interior. For a given ↵, the penetration depth ⇠glass(↵) ⇠ ln ↵ isfinite. So that in deep enough network L/2 > ⇠glass(↵), the interior remains in the liquidphase. Moreover the river terraces at di↵erent layers are synchronized to each other withcommon positions of steps at x1 < x2 . . ., suggesting that basic hierarhical structure andthat of the free-energy landscape is preserved across the glass phases at di↵erent layers.For example, consider again two replicas, i. e. two independent machines subjected tothe same inputs/outputs represented by two ’stars’ in the panels b) and c). For a givenlevel, say n of the hierarchy, associated with the step at xn, the probability that the two

13

More “terraces” under larger constraints

↵ = 4000AAACbnichVHLSsNAFD2Nr1ofrQoiiFgsla7KbRUUQSi6canVPqCKJHG0oWkSkrSgxR9wLy4ERUFE/Aw3/oCLfoK4ERTcuPA2DYiKesNkzpy5586ZO4qla45L1AxIHZ1d3T3B3lBf/8BgODI0nHfMmq2KnGrqpl1UZEfomiFyrubqomjZQq4quigoleXWfqEubEczjQ133xJbVXnP0HY1VXaZKm3KulWWF2eJaDsSoyR5Ef0JUj6IwY9VM3KNTezAhIoaqhAw4DLWIcPhr4QUCBZzW2gwZzPSvH2BQ4RYW+MswRkysxX+7/Gq5LMGr1s1HU+t8ik6D5uVUcTpgW7ohe7plh7p/ddaDa9Gy8s+z0pbK6zt8NHY+tu/qirPLsqfqj89u9jFvOdVY++Wx7Ruobb19YOTl/WFbLwxTZf0xP4vqEl3fAOj/qperYnsKUL8AKnv7f4J8ulkaiaZXpuNZZb8pwhiHFNIcL/nkMEKVpHzOnaMM5wHnqVRaUKabKdKAV8zgi8hJT4AWBKNSw==

spin

bond

Summary: depth dependent free-energy landscape


Figure 9: River terrace like glass order parameter and its implications: a) schematic pictureof the river terace like glass order parameter q(x) (or Q(x)) b) hierarhical clustering ofreplicas c) shcematic free-energy landscape and trees representing ultrametric organizationof overlaps between metastable states.

replicas sit in the common valley at level n is 1� xn which is the same at all layers up tothe boundary l = n, n � 1, n � 2, . . . , 2, 1. It is tempting to speculate that these featureshave important conseuqences on learning in deep neural networks. We discuss possibleimplications of these in the last section.

5 Fluctuating boundary

Figure 10: Fluctuating input layer with hierarhical overlap structure

Now we analyze the case of fluctuating boundary: spin configuration on the boundariesare allowed to fluctuate during learning following certain probability distributions. Here

14

energy landscape become simpler and flatter at deeper layers

Fluctuating boundary


Figure 9: River terrace like glass order parameter and its implications: a) schematic pictureof the river terace like glass order parameter q(x) (or Q(x)) b) hierarhical clustering ofreplicas c) shcematic free-energy landscape and trees representing ultrametric organizationof overlaps between metastable states.

replicas sit in the common valley at level n is 1� xn which is the same at all layers up tothe boundary l = n, n � 1, n � 2, . . . , 2, 1. It is tempting to speculate that these featureshave important conseuqences on learning in deep neural networks. We discuss possibleimplications of these in the last section.

5 Fluctuating boundary

Figure 10: Fluctuating input layer with hierarhical overlap structure

Now we analyze the case of fluctuating boundary: spin configuration on the boundariesare allowed to fluctuate during learning following certain probability distributions. Here

14

“1RSB" boundary

“full RSB" boundary


we consider cases such that the overlap distribution of the spins on the input layer (l = 0)exibit hierahircal, stucture as described by the Parisi’s matrix Eq. (20) (Fig. 10).

5.1 One RSB type boundary

Here we consider the simplest case of ’1RSB’.

qi(0) =

(r mi < xinput

1 mi > xinput

It can be seen that alignment can be induced from the boundary.Two machines : those subjected to similar data are likely to be similar.... this is a hint

of generalization

Figure 11: Glass order parameters with 1 RSB input. (solid lines) Here L = 20, xinput =0.2 and r = 0.8 for the solid lines and xinput = 0.2 a),b) ↵ = 50 c),d) ↵ = 4000. (dotedlines) the glass order parameters with frozen boundary.

5.2 Full RSB type boundary

Let us next consider ’full RSB’ case. More specifically we consider the simplest full RSBstructure in the input layer,

qi(0) = min(ami, 1) (29)

15


with a certain constant a > 0. Thus q(x, 0) function consists of two parts: 1) ’continuouspart’ q(x, 0) = ax with slope a in the interval 0 < x < 1/a and 2) ’plateau’ q(x, 0) =qEA(0) = 1 in the interval 1/a < x < 1.

We analyze the saddle point solutions numerically as before (see sec. B.3.3). In thefollowing we present results using k = 100 step RSB and the depth of the system L = 20.We chose 1/a = 0.8. As shown in Fig. 12, the glass phase grows increasing ↵ much asin the case of quenched (RS) boundary condition. We limit ourselves to ↵ such that⇠glass(↵) < L/2 so that the we have a liquid phase at the center of the system. In thiscircumstance the boundary condition on the other side qab(L) is irrelvant.

Figure 12: Glass order parameters with full RSB input. Here L = 20 and 1/a = 0.8. a),b)↵ = 50 c),d) ↵ = 4000.

A remarkable feature of the resultant glass order parameter is that the hierahircalstructure put in the input propagate into the interior of the netwok preserving its basicstructure. Numerical solution suggests that the q(x, l) function at a given layer l consistsof three parts: 0) q(x, l) = 0 for some interval 0 < x < xl 1) ’continuous part’ q(x, l) =a(x � xl) in the interval xl < x < 1/a with the same slope a as in the input 2) ’plateau’q(x, l) = qEA(l) = 1�axl in the last interval 1/a < x < 1 as in the input. Correspondinglythe overlap distribution function P (q) = dx(q)/dq becomes,

P (q, l) = xl�(q) +1

a+

✓1 � 1

a

◆�(q � (1 � axl)) (30)

which consists of three parts: 0) delta peak at q = 0 1) constant part with height 1/a in

16

“full RSB” boundaryspin bond

↵ = 4000AAACcHichVHLSsNAFD2N7/qquim48FEUcVFuVFAEQXTjsrW2FaqUJE7bYJqEJC1q8QfciwtBURAUP8ONP+DCTxB3Krhx4W0aEC3qHZg5c+49lzN3VNvQXY/oMSS1tLa1d3R2hbt7evv6IwODGdeqOJpIa5ZhOZuq4gpDN0Xa0z1DbNqOUMqqIbLq7mo9n60Kx9Utc8Pbt8V2WSmaekHXFI+p3JZi2CVlaY6I8pEYxcmP0WYgByCGIBJW5Bpb2IEFDRWUIWDCY2xAgcsrBxkEm7lt1JhzGOl+XuAQYdZWuEpwhcLsLu9FvuUC1uR7vafrq/e4wmKlxV0OMYoJeqAbeqV7uqUn+vi1W83vUnezz6fa0Ao7338UTb3/qyrz6aH0pfrTtYcCFny3Oru3fab+Dq2hrx6cvKYW1ydqk3RJz+z/gh7pjl9gVt+0q6RYP0WYv0D+OfBmkJmJy7PxmeRcbHkl+IxODGMcUzzxeSxjDQmk/Ykd4wznoRcpKo1IY41SKRRohvAtpOlPiACNzg==

↵ = 50AAACbHichVG7SgNBFD1ZXzG+4qMQRBCDYhVuoqIIgmhj6SsaSUR210lcsi92N4EY/AFbCwu1UBARP8PGH7DIJ4hgE8HGwpvNgmhQ7zAzZ87cc+fMjGLrmusRVUNSS2tbe0e4M9LV3dPbF+0f2HatoqOKlGrplpNWZFfomilSnubpIm07QjYUXewohZX6/k5JOK5mmVte2RZ7hpw3tZymyh5Tu1lZtw/lxVnaj8YoTn6MNYNEAGIIYs2K3iKLA1hQUYQBARMeYx0yXG4ZJECwmdtDhTmHkebvCxwjwtoiZwnOkJkt8JjnVSZgTV7Xa7q+WuVTdO4OK8cwQU90RzV6pHt6po9fa1X8GnUvZZ6VhlbY+30nw5vv/6oMnj0cfqn+9Owhh3nfq8bebZ+p30Jt6EtHZ7XNhY2JyiRd0wv7v6IqPfANzNKberMuNs4R4Q9I/HzuZrCdjCem48n1mdjScvAVYYxgHFP83nNYwirWkOJzDZziApehV2lIGpFGG6lSKNAM4ltIk59I7IzY

Teacher student setting

J teacherijOut put of teacher

J studentijStudent is forced to

reproduce teacher’s output

Randomly quenched1) Training

random training data

random test data

J teacherij

J studentij

Compare

Randomly quenched

Now quenched

2) Test

a statistical inference problem (SIP)

teacher+student machine (inference)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

2 4 6 8 10 12 14 16 18 20

0

0.2

0.4

0.6

0.8

1

0 5 10 15 20

RS solution

R,QAAACaXichVG7SgNBFD1Z3/GRqE3QJhgUCwkTFRQr0cbSRPMADbK7jmbIZnfZnQQ1+AMWthGsFATFz7DxByzyCZIygo2FdzcLokG9AzNnzr3nzpkZzTaEKxlrhpSe3r7+gcGh8PDI6FgkOj6Rc62qo/OsbhmWU9BUlxvC5FkppMELtsPVimbwvFbe9PL5GndcYZm78tTmxYp6bIojoavSozIL8fRBNMGSzI94N0gFIIEgtq3oPfZxCAs6qqiAw4QkbECFS2MPKTDYxBVRJ84hJPw8xznCpK1SFacKldgyzce02wtYk/ZeT9dXn1CFRUqLupwjjln2wh5Ymz2zR/bKPn7tVve7eG5OadU6Wm4fRC5iO+//qiq0SpS+VH+6ljjCqu9WkHvbZ7x76B197azR3lnLzNbn2C1rkf8b1mRPdAOz9qbfpXnmGmH6gtTPB+8GucVkaim5mF5OrG8EnzGIacxgnl58BevYwjaydG4Jl2jgKtRSxpWYMtUpVUKBZhLfQkl8AoJEixQ=

r, qAAACaXichVFNLwNBGH66vqo+2nIRLo0NcZBmtiTEqeHiqKVIENldg4n9srttVNM/4OBaiROJhPgZLv6Ag58gPVbi4uDd7SaC4J1k5pnnfZ93npnRHEN4PmPPMamjs6u7J96b6OsfGEym0kPrnl12dV7SbcN2NzXV44aweMkXvsE3HZerpmbwDe1oKchvVLjrCdta86sO3zHVA0vsC131A8qdzhzvpmSWZWFkfgIlAjKiWLFTt9jGHmzoKMMEhwWfsAEVHo0tKGBwiNtBjTiXkAjzHHUkSFumKk4VKrFHNB/QbitiLdoHPb1QfUIVNilt6lJHBhPsid2xFntk9+yFvf/arRZ2CdxUadXaWu7sJs9GVt/+VZm0+jj8VP3p2sc+5kO3gtw7IRPcQ2/rK6eN1upCcaI2ya5Zk/xfsWf2QDewKq/6TYEXL5GgL1C+P/hPsJ7LKjPZXGFWzi9GnxHHGMYxRS8+hzyWsYISnXuIczRwEWtKaWlEGm2XSrFIM4wvIckfAzOLVA==

1+s replica

student machine

input outputrandom teacher machine

r,RAAACaXichVG7SgNBFD1Z3/GRRBvRJhgUCwmTKChWQRtLjSYKGsLuOjFLNjvL7iQYgz9gYRvBSkFQ/Awbf8DCTxDLCDYW3t0siAb1DsycOfeeO2dmNNs0XMnYc0jp6e3rHxgcCg+PjI5ForHxvCtqjs5zujCFs6epLjcNi+ekIU2+ZztcrWom39Uq615+t84d1xDWjmzYvFBVjyyjZOiq9ChnIZ4tRhMsyfyId4NUABIIYlNEb3GAQwjoqKEKDguSsAkVLo19pMBgE1dAkziHkOHnOU4RJm2NqjhVqMRWaD6i3X7AWrT3erq++pgqBCkFdTlFHLPsid2xNntk9+yFffzarel38dw0aNU6Wm4XI2eT2+//qqq0SpS/VH+6lihhxXdrkHvbZ7x76B19/aTV3l7Nzjbn2DV7Jf9X7Jk90A2s+pt+s8WzlwjTF6R+Png3yKeTqcVkemspkVkLPmMQ05jBPL34MjLYwCZydG4Z52jhIvSqxJRJZapTqoQCzQS+hZL4BMTmizU=

overlap

l l

↵ = 0.01AAACcHichVFNSwJRFD1OX2ZfVhuhRR9iRAt5Y0ERBFGblmppgUrMTE8dmi9mRsnEP9A+WgRFQVD0M9r0B1r0E6JdBm1adB0HoqS6D94977x7LufdJ1ua6riMPQWEru6e3r5gf2hgcGh4JDw6lnXMiq3wjGJqpr0rSw7XVINnXNXV+K5lc0mXNb4jH2y07neq3HZU09h2axYv6FLJUIuqIrlE5fKSZpWlVRZn4l44SsmLqU4g+iAKP5Jm+AZ57MOEggp0cBhwCWuQ4NDKQQSDRVwBdeJsQqp3z9FAiLQVquJUIRF7QHuJTjmfNejc6ul46kOqMElpUpcGphBjj+yWNdkDu2PP7OPXbnWvS8tNjbLc1nJrb+Q4svX+r0qn7KL8pfrTtYsilj23Krm3PKb1DqWtrx6dNrdW0rH6LLtiL+T/kj2xe3qBUX1TrlM8fYYQfYH4c+CdIJuIiwvxRGoxurbuf0YQE5jBHE18CWvYRBIZb2InOMdF4FWICJPCdLtUCPiacXwLYf4TfeaNyQ==

↵ = 0.02AAACcHichVHLSsNAFD2Nr1pfVTcFFz6KIi7KtAqKIIhuXLbVPqAWSeK0huZFkhZr6Q+4FxeCoiAofoYbf8CFnyDurODGhTdpQLSod2DumTP3XM7ckUxVsR3GngJCV3dPb1+wPzQwODQ8Eh4dy9pG1ZJ5RjZUw8pLos1VRecZR3FUnjctLmqSynNSZdO9z9W4ZSuGvuPUTV7UxLKulBRZdIgq7IqqeSCusRhL7IWjlLyY6gRxH0ThR9II32AX+zAgowoNHDocwipE2LQKiIPBJK6IBnEWIcW752giRNoqVXGqEImt0F6mU8FndTq7PW1PfUgVBikN6tLEFGbZI7tlLfbA7tgz+/i1W8Pr4rqpU5baWm7ujRxHtt//VWmUHRx8qf507aCEFc+tQu5Nj3HfIbf1taPT1vZqerYxx67YC/m/ZE/snl6g197k6xRPnyFEXxD/OfBOkE3E4ouxRGopur7hf0YQE5jBPE18GevYQhIZb2InOMdF4FWICJPCdLtUCPiacXwLYeETf+iNyg==

Bayes optimal, Nishimori condition

Review: Zdeborova-Krzakala (2017)

Symmetry breaking field: remanent bias in the liquid phase

O(ln(N)/N)AAACm3ichVHBShtRFD2Otmq0NepGKEIwWCKU8cZEYlwFsylF1MRGhSSEmfEZByczw8wkqEN+wB8Q6spCocWfKHTTuu/CTyguFdx00TuTRBFpex+8d9+559x33nuqbeiuR3TVJ/UPPHs+ODQcGRl98XIsOj6x5VpNRxMlzTIsZ0dVXGHopih5umeIHdsRSkM1xLZ6kA/q2y3huLplvveObFFtKHVT39M1xWOoFk37lbBJ2amrVT8pUxhvSF7MLCVpkZMO0l5PVAwzsTY3vzbXrkXjPWaM5AxRNp2N9ZixXimObmxY0c+oYBcWNDTRgIAJj3MDClweZSRBsBmrwmfM4UwP6wJtRFjbZJZghsLoAc913pW7qMn7oKcbqg+ZYbHS4i5txDBLP+kL3dB3uqBf9Puv3fywS+DmiFe1oxV2bexkavPuv6oGrx72H1T/dO1hD0uhW53d2yES3EPr6FvHpzeby8VZ/zV9pGv2f05X9I1vYLZutU8FUTxDhL+A5GwQqfsHf5psLcjJlJwqpOO5le5nDOEVZpDgF88gh7fYQInP/YCv+IFLaVrKS++k1Q5V6utqJvEopNIfg96Zwg==

Simulations of learning

random inputs/outputs

e��V (h) = e��✏h2

✓(h) � =1

kBTAAACunichVFNSxtBGH5cq9X4Fe1F6GUxROzBMBsLllIhpJce/UoUXA2z68QM2S93JwG77B/oH/DgqYVSSn9Cj734Bzx49KgeI/TSg282C9JK23eYmXeeeZ53npmxAkdGirHLIW34ycjo07Hx3MTk1PRMfnauHvmd0BY123f8cNfikXCkJ2pKKkfsBqHgruWIHav9tr+/0xVhJH1vW50EYt/lR55sSpsrghr5mjiIl01LKK7Xl1ovEn3tATBFEEnH9/TWQVxOElO1CCWSbh4fd/ihnrLWzGbI7dhI4nYjNkNXrybbSSNfYCWWhv44MbKkgCzW/fwXmDiEDxsduBDwoCh3wBFR24MBhoCwfcSEhZTJdF8gQY60HWIJYnBC2zQe0WovQz1a92tGqdqmUxzqISl1FNkF+8p67Jx9Y9fs119rxWmNvpcTmq2BVgSNmQ/zWz//q3JpVmg9qP7pWaGJV6lXSd6DFOnfwh7ou+9Pe1uvN4vxIvvEbsn/R3bJftANvO6d/XlDbJ4hRx9g/Pncj5N6uWSslMobLwuVavYVY3iOBSzRe6+igndYR43O/Y4r3OBWe6NZmtTaA6o2lGme4bfQ1D0pJKq0

soft-core potential

Monte Carlo simulation

H = �X

⌅

MX

µ=1

V�rµ⌅�

AAAlkXic3VrNbxtFFB++wXz0S0hIXCxCUCuVyCkgClKkJo2ThqaNGydp2rqNdp31h7K2N7u2a8eEP4B/gAMnkDggzvwFXPgHOCCucEAci8SFA++9mfXO164tBKXUT/bOvvm9N+/NvHk7M2s38JtRt1D48Yknn3r6mWefe/6F3IsvvfzKiZOnTu9EnV5Y9barHb8T7rpO5PnNtrfdbXZ9bzcIPafl+t5N9+Ay1t/se2HU7LS3usPAu9ty6u1mrVl1usDaO7lxZeHtStRr7Y0qru9UD6LDnhN6x3nBa/UW5o/vja4d7+Qrvlfrng3vIfNYg1fCZr3RPZfL5fZOzhTmCvTJjwvzemGGiU+pc+r0q6zC9lmHVVmPtZjH2qwLZZ85LAK6w+ZZgQXAu8tGwAuh1KR6jx2zHMj2AOUBwgHuAfzW4e6O4LbhHnVGJF2FVnz4hiCZZ7OFHwpfFx4Uvi98U/i18GeqrhHpQFuGcHW5rBfsnfj0tfIfE6VacO2yRiKVIRHBnUOehfD1J/jXZTV2kfxqgp8BcdDjKm+rf/TZg/KHm7OjtwpfFn4DX78o/Fj4Drxt93+vfnXD2/w8w5LY6hBqa/BFjdP3ThqyDvoc4DZoBAfsvDYiaXIuUCsT0QK+T5ERgrb7GT03nbbpvUcPWtIop+HQtgN2pFgW83y4utQ3IfQixkGD+gnj9jxxuVd4F0ApAjzKdmC0cezr1JMYOdUMCyKQaYk2Qphx6UjePzk2m6GrQRbhLHUpVqOxRCBQ2N6QonkEPESirYhC3KNKf8e2PNAyK7IVtsauw/c2fDeohHz8XaH7LUIiPTzbHl6/YVzXKCa7UGpRRuhCjCxQGWPqA/A8yXH7hMJYiShzBZDpH6fYyBtUhli4xpYgFtYpKnbYItukko3+Tdsedr9hbPiU8Sow4gv069OTwKHcGte5VOfSnVpTp5o6PW1aWt0+1e1THkI5uTWP6jC6MGv6FJ+y7BHVH1labAhJnX9A/APK0IHWmk91PlnpgkWqZItq8Vklc9vEbWvcAXEHYLPMDYgbAFduNSIu+lc3+qZLdeiDqr9G/D49D/gTWW2nYdQntVWqqxI3p0hFwr5I0zckfi91FJCbtGf3ZJUQq+MIkFteprrlcQTIclvCF7zy0Uzq1qlmPWW8dql2V/OlRNyS5ntZeFC22r4iZPR+LokeKxk9dov420qPqXHtST2WHt8hoUJ6Xqtx0RCREfeKXItriwW6tiFS8Dkut90RkdyhtXVdjAXOvxrJ4BMfV6VzUJoDvce01ojrUQLrsa4+rs8pOkpAiKmIp0Ue7lUtRSAVUdQQ14BUxDVC5CXMDpCK2RHWJJjLQCrmstbSCpCKWDFaugKkYq5oWm4CqYibhi1bQCpmS9NyHUhFXDdsWQJSMUualjUgFbFm2LIJpGI2NS3rQCpi3bDlIyAV85GB2QBSMRuGNctAKmZZs2YRSEUsaohVIBWxathSBlIxZcOWq0Aq5qrW0jaQitg2WroBpGJuGJhdIBWza1hzC0jF3NKsuQ2kIm5b5uSmmNeYTVo02h5pOa9FTktBrdFegqNUfQ0gjnThPk+7J9W7RWnMOGbRwBwCyYhDw/8lKd45Zsnaj7sKZtfAlKSsxDElA4MrAhkRGNb0gWRE39AxAJIRAwMxpF1ighgaiCMgGXFk2OEQyRhHiwqXSEa4GqKRMoZqHllTEGYe6QHJiJ4l9nB3P6Jrm3gh7ZnbFIGy33iaEGvrQN/KyJHQrEogokZ43IMM4HckeLqdHWGDrle3NaS9TV94hE9nPspqtuK6ltmeNFdQoxm/EWGjKZAuPc2TdiviTCKnWRjRnKmMTy34icjI2nZgRZpxHUF7NmRoQR5SL1foDMMnfro1pj3TyAZW2XAq2dAq27NiexZk34rsW5ANK7JhQQ6syIEFObQihxZk3YqsW5HT9Fvd2m9NK9bMAgM60zJbQb6Z0W06lwzcjhVnrvA64xlh5gwerzaJeoaE2Yud8QyxS5jzpDOefXYJ2xxctXq8akEWrciiJfdiPsMTmpBObeNzxxk2D98LipYq7TIQ5YEG+6kGJ+4B7iZckBiArgpZ0aFIf5Pq90FTJE4Oh9JpYp7aPhb5EM+T0DLMCyrmAmDeJG+yLEnzD/fLaAtvayT0TdI2jV/xc+bx8Sjd4jSv/t8ePtreHk8xe7lsMoe5FlylV8Q7q/jp1AfJnrAYTwrwdKMBevKwc79IPdUFyTy1F1AW4DXvkQ0B2Rb3ynlpRP7rM8ms08rJmHhs1iFj4p68CLuhIsP9Il4vw94e74qEWIR6fC+Ad3jNU3lTSG5II/3P2Pbf9Zscd12IM75ejSMmeY+lrsB92ofw06WaOEV16K2Wb6zvQ4FNtHJegtb3rPyUQp7R8ruJ+MxClimLPX66TLzjV/d1XGYH7qtQMk8FOhrC9lwvj2cwSiLvvvSGEfc+KOfQavIOzKW7GT3HM1I+o7f2lByZ3aZHqyB8G3+X1gmHMMs/YWdFhqmN89A54XWaHldoist18n1E3B6tI3iGivdp6fYkejzBtemYbEs7xZq2iFhH2l9nW9O22KNrybLnPq39ealJcdMgb7Lbvk8IWSpZQWe3t0WrTnXvuyWtRNPkWtRL/PxHlZZrJkVxlU7obGd2k6K/RucAVSkecS4FNJccyjzh+J8f8vo0kuLUPD8satpHY2tklKqR78aXNZS+Q+Ko/YmeuXR64Sl+xX2b/FNiJOESb7L08hOJSVpjlKozpGi26+WZLNGbnOEkGrLsatK74rKIIn6HOeljkU/vQRmz08dsgZ/d752cMf5LZBZ2LszNvzN34ca7M5eWxP+MnmevszcgW82z99kleDqX2DbY8y37if3Mfpk9M/vB7KVZgX3yCSFzhimf2at/AYvumGc=

Hamiltonian

Gap

with random boundaries

Dynamical variables J i⌅, Sµ⌅ (i = 1, 2, . . . , N)(µ = 1, 2, . . . ,M)

(⌅ = 1, . . . , LN)

(i = 1, 2, . . . , N)S0,i,SL,i






S2,1 S3,1

S0,N S1,N

rµ⌅ =NX

i=1

1pN

J i⌅Sµ⌅(i)S

µ⌅

AAAC8nichVHNThRBEK4Z/MH1hwUvEi8bNhi8bHrQBGNCQvRCPBBgXSBh2ElP04udnT+6ezZip1/AxLMJnoAYY/QtvPgCHngE43FJuHiwdnYSI6BWp7urv6rvq+ruMIuE0oQcO+7IpctXro5eq1y/cfPWWHV8Yk2luWS8xdIolRshVTwSCW9poSO+kUlO4zDi62H36SC+3uNSiTR5rvcyvhXTnUR0BKMaoaD6RraNH+c2MH4YUdZVuzmV3M7XfJXHgRHznm2bJet3JGXGs8ZXu1IjYH3NX+qivpF825pnZxTaRljbvEh8Rty3tQsjNqjWSYMUVjvveKVTh9KW0+oH8GEbUmCQQwwcEtDoR0BB4dgEDwhkiG2BQUyiJ4o4BwsV5OaYxTGDItrFdQdPmyWa4HmgqQo2wyoRTonMGkyTb+Qj6ZOv5BP5Tn7+VcsUGoNe9nAPh1yeBWOv7zRP/8uKcdfw4jfrnz1r6MCjoleBvWcFMrgFG/J7r972m49Xp809ckh+YP8H5Jh8wRskvRP2foWvvoMKfoB39rnPO2uzDe9BY3blYX3hSfkVo3AXpmAG33sOFmARlqGFdfvOpDPl1F3t7rsH7tEw1XVKzm34w9zPvwCO18S8

t t

Relaxation of autocorrelation functions

Cbond(t,⌅) =1

N

NX

i=1

hJ i⌅(0)J i⌅(t)iAAAlunic3VpLbyPHEW45dmzTj107MGDAF8Kygl1AEEabBHEMC7AelKy1dpcrSlrtLtfCDDV8QEPOaIbkkqLpH+A/kENOCZCD4Z+RS/6AD774HuToAL74kKrqHk6/ZkgEydoxC+T0VH9VXdVdXdPdQy8KOknfcb5Zeu4Xz7/wyxdfern0yquvvX7t+htvniThIG74x40wCONTz038oNPzj/udfuCfRrHvdr3Af+BdbGP9g6EfJ52wd9QfR/6TrtvqdZqdhtsH1tn10fbZpB53y17YO5/e6K/WvcBtXCSXAzf2b27Um7HbmKxPJ3en5Xoy6J5NOhvr00/hth64vVbgl2+DuCQCdZ3pDeemld2/Wa7HJFYqlc6uLztrDn3Ks8K6Xlhm4lMN33jzLVZn5yxkDTZgXeazHutDOWAuS4Aes3XmsAh4T9gEeDGUOlTvsykrgewAUD4gXOBewG8L7h4Lbg/uUWdC0g1oJYBvDJJltuJ87XzpfOf83fnK+YfzQ66uCelAW8Zw9bisH51d++Lt2vdzpbpw7bN2JlUgkcCdS57F8A3m+NdnTfY++dUBPyPioMcN3tbw6o/f1T44XJn82vmL80/w9c/ON87fwNve8F+Nv973D/9UYElqdQy1TfiixsV7Jw/ZAn0ucNs0giO2qo1InpwH1C1EdIEfUGTEoO1pQc8tpm1x79GDrjTKeTi07YJdKZalvACuHvVNDL2IcdCmfsK4XSUu9wrvIiglgEfZEEYbx75FPYmR0yiwIAGZrmgjhhmXj+T9U2IrBbraZBHOUo9iNZlJRAKF7Y0pmifAQyTaiijE/VTpP7GtDLTDKmyX7bO78H0E33tUQj7+7tL9ESGRnp1tz67fMK6bFJN9KHUpI/QhRjaojDH1B/A8y3HnhMJYSShzRZDpf06xUTaoBrFwh21BLBxQVJywTXZIJRv9L2171v2GsRFQxqvDiG/Qb0BPApdya1rnUZ1Hd2pNi2pa9LTpanXnVHdOeQjl5NZ8qsPowqwZUHzKsldUf2VpsS0kdf4F8S8oQ0daawHVBWSlBxapkl2qxWeVzO0Rt6dxR8Qdgc0yNyJuBFy51YS46F/L6Js+1aEPqv4m8Yf0POBPZLWdtlGf1TaorkHckiKVCPsSTd+Y+IPcUUBu1p7dkz1C7M0iQG55h+p2ZhEgyx0JX/DKRzOrO6Cag5zxOqXaU82XKnGrmu814UHNavuukNH7uSp6rGr02EPiHys9psa1L/VYfnzHhIrpea3GRVtERtorci2uLTbo2oNIwee43HYoIjmktXVLjAXOvybJ4BMfV6VrUFoDvVNaa6T1KIH1WNea1ZcUHVUgxNTF06IM96qWCpCKqGiIO0Aq4g4hyhLmBEjFnAhrMsw2kIrZ1lraBVIRu0ZLHwOpmI81LQ+AVMQDw5YjIBVzpGm5C6Qi7hq2bAGpmC1Nyz6Qitg3bDkEUjGHmpYDIBVxYNhyG0jF3DYw94BUzD3Dmh0gFbOjWbMJpCI2NcQekIrYM2ypAamYmmHLJ0Aq5hOtpWMgFXFstHQfSMXcNzCnQCrm1LDmIZCKeahZ8whIRTyyzMlDMa8xm3RptH3SsqpFTldB7dNegqNUfW0gjvTgvky7J9W7TWnMOGbTwFwCyYhLw/8tKd45Zsvaj6cK5tTAVKWsxDFVA4MrAhkRGdYMgWTE0NAxApIRIwMxpl1ihhgbiCsgGXFl2OESyRhXiwqPSEZ4GqKdM4ZqHtlXEGYeGQDJiIEl9nB3P6Frj3gx7Zl7FIGy33iakGoLoW9l5ERoViUQ0SQ87kFG8DsRPN3OUNig69VtjWlvMxQe4dOZj7KarbiuHXYmzRXUaMZvQthkAaRHT/Os3bo4kyhpFiY0Z+qzUwt+IjKxth1ZkWZcJ9CeDRlbkJfUy3U6wwiIn2+Nac8ispFVNl5INrbKDqzYgQU5tCKHFmTbimxbkCMrcmRBjq3IsQXZsiJbVuQi/day9lvHijWzwIjOtMxWkG9mdJvOLQN3YsWZK7xwNiPMnMHj1SbRKpAwezGczRC7hDlPwtnss0vY5uCe1eM9C7JiRVYsuRfzGZ7QxHRqm547LrN1+N5StDRol4EoHzTYTzU4cQ9wN+GBxAh01cmKkCL9Pao/B02JODkcS6eJZWp7KvIhniehZZgXVMwtwLxH3hRZkucf7pfRFt7WROibp20Rv9LnzM/Ho3yL87z6//bwp+3tdIHZy2WzOcy14Cq9Lt5ZpU+nIUgOhMV4UoCnG23QU4ad+/vUU32QLFN7EWUBXvM7siEi29JeWZVG5Mc+kyw6rZyPScfmADIm7skrsBuqMNwv4nUb9vZ4VyHEJtTjewG8w2uZyodC8p400v8d2368fpPjrg9xxteracRk77HUFXhA+xB+utQUp6guvdUKjPV9LLCZVs7L0PqelZ9SyDNafjeRnlnIMjWxx8+XSXf86r6Oy5zAfQNK5qlAqCFsz/XabAajJPKeSm8Yce+Dci6tJh/DXHpS0HM8I5ULeutMyZHFbfq0CsK38U9onXAJs/xzdkNkmOYsD90UXufp8YSmtNwi3yfEHdA6gmeodJ+Wb0+mxxdcm475tvRyrOmJiHWl/XWxNT2LPbqWInue0tqflzoUN23yprjtp4SQpbIVdHF7R7TqVPe+R9JKNE+uS73Ez39UablmXhQ36ITOdmY3L/qbdA7QkOIR51JEc8mlzBPP/vkhr08TKU7N88OKpn0ys0ZGqRr5bnxHQ+k7JI46n+uZR6cXvuJX2rfZPyUmEi7zpkgvP5GYpzVFqTpjima7Xp7JMr3ZGU6mociuDr0rroko4neYkz4T+fRTKGN2+oxt8LP7s+vLxn+JzMLJrbX136zduv/b5Y+2xP+MXmLvsHchW62z37OP4OlcZcdgz7dLzy+9tvT6yocr3kpn5YJDn1sSMr9iymel/2+LVqXS

Cspin(t,⌅) =1

M

MX

µ=1

hSµ⌅(0)Sµ⌅(t)i

AAAlwHic3VrNbiPHEW45ieMwib22EyBALoRlBbuAIIw2CeIEEGD9ULJs7S5XlLTaXa6FGWr4A81wRjMklxRNP4BfwAefEiCHII+RS14gB99zCXJ0gFx8cFV1D6f/ZkgEydqxCuTMVH1VXdVdXdPdlBcHvXTgOJ+vvPSd737v5e+/8oPKD3/041dfu/X6G2dpNExa/mkrCqLk3HNTP+j1/dNBbxD453Hiu6EX+I+8q12UPxr5SdqL+ieDSew/C91Ov9futdwBsC5ufbx7MW0mYTWNe/3Z7cF60wvc1lV6PXQT/85Ws524renmbHpvVm2mwxCw4XBrc/YhMJqB2+8EfrUBTEkJZICZ3XbuFAgGd6rNhFQrlcrFrVVnw6G/6vxmU79ZZeKvHr3+xk9Zk12yiLXYkIXMZ302gPuAuSwFeso2mcNi4D1jU+AlcNcjuc9mrAK6Q0D5gHCBewXfHXh6Krh9eEabKWm3oJUAPgloVtma8zfnT84Xzl+dPzv/cL4stDUlG+jLBK4e1/Xji9c++Vnj3wu1QrgOWDfXKtFI4cmlyBL4BAviG7A2e4fi6kGcMXEw4hZva3Tz6ReN3x2vTX/h/MH5J8T6e+dz5y8QbX/0r9YfH/rHn5V4knmdgLQNH7S4fO8UITtgzwVul0ZwzNa1ESnS84DCUkQI/IAyIwFrz0t6bjlry0ePEYTSKBfh0LcrdqN4lvECuHrUNwn0IuZBl/oJ83aduDwqfIrhLgU86kYw2jj2HepJzJxWiQcp6ISijQRmXDGS90+FrZXY6pJHOEs9ytV0rhELFLY3oWyeAg+R6CuiEPdNpf/EtyrQHquxfXbI7sPnCXwe0B3y8Xufnk8IifTifHtx/YZ53aacHMBdSBVhADmyRfeYU7+FyPMad0kozJWUKlcMlf7blBtVgxqQC/fYDuTCEWXFGdtmx3Rno/+lby+63zA3Aqp4TRjxLfoO6E3gUm3NZB7JPHpSJR2SdOhtE2qyS5JdUh1CPbk1n2SYXVg1A8pPWfeG5DeWFrtCU+dfEf+KKnSstRaQLCAvPfBI1QxJiu8qmdsnbl/jjok7Bp9lbkzcGLhyqylxMb6O0TcDkmEMqv028Uf0PuBvZLWdriHPpS2StYhbUbRS4V+q2ZsQf1g4CsjN27NHckCIg3kGyC3vkWxvngGy3omIBa98NHPZEUmOCsbrnKTnWix14ta12BsigobV932ho/dzXfRY3eixx8Q/VXpMzWtf6rHi/E4IldD7Ws2LrsiMrFdkKa4ttujah0zB97jcdiQyOaK1dUeMBc6/NungGx9XpRtwtwF2Z7TWyOSogXKUdebyimKjDoSYpnhbVOFZtVIDUhE1DXEPSEXcI0RVwpwBqZgz4U2O2QVSMbtaS/tAKmLfaOk9IBXznmblEZCKeGT4cgKkYk40K/eBVMR9w5cdIBWzo1k5BFIRh4Yvx0Aq5lizcgSkIo4MX94HUjHvG5gHQCrmgeHNHpCK2dO82QZSEdsa4gBIRRwYvjSAVEzD8OUDIBXzgdbSKZCKODVaegikYh4amHMgFXNuePMYSMU81rx5AqQinljm5LGY11hNQhptn6ysa5kTKqhD2ktwlGqvC8SRHjxXafekRrctjRnHbBuYayAZcW3EvyPlO8fsWPvxXMGcG5i6VJU4pm5gcEUgI2LDmxGQjBgZNsZAMmJsICa0S8wREwNxAyQjbgw/XCIZ42pZ4RHJCE9DdAvGUK0jhwrCrCNDIBkxtOQe7u6ndO0TL6E9c58yUI4bTxMyaxH0rYycCsuqBiLahMc9yBi+p4Kn+xkJH3S7uq8J7W1GIiJ8O/NRVqsVt7XHLqS5ghbN/E0Jmy6B9OhtnrfbFGcSFc3DlOZMc35qwU9Epta2YyvSzOsU2rMhEwvymnq5SWcYAfGLvTH9WUY3tuomS+kmVt2hFTu0IEdW5MiC7FqRXQtybEWOLciJFTmxIDtWZMeKXKbfOtZ+61mxZhUY05mW2QryzYpus7lj4M6sOHOFF81nhFkzeL7aNDolGmYvRvMZYtcw50k0n312DdscPLBGfGBB1qzImqX2Yj3DE5qETm2zc8dVtgmfu4qVFu0yEOWDBfupBiceAe4mPNAYg60meRFRpr9N8kuwlIqTw4l0mliltmeiHuJ5EnqGdUHF3AXM2xRNmSdF8eF+GX3hbU2FvUXWlokre898eyIq9rgoqv/vCL/Z0c6WmL1cN5/D3Aqu0pviN6vs7TQCzaHwGE8K8HSjC3aqsHN/h3pqAJpVai+mKsAlvyYfYvIt65V1aUS+7jPJstPKxZhsbI6gYuKevAa7oRrD/SJed2Fvj081QmyDHH8XwCe8Vun+WGg+kEb6v+Pb19dvct4NIM/4ejXLmPx3LHUFHtA+hJ8utcUpqku/agXG+j4R2Nwq5+Vofc/KTynkGS3/NpGdWcg6DbHHL9bJdvzqvo7rnMFzC+7MU4FIQ9je6435DEZN5D2XfmHEvQ/qubSafApz6VlJz/GKVC3prQulRpa36dMqCH+Nf0brhGuY5R+z26LCtOd16I6IusiOJyxl9x2KfUrcIa0jeIXK9mnF/uR2fMG12VjsS7/Am77IWFfaX5d707f4o1sp8+c5rf35XY/ypkvRlLf9nBCyVr6CLm/vhFad6t73RFqJFumF1Ev8/EfVliWLsrhFJ3S2M7tF2d+mc4CWlI84l2KaSy5VnmT+nx/y+jSV8tQ8P6xp1qdzb2SUapHvxvc0lL5D4qjLhZF5dHrhK3FlfZv/p8RUwuXRlNnlJxKLrGYo1WZC2Wy3yytZbjc/w8ktlPnVo9+KGyKL+BPWpI9EPf0Q7rE6fcS2+Nn9xa1V43+JzJuzuxubv9y4+/BXq+/uiP8zeoX9nL0F1WqT/Ya9C2/nOjsFf/6+Ull5c+Unaztr3bVo7ZpDX1oROm8y5W/t5isZZKi0

Cspin(t, l)AAAlVnic3VpfbxtFEN8CpcX8aVpAQuLFIg1qpShyCoiCVKn546Rp09aNkzRtXSKfc/6jnO3rne3accMH4AvwwAsg8VD4GLzwBZDoR0A8FokXhJiZ3fPtvztbCNpSj+zbm/3N7Mzu7Nzunh3fa4SdXO7RkRdefOnoy8eOv5J59bXX3zgxdfLUdtjuBhV3q9L22sGOUw5dr9FytzqNjufu+IFbbjqee9PZX8L6mz03CBvt1mZn4Lt3m+Vaq1FtVModYO1OvbW0OywFzWzoN1qHZzqz3tlMJrM7NZ2by9EnOyrM64VpJj6F9slTb7MS22NtVmFd1mQua7EOlD1WZiHQHTbPcswH3l02BF4ApQbVu+yQZUC2CygXEGXg7sNvDe7uCG4L7lFnSNIVaMWDbwCSWTaT+zn3MPc491Puh9yvuT8TdQ1JB9oygKvDZV1/98QX7xT/GCvVhGuH1WOpFIkQ7srkWQBfb4x/HVZl58mvBvjpEwc9rvC2egdfPi5+ujEzfD/3be438PWb3KPcj+Btq/d75bsb7sZXKZZEVgdQW4Uvapy8d5KQNdBXBm6dRrDPZrURSZJzgJqpiCbwPYqMALTdT+m5ybRN7j160JRGOQmHtu2zA8WyiOfB1aG+CaAXMQ7q1E8Yt7PE5V7hnQ+lEPAo24bRxrGvUU9i5FRSLAhBpinaCGDGJSN5/2TYTIquOlmEs9ShWA1HEr5AYXsDiuYh8BCJtiIKcc8q/RPbskDLLM9W2Bq7Bt/b8L1OJeTj7wrdbxIS6cnZ9uT6DeO6SjHZgVKTMkIHYuQClTGmPgHP4xy3RyiMlZAylw+Z/nmKjaxBRYiFq2wRYmGdomKbLbANKtnov7TtSfcbxoZHGa8EI36Bfj16EpQpt0Z1DtU5dKfW1KimRk+bpla3R3V7lIdQTm7NpTqMLsyaHsWnLHtA9QeWFutCUufvE3+fMrSvteZRnUdWOmCRKtmkWnxWydwWcVsat0/cPtgsc33i+sCVWw2Ji/7VjL7pUB36oOqvEr9HzwP+RFbbqRv1cW2F6irEzShSobAv1PQNiN9NHAXkxu3ZPVklxOooAuSWl6lueRQBstym8AWvfDTjunWqWU8Yrx2q3dF8KRC3oPleFB4UrbavCBm9nwuixwpGj90i/pbSY2pcu1KPJcd3QKiAntdqXNRFZES9Itfi2uICXVsQKfgcl9tui0hu09q6JsYC51+VZPCJj6vSOSjNgd5DWmtE9SiB9VhXG9VnFB0FIMSUxNMiC/eqljyQishriKtAKuIqIbISZhtIxWwLa2LMEpCKWdJaWgFSEStGS5eAVMwlTctNIBVx07BlE0jFbGpargGpiGuGLYtAKmZR07IGpCLWDFs2gFTMhqZlHUhFrBu2XAZSMZcNzHUgFXPdsGYZSMUsa9YsAKmIBQ2xCqQiVg1bikAqpmjYcgVIxVzRWtoCUhFbRks3gFTMDQOzA6RidgxrbgGpmFuaNbeBVMRty5zcEPMas0mTRtslLbNa5DQV1BrtJThK1VcH4kgH7rO0e1K9W5DGjGMWDMw9IBlxz/B/UYp3jlm09uOOgtkxMAUpK3FMwcDgikBG+IY1PSAZ0TN09IFkRN9ADGiXGCMGBuIASEYcGHaUiWRMWYsKh0hGOBqinjCGah5ZUxBmHukCyYiuJfZwdz+ka4t4Ae2ZWxSBst94mhBpa0Pfysih0KxKIKJKeNyD9OF3KHi6nW1hg65XtzWgvU1PeIRPZz7KarbiupbZrjRXUKMZvyFhwwmQDj3N43ZL4kwio1kY0pwpjU4t+InI0Nq2b0WacR1CezZkYEHeo14u0RmGR/xka0x7JpH1rbLBRLKBVbZrxXYtyJ4V2bMg61Zk3YLsW5F9C3JgRQ4syJoVWbMiJ+m3mrXfGlasmQX6dKZltoJ8M6PbdC4auG0rzlzhtUczwswZPF5tErUUCbMX26MZYpcw50l7NPvsErY5uGr1eNWCzFuReUvuxXyGJzQBndpG547TbB6+5xQtFdplIMoFDfZTDU7cA9xNOCDRB10lsqJNkX6a6vdAUyhODgfSaWKW2j4U+RDPk9AyzAsq5hxgTpM3aZYk+Yf7ZbSFtzUU+sZpm8Sv6Dnz/HiUbHGSV/9vD59tbw8nmL1cNp7DXAuu0kvinVX0dOqBZFdYjCcFeLpRBz1Z2Lmfp57qgGSW2vMpC/Caj8gGn2yLemVWGpGnfSaZdlo5HhONzTpkTNyT52E3lGe4X8TrEuzt8S5PiAWox/cCeIfXLJU3hOR1aaT/HdueXr/JcdeBOOPr1Shi4vdY6grco30IP12qilPUMr3V8oz1fSCwsVbOi9H6npWfUsgzWn43EZ1ZyDJFscdPlol2/Oq+jstsw30FSuapQFtD2J7rxdEMRknk3ZfeMOLeB+XKtJq8A3PpbkrP8YyUTemtXSVHprfp0ioI38bfpXXCPZjln7MzIsNUR3norPA6SY8jNEXlGvk+JG6X1hE8Q0X7tGR7Yj2u4Np0jLellWBNS0RsWdpfp1vTstija0mz5z6t/XmpQXFTJ2/S275PCFkqXkGnt7dJq05177sprUST5JrUS/z8R5WWa8ZFcYVO6GxnduOiv0rnABUpHnEu+TSXypR5gtE/P+T1aSjFqXl+mNe0D0fWyChVI9+NL2sofYfEUXtjPXPo9MJV/Ir6Nv6nxFDCxd6k6eUnEuO0RihVZ0DRbNfLM1msNz7DiTWk2dWgd8VFEUX8DnPSA5FPP4MyZqcH7AI/u9+dmjb+S2QWts/NzX8wd+7Gh9MXF8X/jI6zd9l7kK3m2cfsIjydC2wL7DlgX7OH7PvTv5z+a+bozDEOfeGIkHmTKZ+Zqb8B58eAeQ==

Cbond(t, l)AAAlVnic3VpbbxtFFJ4CpcVcGm4SEi8WaRCVosgpIApSpebipKFp68ZJmrYukddZX5S1d7tru3bc8AP4AzzwAkg8AD+DF/4AEv0JiEeQeEGIc87Meue2awtBKfjI3tkz3zlzzsyZszOzdgKvFXULhYcnnnjyqZNPnzr9TO7Z555/4czMiy/tRn4vrLk7Nd/zwz2nGrleq+PudFtdz90LQrfadjz3pnO4gvU3+24YtfzOdncYuHfb1UanVW/Vql1g7c+8srI/qoTtvON3Do7f6s5753K53P7MbGGhQJ/8uLCoF2aZ+JT8F196lVXYAfNZjfVYm7msw7pQ9liVRUB32CIrsAB4d9kIeCGUWlTvsmOWA9keoFxAVIF7CL8NuLsjuB24R50RSdegFQ++IUjm2Vzhh8LXhV8K3xe+LfxU+D1V14h0oC1DuDpc1g32z3zyWvm3iVJtuHZZM5HKkIjgrkqehfD1JvjXZXV2gfxqgZ8BcdDjGm+rf/TpL+UPtuZGbxa+LPwMvn5ReFj4Drzt9H+tfXXD3fosw5LY6hBq6/BFjdP3ThqyAfqqwG3SCA7YvDYiaXIOUDsT0Qa+R5ERgrb7GT03nbbpvUcP2tIop+HQtkN2pFgW8zy4OtQ3IfQixkGT+gnjdp643Cu8C6AUAR5lfRhtHPsG9SRGTi3Dgghk2qKNEGZcOpL3T47NZehqkkU4Sx2K1WgsEQgUtjekaB4BD5FoK6IQ97jSX7EtD7TKimyNbbBr8L0N3+tUQj7+rtH9NiGRHp1tj67fMK7rFJNdKLUpI3QhRi5SGWPqffA8yXEHhMJYiShzBZDp/0+xkTeoDLFwlS1DLGxSVOyyJbZFJRv9k7Y96n7D2PAo41VgxC/Sr0dPgirl1rjOoTqH7tSaBtU06GnT1uoOqO6A8hDKya25VIfRhVnTo/iUZY+o/sjSYlNI6vxD4h9Shg601jyq88hKByxSJdtUi88qmdshbkfjDog7AJtlbkDcALhyqxFx0b+G0TddqkMfVP114vfpecCfyGo7TaM+qa1RXY24OUUqEvZFmr4h8Xupo4DcpD27J+uEWB9HgNzyKtWtjiNAltsWvuCVj2ZSt0k1mynjtUe1e5ovJeKWNN/LwoOy1fY1IaP3c0n0WMnosVvE31F6TI1rV+qx9PgOCRXS81qNi6aIjLhX5FpcW1ykawciBZ/jctu+iGSf1tYNMRY4/+okg098XJUuQGkB9B7TWiOuRwmsx7rGuD6n6CgBIaYinhZ5uFe1FIFURFFDXAVSEVcJkZcwu0AqZldYk2BWgFTMitbSGpCKWDNaugykYi5rWm4CqYibhi3bQCpmW9NyDUhFXDNsWQZSMcualg0gFbFh2LIFpGK2NC2bQCpi07DlQyAV86GBuQ6kYq4b1qwCqZhVzZolIBWxpCHWgVTEumFLGUjFlA1brgCpmCtaSztAKmLHaOkGkIq5YWD2gFTMnmHNLSAVc0uz5jaQirhtmZNbYl5jNmnTaLukZV6LnLaC2qC9BEep+ppAHOnAfZ52T6p3S9KYccySgbkHJCPuGf4vS/HOMcvWftxTMHsGpiRlJY4pGRhcEciIwLCmDyQj+oaOAZCMGBiIIe0SE8TQQBwByYgjw44qkYypalHhEMkIR0M0U8ZQzSMbCsLMIz0gGdGzxB7u7kd07RAvpD1zhyJQ9htPE2JtPvStjBwJzaoEIuqExz3IAH5Hgqfb6QsbdL26rSHtbfrCI3w681FWsxXXtcr2pbmCGs34jQgbTYF06GmetFsRZxI5zcKI5kxlfGrBT0RG1rYDK9KM6wjasyFDC/Ie9XKFzjA84qdbY9ozjWxglQ2nkg2tsj0rtmdB9q3IvgXZtCKbFuTAihxYkEMrcmhBNqzIhhU5Tb81rP3WsmLNLDCgMy2zFeSbGd2mc9nA7Vpx5grPH88IM2fweLVJNDIkzF70xzPELmHOE388++wStjm4bvV43YIsWpFFS+7FfIYnNCGd2sbnjrNsEb7nFS012mUgygUN9lMNTtwD3E04IDEAXRWywqdIP0v1B6ApEieHQ+k0MU9tH4t8iOdJaBnmBRVzHjBnyZssS9L8w/0y2sLbGgl9k7RN41f8nPn/eJRucZpX/20PH29vj6eYvVw2mcNcC67SK+KdVfx06oNkT1iMJwV4utEEPXnYuV+gnuqCZJ7aCygL8Jp3yYaAbIt7ZV4akX/7TDLrtHIyJh6bTciYuCcvwm6oyHC/iNcV2NvjXZEQS1CP7wXwDq95Km8JyevSSP89tv17/SbHXRfijK9X44hJ3mOpK3CP9iH8dKkuTlGr9FbLM9b3ocAmWjkvQet7Vn5KIc9o+d1EfGYhy5TFHj9dJt7xq/s6LrML9zUomacCvoawPdfL4xmMksi7L71hxL0PylVpNXkH5tLdjJ7jGSmf0Vv7So7MbtOlVRC+jb9L64R7MMs/Zm+JDFMf56Fzwus0PY7QFJcb5PuIuD1aR/AMFe/T0u1J9LiCa9Mx2ZZOijUdEbFVaX+dbU3HYo+uJcue+7T256UWxU2TvMlu+z4hZKlkBZ3d3jatOtW977a0Ek2Ta1Mv8fMfVVqumRTFNTqhs53ZTYr+Op0D1KR4xLkU0FyqUuYJx//8kNenkRSn5vlhUdM+Glsjo1SNfDe+qqH0HRJHHUz0zKHTC1fxK+7b5J8SIwmXeJOll59ITNIao1SdIUWzXS/PZIne5Awn0ZBlV4veFZdFFPE7zEkPRD79CMqYnR6wi/zsfn9m1vgvkVnYPb+w+PbC+RvvzF5aFv8zOs1eZ29Atlpk77FL8HQusR2w54h9zr5m35z98ewfcyfnTnHoEyeEzMtM+czN/AmWwIBi

T = 0.015 240 samplesN = 20,M = 200(↵ = 10) L = 10

0

0.2

0.4

0.6

0.8

1

0×100 1×104 2×104 3×104 4×104 5×104 6×104 7×104 8×104 9×104 1×105

l=123456789

10 0

0.2

0.4

0.6

0.8

1

0×100 1×104 2×104 3×104 4×104 5×104 6×104 7×104 8×104 9×104 1×105

123456789

teacher-studen

from complex to simple: hierarchical free-energy landscape...

Documents