rcim 2008 - janus
TRANSCRIPT
![Page 1: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/1.jpg)
Filippo MantovaniPhysics Department
Università degli Studi di Ferrara
Milano, December 19th 2008
Janus: FPGA Based System for Scientific Computing
![Page 2: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/2.jpg)
Filippo Mantovani, December 19th 2008 - 2 of 31 -
Overview:
1. The physical problem:- Ising model and Spin Glass in short;
2. The computational challenge: - Monte Carlo “tools”;- Computational features of Spin Glass problems;
3. The Janus system: - an exotic attempt to simulate spin systems;
4. Conclusions
![Page 3: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/3.jpg)
Filippo Mantovani, December 19th 2008 - 3 of 31 -
Statistical mechanics...
Statistical mechanics tries to describe the macroscopic behavior of matter in terms of
average values of microscopic structure.
For instance, try to explain why magnets have a transition temperature (beyond which they lose their magnetic state)...
...in terms of elementary magnetic dipoles attached to each atom in the material (that we call spins).
![Page 4: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/4.jpg)
Filippo Mantovani, December 19th 2008 - 4 of 31 -
The Ising model
Each spin takes only the values +1 (up) or -1 (down) and interacts only with its nearest neighbours in a discrete D-dimensional mesh (e.g. 2D).
Note that: A) Each configuration {S
i} has a probability given
by the Boltzmann factor:
U {S i}=−J∑⟨ ij ⟩si s j
Energy of the system for a given configuration {S
i}:
P{S i }∝e−U {S i} / kT ≡ e− U {S i}
B) Macroscopic observables have averages obtainedby summing on each configurations, each weightedby its probability.
M≡∑isi ⟨M ⟩∝∑{Si }
M {S i} P{S i}Magnetization:
![Page 5: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/5.jpg)
Filippo Mantovani, December 19th 2008 - 5 of 31 -
The Ising model
What about the “J”?!?In the Ising model we assume that J are constant (=+1 or -1).The energy profile is therefore smooth (and in this case symmetric):
U {S i}=−J∑⟨ ij ⟩si s j
![Page 6: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/6.jpg)
Filippo Mantovani, December 19th 2008 - 6 of 31 -
The Spin Glass model
Spin glasses are a generalization of Ising systems. An apparently trivial change in the energy function:
makes spin glass models much more interesting and complex than Ising systems: if we leave J NOT fixed, we obtain 2 important consequences:
1) frustration: 2) corrugation inenergy function:
U{Si }=−∑⟨ij ⟩Jijsisj , s={1,−1}, J={1,−1 }
![Page 7: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/7.jpg)
Filippo Mantovani, December 19th 2008 - 7 of 31 -
Problems around that...
Problems like finding the minimum energy configuration at a given temperature or
finding the critical temperature are absolutely not trivial and represent a
computational nightmare!!!
HINT:If we consider a 3D lattice, with 48 spins per side, we obtain that the possible configurations are 2483
!!!
![Page 8: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/8.jpg)
Filippo Mantovani, December 19th 2008 - 8 of 31 -
Overview:
1. The physical problem:- Spin Glass and Ising model in short;
2. The computational challenge: - Monte Carlo “tools”;- Computational features of Spin Glass problems;
3. The Janus system: - an exotic attempt to simulate spin systems;
4. Conclusions
![Page 9: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/9.jpg)
Filippo Mantovani, December 19th 2008 - 9 of 31 -
The Markov chains theory assures us that is valid the property of ergodicity:
So the probability of the configuration i, after a time long enough, tends to a probability (equilibrium condition).
Monte Carlo algorithms
As seen before, relative small lattices produce a huge number of possible configurations.
Monte Carlo algorithms are tools allowing us to visit configuration according to its probability.
![Page 10: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/10.jpg)
Filippo Mantovani, December 19th 2008 - 10 of 31 -
Monte Carlo algorithms
A direct consequence of this Markov chains property is that we can produce configurations that are distributed following a given distribution.
The “good news” is that observables are trivially computed as unweighted sums of Monte Carlo generated configurations*:
*we do not need any more to sum the observable weighted with the probability.
⟨ f ⟩∝∑{C }f CP C=∑i
f CiMC
C iMC ; i=1n
![Page 11: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/11.jpg)
Filippo Mantovani, December 19th 2008 - 11 of 31 -
The Metropolis algorithm
Assuming a 3D lattice composed of spins, we can update each lattice site following these steps:
✔ consider one spin s0 and compute his local energy E:
✔ flip the value of the spin: s0' = -s0
✔ compute the new local energy E':
✔ if E'<E then the spin flip is accepted;else we compute and generate a pseudo-random number R (0≤R≤1):
if R<X then the spin flip is accepted as wellelse remains s.
NOTE: If we replace the spin lattice with the topology of a random graph the algorithmkeeps working (e.g. this is exactly what we used to implement graph coloring).
E=∑⟨0j⟩J ij s j s0
E'=∑⟨0j⟩J ij s j s0 '
X=exp−E '−E/T
![Page 12: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/12.jpg)
Filippo Mantovani, December 19th 2008 - 12 of 31 -
Or easier...
![Page 13: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/13.jpg)
Filippo Mantovani, December 19th 2008 - 13 of 31 -
Computational features
• Operations on spins ≡ bit-manipulation.
• Computation of a limited set of exponential.(We can store them in LUTs).
• Random numbers should be of “good quality” and with a long period.
• A huge degree of available parallelism.
• Regular program flow.(orderly loops on the grid sites)
• Regular, predictable memory access pattern.
• Information-exchange (processor ‹› memory) is huge, however the size of the data-base is tiny.(On-chip memory seems to be ideal...)
![Page 14: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/14.jpg)
Filippo Mantovani, December 19th 2008 - 14 of 31 -
The PC approach...Using standard architectures as Spin Glass engines two different algorithms are commonly used:
Synchronous Multi-Spin Coding (SMSC): one CPU handles one single system; replicas are handled on a farm of several CPU (128 - 256); at every step each CPU update in parallel N spins (N=2 or 4);
NOTE: bottleneck is the number of floating point random numbers can be generated in parallel.
Asynchronous Multi-Spin Coding (AMSC): each CPU handles several (64 - 128) systems in parallel; replicas are handled on CPU farm (smaller than previous); at every step each CPU update in parallel 64 - 128 spins of different systems; on each single CPU random number is shared among
all systems.
![Page 15: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/15.jpg)
Filippo Mantovani, December 19th 2008 - 15 of 31 -
Overview:
1. The physical problem:- Spin Glass and Ising model in short;
2. The computational challenge: - Monte Carlo “tools”;- Computational features of Spin Glass problems;
3. The Janus system: - an exotic attempt to simulate spin systems;
4. Conclusions
![Page 16: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/16.jpg)
Filippo Mantovani, December 19th 2008 - 16 of 31 -
The Janus project
A collaboration of: Universities of Rome “La Sapienza” and Ferrara Universities of Madrid, Zaragoza, Extremadura BIFI (Zaragoza) Eurotech
![Page 17: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/17.jpg)
Filippo Mantovani, December 19th 2008 - 17 of 31 -
The Janus approach
We store the lattice within the embeddedmemories of the FPGA.
We use a set of registers to have no latencyaccess to a “small set” of spins (and couplings)
We house a set (as large as possible) of Update Engines...
Preamble: We split the 3D lattice in 2 parts, “black” spins and “white” spins (slice = checkerboard). We cannot update in the same time black spins and white spins.
![Page 18: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/18.jpg)
Filippo Mantovani, December 19th 2008 - 18 of 31 -
The Janus approach
Each Update Engine: computes the local contribution to U:
addresses a probability table according with U;
compares with a freshly generated;random number;
sets the new spin value;
U=−∑⟨ ij⟩si Jijs j
![Page 19: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/19.jpg)
Filippo Mantovani, December 19th 2008 - 19 of 31 -
The Janus approach
A whole picture:
![Page 20: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/20.jpg)
Filippo Mantovani, December 19th 2008 - 20 of 31 -
The toy...
The basic hardware elements are: ▸ a 2-D grid of 4 x 4 FPGA-based processors (SPs); ▸ data links among nearest neighbours
on the FPGA grid; ▸ one control processor on each board (IOP)
with 2 gbit-ethernet ports; ▸ a standard PC (Janus host) for each 2 Janus boards.
![Page 21: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/21.jpg)
Filippo Mantovani, December 19th 2008 - 21 of 31 -
Janus summary:
256 Xilinx Virtex4-LX200 (+16 IOPs);
1024 update engines on each processor;
pipelineable to one spin update per clock cycle;
88% of available logic resources;
using a bandwidth of ~12000 read bits + ~1000 written bits per clock cycle
47% of available on-chip memory;⇒
system clock at 62.5 MHz 16 ps average spin update time.⇒
1 SP ~40W, 16 SP ~640W, rack ~11KW
![Page 22: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/22.jpg)
Filippo Mantovani, December 19th 2008 - 22 of 31 -
Performance (for physicists)
The relevant performance measure is the average update time per spin (R)
measured in pico-sec per spin.
For each FPGA in the system we have:R = 1024 spin updates for each clock cycle
= 1024 * 16 ns ~ 16 ps / spin
For one complete 16 processors board:R = 1/16 * 1024 spin updates for each clock cycle
= 1/16 * 1024 * 16 ns ~ 1 ps / spin
![Page 23: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/23.jpg)
Filippo Mantovani, December 19th 2008 - 23 of 31 -
Performance(for computer scientists)
Each update-core performs 11 + 2 sustained pipelined operations* per clock cycle (@ 62.5 MHz)1024 update engines perform:
1024 × 13 ops / 16 ns = 832 Gops1024 × 3 ops / 16 ns = 192 Gops**
* these operations are on very short data words!!!** submitted to Gordon Bell Prize 2008.
Sustaining these performance requires a huge bandwidth to/from (fine-grained) memory: ▸ processing just one bit implies getting 12 more bits
from memory: 6 neighbours + 6 “J” = 12 bits; ▸ generating pseudo-randoms requires 3 32-bit data; ▸ 1 more 32-bit data (from the LUT) is necessary
for MC step;All in all the memory bandwidth sustained is:
140 bit × 1024/16 ns ~ 1 TB/s
Op
era
tion
sM
em
ory
ban
dw
idth
![Page 24: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/24.jpg)
Filippo Mantovani, December 19th 2008 - 24 of 31 -
Janus vs PC
Assuming to perform 1012 Monte Carlo steps on a lattice size of 643 we obtain the following performances:
JANUS AMSC SMSCprocessor 1 SP 1 CPU 1 CPUstatistic 1 (16) 1 (128) 1wall clock time 50 days 770 years 25 yearsenergy 2.7 GJ 2.3 TJ 77 GJ
processor 256 SPs 2 CPUs 256 CPUsstatistic 256 256 256wall clock time 50 days 770 years 25 yearsenergy 43 GJ 4.6 TJ 20 TJ
NOTE: 256 replicas are “physically/statistically interesting”: this is thereason of this choice!!!
![Page 25: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/25.jpg)
Filippo Mantovani, December 19th 2008 - 25 of 31 -
Janus gallery: SP
![Page 26: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/26.jpg)
Filippo Mantovani, December 19th 2008 - 26 of 31 -
Janus gallery: IOP
![Page 27: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/27.jpg)
Filippo Mantovani, December 19th 2008 - 27 of 31 -
Janus gallery: board
![Page 28: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/28.jpg)
Filippo Mantovani, December 19th 2008 - 28 of 31 -
Janus gallery: rack
![Page 29: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/29.jpg)
Filippo Mantovani, December 19th 2008 - 29 of 31 -
Overview:
1. The physical problem:- Spin Glass and Ising model in short;- LQCD vs statistical mechanics;
2. The computational challenge: - Monte Carlo “tools”;- Computational features of Spin Glass problems;
3. The Janus system: - an exotic attempt to simulate spin systems;
4. Conclusions
![Page 30: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/30.jpg)
Filippo Mantovani, December 19th 2008 - 30 of 31 -
Conclusions
✔ the Janus system is an heterogeneous massivelyparallel systems;
✔ made by highly parallel processors implementedon FPGA;
✔ delivers impressive performance: 1 SP is equivalentto ~100 PC and ~30-50 new multi-core architectures;
✔ used for scientific applications like Spin Glass,but in principle available for any kind of problemsfitting the topology of the machine and theresources of the FPGAs.
✔ uses NON-challenge technology... (plug the LEGO bricks and play)
![Page 31: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/31.jpg)
Filippo Mantovani, December 19th 2008 - 31 of 31 -
A slightly broader view...
The study of these systems: ✰ for a physicist means a better understanding
of condensed matter; ✰ for a computer scientist means developing
new algorithms and improving old ones; ✰ for a “computer mason” (like me) means
developing a massively parallel high performance system.
Moreover:All these systems are surprisingly strong related with other areas, like e.g. graph coloring.
![Page 32: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/32.jpg)
Filippo Mantovani, December 19th 2008 - 32 of 31 -
![Page 33: RCIM 2008 - Janus](https://reader034.vdocuments.net/reader034/viewer/2022051413/5550b7b1b4c905ff618b4c3b/html5/thumbnails/33.jpg)
Filippo Mantovani, December 19th 2008 - 33 of 31 -
About the random numbers
The pseudo random numbers are generated using a shift-register of N words of 32 bits
and can generate up to ~100 random numbers per clock cycle.
NOTE: 1 shift-register generating 32 numbers uses ~2% of the FPGA resources (~2500 Slices).