overview of supercomputer systems supercomputing division...
TRANSCRIPT
Ove
rvie
w o
f Su
perc
ompu
ter S
yste
ms
Supe
rcom
putin
g D
ivis
ion
Info
rmat
ion
Tech
nolo
gy C
ente
rTh
e U
nive
rsity
of T
okyo
2
Sup
erco
mpu
ters
at I
TC, U
. of T
okyo
Tota
l Use
rs >
2,0
002
Total
Pea
k per
forma
nce
: 140
TFL
OPS
Total
numb
er o
f nod
es: 9
52To
tal m
emor
y: 3
2000
GB
Peak
perfo
rman
ce /
node
: 147
.2 GF
LOPS
Main
memo
ry pe
r nod
e: 3
2 GB,
128 G
BDi
sk ca
pacit
y: 1
PB
AMD
Quad
Cor
e Opt
eron
2.3G
Hz
T2K-
Toda
i(
Hitac
hi HA
8000
-tc/R
S425
)To
tal P
eak p
erfor
manc
e: 5
4.9 T
FLOP
STo
tal nu
mber
of n
odes
: 56
Total
mem
ory
: 11
200 G
BPe
ak pe
rform
ance
/ no
de: 9
80.48
GFL
OPS
Main
memo
ry pe
r nod
e: 2
00 G
BDi
sk ca
pacit
y: 5
56 T
BIB
M PO
WER
7 3.8
3GHz
Yayo
i (H
itach
i SR1
6000
/M1)
Total
Pea
k per
forma
nce
: 1.13
PFL
OPS
Total
numb
er o
f nod
es: 4
800
Total
mem
ory
: 15
0 TB
Peak
perfo
rman
ce /
node
: 23
6.5 G
FLOP
SMa
in me
mory
per n
ode
: 32 G
BDi
sk ca
pacit
y: 1
.1 PB
+ 2.
1 PB
SPAR
C64 I
xfx1
.84GH
z
Oakle
af-fx
(Fuji
tsu P
RIME
HPC
FX10
)
(ret
ired,
Mar
ch 2
014)
“Oak
brid
ge-fx
” with
576
no
des
inst
alle
d in
Apr
il 20
14 (s
epar
ated
) (13
6TF)
•H
PC
I •
Sup
erco
mpu
ter S
yste
ms
in S
CD
/ITC
/UT
•O
verv
iew
of F
ujits
u FX
10 (O
akle
af-F
X)
•P
ost T
2K S
yste
m
3
Inno
vativ
e H
igh
Perf
orm
ance
C
ompu
ting
Infr
astr
uctu
re(H
PCI)
•H
PC
I–
Infra
stru
ctur
e (S
uper
com
pute
rs &
Dis
tribu
ted
Sha
red
Sto
rage
Sys
tem
)•
Sea
mle
ss a
cces
s to
K, S
C’s
(9 U
niv’
s), &
use
r's m
achi
nes
–P
rom
otio
n of
Com
puta
tiona
l Sci
ence
•S
trate
gic
Pro
gram
s fo
r Inn
ovat
ive
Res
earc
h (S
PIR
E)
–R
&D
for F
utur
e S
yste
ms
(Pos
t-pet
a/E
xasc
ale)
•H
PC
I Con
sorti
um–
Pro
vidi
ng p
ropo
sals
/sug
gest
ions
to th
e go
vern
men
t and
re
late
d or
gani
zatio
ns, o
pera
tions
of i
nfra
stru
ctur
e–
38 o
rgan
izat
ions
–O
pera
tions
sta
rted
in F
all 2
012
•ht
tps:
//ww
w.h
pci-o
ffice
.jp/
4
Nove
mbe
r 201
3
AICS
, RIK
EN:
K co
mpu
ter (
11.28
PF,
1.27P
iB)
Univ.
of T
okyo
:FX
10 (1
.13PF
, 150
TB)
SR16
000/M
1(54
.9TF,
10.94
TB)
T2K
(75.3
6TF,1
6TB/
140 T
F, 31
.25TB
)Ea
stHu
bPCC
lust
er(1
0TF,5
.71TB
/13TF
,8.15
TB)
GPU
Clus
ter(C
PU 4.
5TF,
GPU
16.48
TF,1.
5TB)
Wes
tHub
PCCl
uste
r(12.3
7TF,8
.25TB
)RE
NKEI
-VPE
:VM
Host
ing
Kyus
hu U
niv.:
FX10
(181
.6TF,
24TB
)CX
400 (
811.9
TF, 1
84.5T
B)HA
8000
-tc (7
12.5T
F, 24
.7TB)
SR16
000 V
M1 (8
.19TF
, 16T
B)
Hokk
aido
Univ.
:SR
1600
0/M1(
172T
F, 22
TB)
BS20
00 (4
4TF,
14TB
)
Nago
ya U
niv.:
FX10
(90.8
TF, 1
2TB)
CX40
0 (47
1TF,
43TB
)
Toho
ku U
niv.:
SX-9
(29.4
TF, 1
8TB)
Expr
ess5
800
(1.74
TF),
3TB)
Toky
o In
stitu
te o
f Tec
hnol
ogy
: TSUB
AME2
.5(5
.79 P
F, 15
0TB)
Osak
a Uni
v.:SX
-9 (1
6TF,
10TB
)SX
-8R
(5.3T
F, 3.3
TB)
PCCl
uster
(22.7
TF,
4.6TB
)
Kyot
o Un
iv.XE
6 (30
0.8 T
F, 59
TB)
Gree
nBlad
e800
0(24
2.5TF
, 38T
B)25
48X(
10.6T
F, 24
TB)
Univ.
of T
suku
ba:
T2K
(95.4
Tflo
ps, 2
0TB)
HA-P
ACS
(802
Tflo
ps,
34.3T
B)FI
RST
(36.1
TFlo
ps, 1
.6TB)
>22
PFLOPS
9 su
perc
ompu
ter c
ente
rs lo
cate
d at
un
iver
sitie
s in
Jap
anFi
scal
Year
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
Hokk
aido
Toho
ku
Tsuk
uba
Toky
o
Toky
oTe
ch.
Nago
ya
Kyot
o
Osak
a
Kyus
hu
The
next
nat
iona
l fla
gshi
p m
achi
ne
T2K
Toda
i(140
TF)
50+
PF
100+
PF
PostT
2K (2
0+ P
F, (1
00+
TiB, 6
00TiB
, 4.0
+ PB
/s, 0
.68+
PB/s)
)
Tsub
ame3
.0 (2
0+ P
F)Ts
ubam
e4.0
(100
+ PF
)
SX-8
+ S
X-9 (
21.7
TF, 3
.3 TB
, 50.4
TB/
s)(5
00+
TiB/s)
(5+
PiB/
s)
Cray
XC3
0 (4
00TF
)60
0TF
600T
F10
+ PF
50+
PF
(y
y)(
)(M
anyc
ores
ystem
) (70
0+ T
F)HA
-PAC
S (8
00 T
F)
Hitac
hi SR
1600
0/M1 (
172 T
F, 22
TB)
Clou
d Sys
tem H
itach
i BS2
000 (
44TF
, 14T
B)10
+ PF
50+
PF
()
Hitac
hi SR
1600
(25T
F)
Fujits
u FX1
0(27
0TF,
65.28
TB/
s), C
X400
(510
TF, 1
52.5
TiB, 1
51.14
TB/
s), G
PGPU
(256
TF, 3
0 TiB
, 53.9
2 TB/
s)
10+
PFHi
tachi
HA8
000tc
/HT2
10(5
00TF
, 215
TiB
, 98
.82TB
/s), X
eon P
hi (2
12TF
, 26.2
5 TiB
, 67
.2 TB
/s), S
R160
00(8
.2TF,
6 T
iB, 4
.4 TB
/s)50
+ PF
()
NEC
SX-9
+ E
xp58
00
(31T
F)~1
PFlop
/s30
+Pflo
p/s
Tsub
ame2
.5 (5
.5+ P
F,
110+
TB,
1160
TB/
s)Ts
ubam
e2.0
(2.4P
F,
97TB
, 744
TB/
s)
Cray
XE6
(30
0TF,
92.6T
B/s),
Gree
nBlad
e80
00(2
43TF
, 61.5
TB/
s)
50+
PFlo
psFu
jitsu F
X10
(90.8
TF, 3
1.8 T
B/s),
CX4
00(4
70.6T
F, 55
TB
/s)
Fujits
u FX1
0 (1
PFlop
s, 15
0TiB
, 408
TB/
s),
Hitac
hi SR
1600
0/M1
(54.9
TF,
10.9
TiB, 5
.376 T
B/s)
Fujits
u M90
00(3
.8TF,
1TB
/s)
HX60
0(25
.6TF,
6.6T
B/s)
FX1(
30.7T
F, 30
TB/
s)Up
grad
e (3.6
PF)
6c/
o Y.
Ishi
kaw
a (U
.Tok
yo)
SPIR
E/H
PCI
Stra
tegi
c Pr
ogra
ms
for I
nnov
ativ
e R
esea
rch
•O
bjec
tives
–S
cien
tific
resu
lts a
s so
on a
s K
com
pute
r sta
rts it
s op
erat
ion
–E
stab
lishm
ent o
f sev
eral
cor
e in
stitu
tes
for c
omp.
sci
ence
•
Ove
rvie
w
–S
elec
tion
of th
e fiv
e st
rate
gic
rese
arch
fiel
ds w
hich
will
cont
ribut
e to
find
ing
solu
tions
to s
cien
tific
and
soc
ial I
ssue
s•
Fiel
d 1:
Life
sci
ence
/Dru
g m
anuf
actu
re
•Fi
eld
2: N
ew m
ater
ial/e
nerg
y cr
eatio
n •
Fiel
d 3:
Glo
bal c
hang
e pr
edic
tion
for d
isas
ter p
reve
ntio
n/m
itiga
tion
•Fi
eld
4: Mono-zukuri
(Man
ufac
turin
g te
chno
logy
) •
Fiel
d 5:
The
orig
in o
f mat
ters
and
the
univ
erse
–
A n
atio
n w
ide
rese
arch
gro
up is
form
ed b
y ce
nter
ing
the
core
or
gani
zatio
n of
eac
h re
sear
ch a
rea
desi
gnat
ed b
y M
EX
T.
–Th
e gr
oups
are
to p
rom
ote
R&
D u
sing
K c
ompu
ter a
nd to
co
nstru
ct re
sear
ch s
truct
ures
for t
heir
own
area
7
HPC
I戦略
プログラム
Stra
tegi
c Pr
ogra
ms
for I
nnov
ativ
e R
esea
rch
8
•H
PC
I•
Supe
rcom
pute
r Sys
tem
s in
SC
D/IT
C/U
T•
Ove
rvie
w o
f Fuj
itsu
FX10
(Oak
leaf
-FX
)
•P
ost T
2K S
yste
m
9
Cur
rent
Sup
erco
mpu
ter S
yste
ms
Uni
vers
ity o
f Tok
yo•
Tota
l num
ber o
f use
rs ~
2,0
00•
Hita
chi H
A80
00 C
lust
er S
yste
m (T
2K/T
okyo
) (20
08.6
-20
14.3
)–
Clu
ster
bas
ed o
n A
MD
Qua
d-C
ore
Opt
eron
(Bar
celo
na)
–14
0.1
TFLO
PS
•H
itach
i SR
1600
0/M
1 (Y
ayoi
) (20
11.1
0-)
–P
ower
7 b
ased
SM
P w
ith 2
00 G
B/n
ode
–54
.9 T
FLO
PS
•Fu
jitsu
PR
IME
HP
C F
X10
(Oak
leaf
-FX
) (20
12.0
4-)
–S
PA
RC
64 IX
fx–
Com
mer
cial
ver
sion
of K
com
pute
r–
1.13
PFL
OP
S (1
.043
PFL
OP
S fo
r LIN
PA
CK
, 36t
hin
43r
dTO
P50
0)–
Add
ition
al 5
76 N
odes
with
136
TF
(Oak
brid
ge-F
X, 2
014.
04-)
10
11
His
tory
of W
ork
Rat
io
Res
earc
h A
rea
base
d on
CPU
Hou
rsFX
10 in
FY.
2013
(201
3.4~
2014
.3E)
12
Engine
ering
Earth/Space
Material
Energy/Physic
sInform
ation Sci.
Education
Indu
stry
Bio
Econ
omics
Serv
ice
Fee
•N
ot F
RE
E•
Ser
vice
Fee
= C
ost f
or E
lect
ricity
(Sys
tem
+A/C
)–
2M U
SD
for O
akle
af-F
X (2
MW
)–
1M U
SD
for T
2K (1
MW
) (~M
arch
201
4)
13
Serv
ices
for I
ndus
try
•O
rigin
ally
, onl
y ac
adem
ic u
sers
hav
e be
en a
llow
ed to
ac
cess
our
sup
erco
mpu
ter s
yste
ms.
•S
ince
FY
.200
8, w
e st
arte
d se
rvic
es fo
r ind
ustry
–m
ainl
y fo
r spr
ead
of la
rge-
scal
e pa
ralle
l com
putin
g–
not c
ompe
te w
ith p
rivat
e da
ta c
ente
rs, c
loud
ser
vice
s …
–ba
sica
lly, r
esul
ts m
ust b
e op
en to
pub
lic
–m
ax 1
0% to
tal c
omp.
reso
urce
is o
pen
for u
sage
by
indu
stry
–
spec
ial q
ualif
icat
ion
proc
esse
s ar
e ne
eded
•C
urre
ntly
onl
y O
akle
af-F
X is
ope
n fo
r ind
ustry
–N
orm
al u
sage
(mor
e ex
pens
ive
than
aca
dem
ic u
sers
)–
Tria
l usa
ge w
ith d
isco
unt r
ate
–R
esea
rch
colla
bora
tion
with
aca
dem
ic ra
te–
4 gr
oups
(FY
.201
4)–
Ope
n-S
ourc
e/In
-Hou
se C
odes
(NO
ISV
/Com
mer
cial
App
.)
14
Educ
atio
n•
Oak
leaf
-FX
onl
y•
2-D
ay “H
ands
-on”
Tut
oria
ls fo
r Par
alle
l Pro
gram
min
g by
Fac
ulty
Mem
bers
of S
CD
/ITC
(Fre
e)–
Fund
amen
tal M
PI (
3 tim
es p
er y
ear)
–A
dvan
ced
MP
I (2
times
per
yea
r)
–O
penM
Pfo
r Mul
ticor
eA
rchi
tect
ures
(2 ti
mes
per
yea
r)–
Par
ticip
ants
from
indu
stry
are
acc
epte
d.•
Gra
duat
e/U
nder
grad
uate
Cla
sses
with
Sup
erco
mpu
ter
Sys
tem
(Fre
e)–
We
enco
urag
e to
facu
lty m
embe
rs to
intro
duce
han
ds-o
n tu
toria
l of s
uper
com
pute
r sys
tem
into
gr
adua
te/u
nder
grad
uate
cla
sses
. –
Up
to 1
2 no
des
of O
akle
af-F
X–
Pro
posa
l–
Not
lim
ited
to C
lass
es o
f the
Uni
vers
ity o
f Tok
yo
15
•H
PC
I •
Sup
erco
mpu
ter S
yste
ms
in S
CD
/ITC
/UT
•O
verv
iew
of F
ujits
u FX
10 (O
akle
af-F
X)
•P
ost T
2K S
yste
m
16
Feat
ures
of F
X10
(Oak
leaf
-FX)
•W
ell-B
alan
ced
Sys
tem
–1.
13 P
FLO
PS
for P
eak
Per
form
ance
–M
ax. P
ower
Con
sum
ptio
n <
1.40
MW
•<
2.00
MW
incl
udin
g A
/C
•6-
Dim
. Mes
h/To
rus
Inte
rcon
nect
–H
ighl
y S
cala
ble
Tofu
Inte
rcon
nect
–5.
0x2
GB
/sec
/link
, 6 T
B/s
ec fo
r Bi-S
ectio
n B
andw
idth
•H
igh-
Per
form
ance
File
Sys
tem
–FE
FS (F
ujits
u E
xaby
te F
ile S
yste
m) b
ased
on
Lust
re•
Flex
ible
Sw
itchi
ng b
etw
een
Full/
Par
tial O
pera
tion
•K
com
patib
le !
•O
pen-
Sou
rce
Libr
arie
s/A
pplic
atio
ns•
Hig
hly
Sca
labl
e fo
r bot
h of
Fla
t MP
I and
Hyb
rid
17
FX10
Sys
tem
(Oak
leaf
-FX)
•A
ggre
gate
mem
ory
band
wid
th: 3
98 T
B/s
ec.
•Lo
cal f
ile s
yste
m fo
r sta
ging
with
1.1
PB
of c
apac
ity a
nd 1
31G
B/s
ec o
f ag
greg
ate
I/O p
erfo
rman
ce(fo
r sta
ging
)•
Sha
red
file
syst
em fo
r sto
ring
data
with
2.1
PB
and
136
GB
/sec
.•
Ext
erna
l file
sys
tem
: 3.6
PB
18
SPA
RC
64TM
IXfx
19
CPU
SPA
RC
64™
IXfx
1.84
8 G
Hz
SPA
RC
64™
VIIIf
x2.
000
GH
z
Num
ber o
f Cor
es/N
ode
168
Siz
e of
L2
Cac
he/N
ode
12 M
B6
MB
Pea
k P
erfo
rman
ce/N
ode
236.
5 G
FLO
PS
128.
0 G
FLO
PS
Mem
ory/
Nod
e32
GB
16 G
B
Mem
ory
Ban
dwid
th/N
ode
85 G
B/s
ec (D
DR
3-13
33)
64 G
B/s
ec (D
DR
3-10
00)
Rac
ks•
A “S
yste
m B
oard
” with
4 n
odes
•A
“Rac
k” w
ith 2
4 sy
stem
boa
rds
(= 9
6 no
des)
•Fu
ll S
yste
m w
ith 5
0 R
acks
, 4,8
00 n
odes
20
Tofu
Inte
rcon
nect
•N
ode
Gro
up–
12no
des
–A
/C-a
xis:
on
syst
em b
oard
, B-a
xis:
3sy
stem
boa
rds
•6D
:(X
,Y,Z
,A,B
,C)
–A
BC
3D
Mes
h: c
onne
cts
12 n
odes
of e
ach
node
gro
up–
XY
Z 3D
Mes
h:co
nnec
ts “A
BC
3D
Mes
h” g
roup
21
Softw
are
of F
X10
22
Com
putin
g/In
tera
ctiv
eN
odes
Logi
n N
odes
OS
Spe
cial
OS(X
TCO
S)R
ed H
at E
nter
pris
e Li
nux
Com
pile
r
Fujit
suFo
rtran
77/
90
C/
C++
GN
UG
CC,
g95
Fujit
su (C
ross
Com
pile
r)Fo
rtran
77/
90
C/
C++
GN
U(C
ross
Com
pile
r)G
CC,
g95
Libr
ary
Fujit
suS
SL
II (S
cien
tific
Sub
rout
ine
Libr
ary
II),
C-S
SL
II,S
SL
II/M
PI
Ope
n S
ourc
eB
LAS,
LAPA
CK,
ScaL
APAC
K,FF
TW,
Supe
rLU,
PETS
c,M
ETIS,
Sup
erLU
_DIS
T,P
aral
lel N
etC
DF
App
licat
ions
Ope
nFO
AM,
AB
INIT
-MP,
PHAS
E,
Fron
tFlo
w/b
lue
Fron
tSTR
,R
EVO
CAP
File
Sys
tem
FEFS
(bas
ed o
n Lu
stre
)
Free
Sof
twar
eba
sh, t
csh,
zsh
, em
acs,
aut
ocon
f, au
tom
ake,
bzi
p2, c
vs, g
awk,
gm
ake,
gzi
p,
mak
e, le
ss, s
ed, t
ar, v
imet
c.
NO
ISV/
Com
mer
cial
App
licat
ions
(e.g
. NA
STR
AN
, AB
AQ
US,
AN
SYS
etc.
)
•H
PC
I•
Sup
erco
mpu
ter S
yste
ms
in S
CD
/ITC
/UT
•O
verv
iew
of F
ujits
u FX
10 (O
akle
af-F
X)
•Po
st T
2K S
yste
m
23
Post
T2K
Sys
tem
•20
-30
PFL
OP
S, F
Y.2
015
•M
any-
core
bas
ed (e
.g. (
only
) Int
el M
IC/X
eon
Phi
)•
Join
t Cen
ter f
or A
dvan
ced
Hig
h P
erfo
rman
ce
Com
putin
g (J
CA
HP
C, h
ttp://
jcah
pc.jp
/)–
Uni
vers
ity o
f Tsu
kuba
–U
nive
rsity
of T
okyo
24
•P
rogr
amm
ing
is s
till d
iffic
ult,
alth
ough
Inte
l com
pile
r wor
ks.
–(M
PI +
Ope
nMP
)–
Tuni
ng fo
r per
form
ance
(e.g
. pr
efet
chin
g) is
ess
entia
l–
Som
e fra
mew
ork
for h
elpi
ng u
sers
ne
eded