![Page 1: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/1.jpg)
Energy-efficient High Performance Computing
with SuperMUC
By Ernst A. Graf
TERATEC Forum 2012, Toward Green Computing Centers,
Bruyères-Le-Chatel, 28.06.2012
Arndt Bode
Chairman of the Board of the Leibniz Supercomputing Centre
and Technische Universität München
![Page 2: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/2.jpg)
2
SuperMUC, TOP 500 39th Release,
Hamburg June 2012
# 1 in Europe
# 4 worldwide (most powerfull pure x86 system)
ISC ’12, Hamburg, Thursday Keynote, June 21, 2012
![Page 3: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/3.jpg)
3
I/O
nodes
NAS
80 Gbit/s
18 Thin node islands
(each >8000 cores)
1 Fat node island
(8200 cores)
also used as Migration System
$HOME
1.5 PB / 10 GB/s
Snapshots/Replika
1.5 PB
(separate fire section)
non blocking
pruned tree (4:1)
SB-EP
16 cores/node
2 GB/core
WM-EX
40cores/node
6.4 GB/core
10 PB
200 GB/s
GPFS for
$WORK
$SCRATCH
Visualization
Internet
Archive and Backup
~ 30 PB
Desaster Recovery Site
Compute nodes Compute nodes
non blocking
SuperMUC General Configuration
![Page 4: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/4.jpg)
4
What‘s special about SuperMUC
Dagstuhl Perspectives Workshop: May 21 - 25, 2012
SuperMUC is the most powerfull pure x86 ISA system of the world
General purpose, standard programming interface, „easy“ to port to,
future-safe for many applications
SuperMUC is the most energy efficient x86 based supercomputer of the world
Dark Center infrastructure at LRZ
Warm Water directly cooling
Energy aware scheduling with xCAT and Load Leveler
Contract including energy for 5 years
![Page 5: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/5.jpg)
Supercomputer Architectures at LRZ since 1990
1
10
100
1,000
10,000
100,000
1,000,000
10,000,000
GF
lop
/s, G
By
te
GFlop/s GByte
HLRB1: Hitachi SR8000+
HLRB2: SGI Altix 4700+ Linux-Cluster
10-fold every 4 yearsDouble every 14.1 Montths
Cray Y-MP8
IBM SP2Cray T90
Fujitsu
KSR
SuperMUCIBM
= 1Tera...
= 1 Peta...
= 1Giga
Cray Y-MP2
5
![Page 6: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/6.jpg)
Energy Efficiency and SuperMUC
Motivation: we pay 15.8 Cents per Kwh (regular contract, expect increase)
![Page 7: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/7.jpg)
7
Energy Efficient HPC
• Reduce the power
losses in the
power supply chain
• Exploit your possibilities
for using compressor-
less cooling und use
energy-efficient
cooling technologies
(e.g. direct liquid
cooling)
• Re-use waste heat of IT
systems
• Use newest
semiconductor
technology
• Use of energy saving
processor and memory
technologies
• Consider using special
hardware or accelerators
tailored for solving
specific scientific
problems or numerical
algorithms
• Monitor the energy
consumption of the
compute systems and
the cooling infrastructure
• Use energy aware
system software
to exploit the energy
saving features of your
target platform
• Monitor and optimize
the performance of your
scientific applications
Energy efficient
hardware
Energy efficient
infrastructure
Energy aware
software environment
• Use most efficient
algorithms
• Use best libraries
• Use most efficient
programming paradigm
Energy efficient
applications
ISC ’12, Hamburg, Thursday Keynote, June 21, 2012
![Page 8: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/8.jpg)
8
Total cost of ownership and environmental footprint
Servers used 330 TWh of electrical energy: $25bn or 2% of the
electricity production (2009).
Data Center Market Drivers and Trends
Torsten Bloth, IBM Lab Services - © IBM Corporation
![Page 9: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/9.jpg)
9
Air-cooled datacenters are inefficient.
Typical cooling needs as much energy as IT equipment and both are
thrown-away.
Provocative: datacenter is a huge “heater with integrated logic.”
APC, Whitepaper #113 (2010)
Energy Consumption of Datacenter
Torsten Bloth, IBM Lab Services - © IBM Corporation
![Page 10: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/10.jpg)
10
Comparison: Water Cooling vs. Air Cooling
Torsten Bloth, IBM Lab Services - © IBM Corporation
![Page 11: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/11.jpg)
11
Passive Rear Door Heat Exchanger (RDHx) provides up to 100% cooling
– No condensation, no need for reheat or humidification
– No moving parts
CDU creates fully isolated, temperature controlled secondary loop
IBM Rear Door Heat Exchangers
Torsten Bloth, IBM Lab Services - © IBM Corporation
![Page 12: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/12.jpg)
12
Direct Water Cooling
Torsten Bloth, IBM Lab Services - © IBM Corporation
![Page 13: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/13.jpg)
IBM iDataplex dx360 M4
• Heat flux > 90% to water; very low chilled water requirement
• Power advantage over air-cooled node:
• Warm water cooled ~10%
(cold water cooled ~15%)
• due to lower Tcomponents and no fans
• Typical operating conditions: Tair = 25 – 35°C, Twater = 18 – 45°C
13 Torsten Bloth, IBM Lab Services - © IBM Corporation
![Page 14: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/14.jpg)
iDataplex DWC Rack w/ water cooled
nodes
(rear view of water manifolds)
IBM System x iDataPlex Direct Water Cooled Rack
iDataplex DWC Rack w/ water cooled
nodes
(front view) 14 Torsten Bloth, IBM Lab Services - © IBM Corporation
![Page 15: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/15.jpg)
Hybrid Datacenter w/ Direct Water Cooled Nodes
• Highly energy-efficient hybrid-cooling solution:
– Compute racks
• 90% Heat flux to warm water
• 10% Heat flux to CRAH
– Switch / Storage racks
• Rear door heat exchangers
• Compute node power consumption reduced ~ 10% due to lower component
temperatures and no fans.
• Power Usage Effectiveness PTotal / PIT: PUE ~ 1.1
15 Torsten Bloth, IBM Lab Services - © IBM Corporation
![Page 16: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/16.jpg)
16 ISC ’12, Hamburg, Thursday Keynote, June 21, 2012
![Page 17: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/17.jpg)
17
From Computer Cube to Computer Cuboid
Nearly double floor space
Supercomputer
Linux-Cluster,
Hosting/Housing, Servers
(Tape) Archive/Backup, Disks
Cooling, Water Air Processing
Power, Transformers, UPS,
ISC ’12, Hamburg, Thursday Keynote, June 21, 2012
![Page 18: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/18.jpg)
18
New Cooling Concept: Dedicated Free Cooling Loop
18 ISC ’12, Hamburg, Thursday Keynote, June 21, 2012
![Page 19: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/19.jpg)
Infrastructure 3rd floor
19 ISC ’12, Hamburg, Thursday Keynote, June 21, 2012
![Page 20: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/20.jpg)
Infrastructure 3rd floor
20 ISC ’12, Hamburg, Thursday Keynote, June 21, 2012
![Page 21: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/21.jpg)
Infrastructure 4th floor
21 ISC ’12, Hamburg, Thursday Keynote, June 21, 2012
![Page 22: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/22.jpg)
Infrastructure roof
22 ISC ’12, Hamburg, Thursday Keynote, June 21, 2012
![Page 23: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/23.jpg)
Reconsider TCO
GPGPUs for Dusty Decks?
Manycore for Scientific Applications
Statistics at LRZ: Percentage of max performance for different application areas
Minimizing Energy Maximizes Programming Cost and Minimizes System Performance Yield
We need tools to find a global optimum!
23 ISC ’12, Hamburg, Thursday Keynote, June 21, 2012
![Page 24: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/24.jpg)
Performance per core by Research area
24
![Page 25: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/25.jpg)
Some Results
Node Power Consumption and Cooling Infrastructure
Using the characteristics of new processor products (SB EP)
• Power Efficiency depends on clock speed (parabolic curve)
• Power Efficiency depends on voltage (lower is better)
• Turbo mode is good for TOP500 - not for Power Efficiency
• Energy loss with increased chip temperature Cool down as much as possible (cost for cooling?) Use liquid instead of air
• Disable unnecessary units (virtualization, …)
• Need to adjust processor, memory - and interconnect speed
25 ISC ’12, Hamburg, Thursday Keynote, June 21, 2012
![Page 26: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/26.jpg)
Measurement data
26
0
5000
10000
15000
20000
25000
30000
35000
40000
45000
50000
0
500
1000
1500
2000
2500
3000
3500
4000
16:30 18:00 19:30 21:00 22:30 0:00 1:30 3:00 4:30 6:00 7:30 9:00 10:30 12:00 13:30
En
erg
y C
on
su
mp
tio
n (
kW
h)
Po
wer
(kW
)
Time (Clock)
SuperMUC HPL Power Consumption (Infrastructure + Machine Room + PDU Measurement)
Power (Machine Room, kW)
Power (PDU, kW)
Power (infrastructure, kW)
Energy (Machine Room, kWh)
Integrated Power (PDUs, kWh)
HPL Start
HPL End
ISC ’12, Hamburg, Thursday Keynote, June 21, 2012
![Page 27: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/27.jpg)
Dagstuhl Perspectives Workshop:
Deutscher Rechenzentrumspreis 2012
![Page 28: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/28.jpg)
SuperMUC users and network
28
LRZ / BAdW
MCSC, BGCE, ENB
KONWIHR
GCS,GA
PRACE
PROSPECT
ETP4HPC
• HPC for science
• HPC for industry
• Support and education
• Future System and Application
research and development
![Page 29: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/29.jpg)
Usage 2010
by Research Area
29
![Page 30: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/30.jpg)
Telescope
Weight: 17.000 kg
Diameter: 2.7m
Elevation range: 20° - 60°
Developed and built in Germany
Platform: Boeing 747 SP
Max. Gross Weight: 300.000 kg
Wingspan: 60m
Max. Range: 15.000km
Stratospheric Observatory For Infrared Astronomy -
SOFIA
30 ISC ’12, Hamburg, Thursday Keynote, June 21, 2012
![Page 31: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/31.jpg)
Shear Layer Visualization
31
Sven Schmid Institut für Aerodynamik
und Gasdynamik
ISC ’12, Hamburg, Thursday Keynote, June 21, 2012
![Page 32: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/32.jpg)
Shear Layer Visualization
32
Sven Schmid Institut für Aerodynamik
und Gasdynamik
ISC ’12, Hamburg, Thursday Keynote, June 21, 2012
![Page 33: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/33.jpg)
SOFIA configuration
with fence
SOFIA configuration
without fence
SOFIA Configuration with Fence
33 ISC ’12, Hamburg, Thursday Keynote, June 21, 2012
![Page 34: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/34.jpg)
Numerical investigation of the noise from
modified nozzle geometries
34
Rolls-Royce Trent 1000 engine on a Boing 787
Dreamliner (copyright Rolls-Royce plc 2010 )
Iso-surfaces of the density showing resolved turbulent flow
structures
a) standard DES;
b) modified DES, unserrated nozzle;
c) modified DES, serrated nozzle;
d) serrated short-cowl nozzle surface mesh with every second
grid line shown
Institute of Fluid Mechanics and Engineering Acoustics,
University of Technology Berlin
Principal Investigator
F. Thiele
Researchers
D. Eschricht, J. Yan, L. Panek and K. Tawackolian
Project Partners
Rolls-Royce Germany
ISC ’12, Hamburg, Thursday Keynote, June 21, 2012
![Page 35: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/35.jpg)
Global high-resolution gravity field
determination
35
Research Institution
Institute for Astronomical and Physical Geodesy
Principal Investigator
Thomas Gruber
Researchers
Thomas Fecher, Prof Roland Pail
The geoid, the physical
shape of the Earth
Gravity anomalies (10-3m/s2)
ISC ’12, Hamburg, Thursday Keynote, June 21, 2012
![Page 36: Energy-efficient High Performance Computing with SuperMUC · 2012-06-21 · Energy-efficient High Performance Computing with SuperMUC By Ernst A. Graf TERATEC Forum 2012, Toward](https://reader033.vdocuments.net/reader033/viewer/2022041814/5e59799d68f2b32afb4a020b/html5/thumbnails/36.jpg)
36
SuperMUC and after
SuperMUC Phase 2 in 2014: manycore technology
better energy efficiency
LRZ in Exascale projects: DEEP (Intel MIC and Xtoll)
Mont-Blanc (ARM technology)
EESI
Successor to SuperMUC needs strong support for users:
Scalability issues for the „Mega-core-system“
New „HPC styles“: Big Data
Realtime HPC
Integrated Visualization
Steering
ISC ’12, Hamburg, Thursday Keynote, June 21, 2012