systems controls @lrz - lawrence livermore national laboratory · • requirements of liquid cooled...
TRANSCRIPT
![Page 2: Systems Controls @lrz - Lawrence Livermore National Laboratory · • Requirements of liquid cooled systems for BMS • Requirements of large HPC systems for systems control • Roadmaps](https://reader034.vdocuments.net/reader034/viewer/2022042117/5e95906815db0a427c021c2b/html5/thumbnails/2.jpg)
17.09.2014 Leibniz-Rechenzentrum2
Agenda
• Introduction
• Power Infrastructure
• Cooling Infrastructure
• IT systems
• Issues @lrz.de
• Discussion
![Page 3: Systems Controls @lrz - Lawrence Livermore National Laboratory · • Requirements of liquid cooled systems for BMS • Requirements of large HPC systems for systems control • Roadmaps](https://reader034.vdocuments.net/reader034/viewer/2022042117/5e95906815db0a427c021c2b/html5/thumbnails/3.jpg)
Leibniz Supercomputing Centre
-3-
Munich Bavaria Germany & Europe
• We provide generic IT services to all Munich universities
• We provide special IT services to all universities in Bavaria• Network, High Performance and Grid Computing
• Backup and Archive Services
• IT Management
• We provide supercomputing resources to scientists in Europe• Member of the German Gauss Supercomputing Centre
• Part of the European HPC Infrastructure PRACE
• Operating Tier-0 Supercomputing Center (SuperMUC system)
• Investigations on Future HPC Systems:
• Hardware Architectures
• Programming Models & System Software
• Zero Emission Data Center
• Re-Use of Waste Heat
![Page 4: Systems Controls @lrz - Lawrence Livermore National Laboratory · • Requirements of liquid cooled systems for BMS • Requirements of large HPC systems for systems control • Roadmaps](https://reader034.vdocuments.net/reader034/viewer/2022042117/5e95906815db0a427c021c2b/html5/thumbnails/4.jpg)
SuperMUC: IBM System x iDataPlex
With Direct Water Cooling
-4-Torsten Bloth, IBM Lab Services - © IBM Corporation
iDataplex DWC Rack w/ water cooled nodes
(rear view of water manifolds)
![Page 5: Systems Controls @lrz - Lawrence Livermore National Laboratory · • Requirements of liquid cooled systems for BMS • Requirements of large HPC systems for systems control • Roadmaps](https://reader034.vdocuments.net/reader034/viewer/2022042117/5e95906815db0a427c021c2b/html5/thumbnails/5.jpg)
Data Center Infrastructure
-5-
![Page 6: Systems Controls @lrz - Lawrence Livermore National Laboratory · • Requirements of liquid cooled systems for BMS • Requirements of large HPC systems for systems control • Roadmaps](https://reader034.vdocuments.net/reader034/viewer/2022042117/5e95906815db0a427c021c2b/html5/thumbnails/6.jpg)
Layout Power Infrastructure
![Page 7: Systems Controls @lrz - Lawrence Livermore National Laboratory · • Requirements of liquid cooled systems for BMS • Requirements of large HPC systems for systems control • Roadmaps](https://reader034.vdocuments.net/reader034/viewer/2022042117/5e95906815db0a427c021c2b/html5/thumbnails/7.jpg)
Controls Power Infrastructure
• Equipment:- Transformer, switching, ...: SIEMENS
- Dyn UPS: Piller
- Battery backup: Emerson
- Diesel generator: MTU
• Metering- SOCOMEC&WinCC (power)
- Piller&WinCC (power, messaging)
- JCI Metasys M5 (power)
- deZem (power)
- SWM - Utility Provider (power)
• Monitoring- Siemens WinCC
-7-
![Page 8: Systems Controls @lrz - Lawrence Livermore National Laboratory · • Requirements of liquid cooled systems for BMS • Requirements of large HPC systems for systems control • Roadmaps](https://reader034.vdocuments.net/reader034/viewer/2022042117/5e95906815db0a427c021c2b/html5/thumbnails/8.jpg)
-8-
Layout Cooling Infrastructure
![Page 9: Systems Controls @lrz - Lawrence Livermore National Laboratory · • Requirements of liquid cooled systems for BMS • Requirements of large HPC systems for systems control • Roadmaps](https://reader034.vdocuments.net/reader034/viewer/2022042117/5e95906815db0a427c021c2b/html5/thumbnails/9.jpg)
-9-
Controls Cooling Infrastructure
• Equipment:
- Cooling towers: Gohl, Jaeggi
- Chiller: McQuay, CARRIER
- CRAC/CRAH: GEA, WEISS, STULZ,
RC Group
- Pumps: Grundfoss/ABB&KSB
• Metering
- Krohne (flow)
- Calec (heat)
- WIKA a.o. (pressure, temperature)
• Monitoring & Operations
- JCI Metasys
![Page 10: Systems Controls @lrz - Lawrence Livermore National Laboratory · • Requirements of liquid cooled systems for BMS • Requirements of large HPC systems for systems control • Roadmaps](https://reader034.vdocuments.net/reader034/viewer/2022042117/5e95906815db0a427c021c2b/html5/thumbnails/10.jpg)
-10-
Monitoring of IT Systems
• SuperMUC
- Vendor solution: IBM tool set based on icinga
- Power and energy readings at server (PDU & Paddle cards
& RAPL counters) and system level
- Temperature at server level and room level
- Pressure/heat at system level
• CoolMUC
- Vendor solution: power/heat/temperature/flow control
• Clusters & servers, NAS systems
- Nagios based inhouse tools
• Tape libraries
• Networking
![Page 11: Systems Controls @lrz - Lawrence Livermore National Laboratory · • Requirements of liquid cooled systems for BMS • Requirements of large HPC systems for systems control • Roadmaps](https://reader034.vdocuments.net/reader034/viewer/2022042117/5e95906815db0a427c021c2b/html5/thumbnails/11.jpg)
-11-
Issues @lrz.de
• Power infrastructure
- Monitoring, reporting (dashboard)
- Quality of reported measurements
• Cooling infrastructure
- Ops of cooling loops (hydraulics, meta controls)
- Ops of cooling towers
• Information management
- Integration&consolidation of heterogeneous data sources
- Interoperatorbility of differing system controls
• General
- Interaction with vendors/contractors of BMS
- Strategy DCIM: in house/vendor based, open source
![Page 12: Systems Controls @lrz - Lawrence Livermore National Laboratory · • Requirements of liquid cooled systems for BMS • Requirements of large HPC systems for systems control • Roadmaps](https://reader034.vdocuments.net/reader034/viewer/2022042117/5e95906815db0a427c021c2b/html5/thumbnails/12.jpg)
-12-
Topics of Interest
• Requirements of liquid cooled systems for BMS
• Requirements of large HPC systems for systems control
• Roadmaps for BMS and DCIM
• Vendors view on status and trends in system controls
• Standardization
• APIs
• Lessons learned and white paper on
„Best practise in systems controls for HPC data centers“
![Page 13: Systems Controls @lrz - Lawrence Livermore National Laboratory · • Requirements of liquid cooled systems for BMS • Requirements of large HPC systems for systems control • Roadmaps](https://reader034.vdocuments.net/reader034/viewer/2022042117/5e95906815db0a427c021c2b/html5/thumbnails/13.jpg)
Zero Emission Supercomputing Centre
Thank You!
![Page 14: Systems Controls @lrz - Lawrence Livermore National Laboratory · • Requirements of liquid cooled systems for BMS • Requirements of large HPC systems for systems control • Roadmaps](https://reader034.vdocuments.net/reader034/viewer/2022042117/5e95906815db0a427c021c2b/html5/thumbnails/14.jpg)
Overview Cooling Infrastructure
Wasseraufbereitung
(3x Umkehrosmose)
UKG 6x
Dach
SuperMUC
Compute
Section (≈ 150 Racks)
HRR
Disks
Tape
Libraries
KW-Verteiler/-Sammler
Serv
ers
(2
5x K
KT
Kra
us R
acks)
Su
pe
rMU
C In
terc
on
ne
ct
(„R
DH
X“)
EGGelände
RLT 2x GEA
NSHVsKä
lte
mas
ch
ine
n5x+
2x
SuperMUC Storage
3.OG 2.OG
NSR
1.OG EG
RLT (2x)
Dir
ekte
Modu
l-K
üh
lun
g
30
– 6
0°C
Kühltürme
(4x Gohl)
Dunstturm
(1x Gohl)
Kühltürme
(2x Gohl
+ 5x Jäggi)
Brunnen
KKG
5x
Netw
ork
& C
ore
Serv
ers
I
Serv
ers
(≈
95
x R
acks)
UKGUKG
Präzisions-
kühler
„fre
ie K
ühlu
ng“
(Win
ter)
Rü
ckkühlu
ng
RLT (2x)
RLT 2xGEA
KKG
4x
Netw
ork
& C
ore
Serv
ers
II
UKG
5x
DAR
KKG
3x+3x
USVstat
3x+3x
UG
UKG
WKZ
NEA
USVdyn.
3x+6x
Trafo
6x+6x
Mittel-
SP
Elektro
Kältemaschinen
(2x McQuay
+ 5x Carrier)
Gelä
nd
e