local hpc resources - cyber infrastructure and advanced...
TRANSCRIPT
Local HPC ResourcesTimothy H. Kaiser, Ph.D.
1
Friday, August 12, 11
Overview
• GECO and RA
• Who is doing what?
• Mio
2Friday, August 12, 11
Golden Energy Computing Organization
GECO - a quick overview
3
Friday, August 12, 11
Golden Energy Computing
OrganizationFront Range High Performance Computing
dedicated to the
Energy Sciences
4
Friday, August 12, 11
GECO - put CSM in the HPC Game
• Computational hub for finding new ways to meet the energy needs of our society
• Energy node for Front Range high performance computing
• Intended Impacts:
• Advance energy research
• Attract large-scale, multi-group projects
• Foster education in high performance computing
• Promote Front Range high performance computing
5Friday, August 12, 11
GECO Partners
• Colorado School of Mines
• National Renewable Energy Laboratory
• National Center for Atmospheric Research
• National Science Foundation
6Friday, August 12, 11
A Balanced Energy Portfolio: 4 Facets
Pursue Renewable Resources
Locate/DevelopExisting Resources
Advance Environmental Stewardship
Design NewMaterials
Friday, August 12, 11
HydrateNucleation/Growth
HydrocarbonDeposit
Characterization
1: Locate/Develop Existing Resources
Locate/DevelopExisting Resources
• Improve physicality of solid earth models
• Better local characterization of fluid dynamics in reservoirs
• Simulate nucleation and growth of hydrates
Seismic imaging
Multi-scale analysis of tight formationsHydrate Physics
Friday, August 12, 11
CarbonSequestrationCO2 Emissions
Advance Environmental Stewardship
2: Advance Environmental Stewardship
• Simulate climate/energy scenarios
• Quantify CO2 trapping, isolation and immobilization mechanisms at depth
Friday, August 12, 11
PolymerBatteries
UltracoldDesigner Solid State Systems
3: Design New Materials
Design NewMaterials
• Optimize molecular structures for ion conduction
• “Disentangle” dynamics of ultracold atoms in optical lattices
Pouch packaging of polymer Lithium ion battery
1-D quantum depletion map as function of lattice barrier height and chemical potential
Friday, August 12, 11
Biomass EnergyConversion
Photo-Productionof Hydrogen
4: Pursue Renewable Resources
Pursue Renewable Resources
• Identify mechanism by which cellulose-degrading enzymes act
• Quantify band bending and charge transfer between POM and substrate
[SiW10V1O39]-9 lacunary on TiO2 Friday, August 12, 11
HydrateNucleation/Growth
HydrocarbonDeposit
Characterization
CarbonSequestration
Biomass EnergyConversion
Photo-Productionof Hydrogen
PolymerBatteries
UltracoldDesigner Solid State Systems
CO2 Emissions
A Balanced Energy Portfolio: 4 Facets, 8 Challenges
Pursue Renewable Resources
Locate/DevelopExisting Resources
Advance Environmental Stewardship
Design NewMaterials
Friday, August 12, 11
• Architecture• Dell with Intel quad-core, dual-socket system • 2144 processing cores in 268 nodes
256 nodes with 512 Clovertown E5355 (2.67 GHz) (quad core dual socket) 184 with 16 Gbytes & 72 with 32 Gbytes 12 nodes with 48 Xeon 7140M (3.4 GHz) (quad socket dual core) 32 Gbytes each
• Memory• 5,632 Gbytes ram (5.6 terabytes) • 300 terabyte disk • 300 terabyte tape back up• 16/32 gigabytes RAM per node
• Performance
• 18 Tflop sustained performance • 23 Tflop peak• like every human on the planet doing 2500 calculations per second
GECO HPC Hardware/Software: “Ra”
Center for Technology and Learning Media
13Friday, August 12, 11
14
GECO Projects
•Alexandra Newman Modeling the Benefits of Energy Storage to Decrease Effects of Partial Loading of Thermal Power Plants•Amadeu K. Sum Molecular Biophysics of Biomembranes and Bionanoconstructs•Amadeu K. Sum Computational Studies of Clathrate Hydrates•Branden Kappes Materials for high density energy storage•Cristian V Ciobanu Structure and Morphology of Graphene Sheets for Carbon-Based Nanoelectronics (NSF CMMI-0825592 )•DV Griffiths Probabilistic Geomechanical Analysis In The Exploitation Of Unconventional Resources•Hossein Kazemi Parallel Computation in Reservoir Simulation of Enhanced Oil and Gas Recovery•Ivar Reimanis, C.V. Ciobanu Mechanical Behavious in Ceramics with Unusual Thermo-Mechanical Properties•Jeffrey C. King TRISO fuel particles containing a burnable poison layer•Lincoln Carr Proposal Title: Quantum Many Body Physics with Ultracold Quantum Gases in Optical Lattices•Mahadevan Ganesh High-performance computational algorithms for scattering by multiple three dimensional particles•Mahadevan Ganesh Simulation of Critical Dynamics in Superconductivity Models•Mark Coffey Quantum lattice gas algorithms for image enhancement•Mark T. Lusk Fission Product Gas Transport in Uranium Dioxide•Mark T. Lusk First Principles Design of Advanced Hydrogen/Carbon Dioxide Membrane Separations•Mark T. Lusk Palladium Alloy Membranes for Hydrogen Purification•Mark T. Lusk Quantum Transport Between Silicon Quantum Dots•Mark T. Lusk Functionalization of Silicon Quantum Dots for Photovoltaic Applications
http://geco.mines.edu/winter2011/Friday, August 12, 11
15
GECO Projects
•Mark T. Lusk Structural and Excited State Properties of Silicon Clathrate Quantum Dots•Mark T. Lusk Density Matrix Estimation of Dephasing Rates of Cat States Involving Two Silicon Quantum Dots•Mark T. Lusk Defect Engineering of Graphene for Low Energy Electronics•Michael Kaufman Phase Transformation and Equilibria of Titanium-Platinum Alloys in the Composition Range 30-50 at.% Pt•Reed Maxwell Forecasts for wind-energy using fully-coupled simulations•Reed Maxwell Heterogeneous simulation of groundwater-surface water interactions for quantifying and predicting surface-groundwater mixing and nutrient delivery in the Santa Fe River, Florida
•Reed Maxwell An Integrated Framework for Simulating Risk from CO2 Leakage Into Groundwater•Reed Maxwell Understanding role of management in hydropower generation in the Klamath Basin•Reed Maxwell Climate Change Impacts on Water Supply and Hydropower Generation•Ryan O Hare An In Silico Study of the Hydration Properties of Yttrium-doped Barium Cerate for the Development of a Novel Ionic Conducting Material
•Xiaolong Yin Numerical simulation of multiphase flow, tracer and particle transport in porous media•Zeev Shayer Advanced Nuclear Battery Reactor•Zhigang Wu First-Principles Calculations of Graphene Nanomesh•Zhigang Wu Electronic Structures of Solid Hydrogen under Ultrahigh Pressures•Zhigang Wu Exciton binding energies in Si, ZnO, and nitrides: the role of electron-state localization•Zizhong (Jeffrey) Chen Optimizing Multi-dimensional MPI Communications on Multi-core Architectures•Zizhong (Jeffrey) Chen Fault Tolerant Extreme Scale Computing: An Algorithmic Approach
http://geco.mines.edu/winter2011/Friday, August 12, 11
• Architecture• Dell with Intel quad-core, dual-socket system • 2144 processing cores in 268 nodes
256 nodes with 512 Clovertown E5355 (2.67 GHz) (quad core dual socket) 184 with 16 Gbytes & 72 with 32 Gbytes 12 nodes with 48 Xeon 7140M (3.4 GHz) (quad socket dual core) 32 Gbytes each
• Memory• 5,632 Gbytes ram (5.6 terabytes) • 300 terabyte disk • 300 terabyte tape back up• 16/32 gigabytes RAM per node
• Performance
• 18 Tflop sustained performance • 23 Tflop peak• like every human on the planet doing 2500 calculations per second
GECO HPC Hardware/Software: “Ra”
Center for Technology and Learning Media
16Friday, August 12, 11
17Friday, August 12, 11
RA problems
• Only for Energy Science
• Allocation is by proposal
• People wanted quick access and RA is heavily loaded
• Little student access
18Friday, August 12, 11
MIOMio.Mines.Edu
Mio.Mines.edu
19
Friday, August 12, 11
MIOMio.Mines.Edu
It’s AllMine
Mio.Mines.edu
19
Friday, August 12, 11
Mio.mines.edu
• New concept in HPC for CSM
• School puts up the money for infrastructure
• Researchers purchase individual nodes
• They own the nodes
• Can use other’s when they are not in use
• Started with the head node and compute 4 nodes, 32 cores
• Documentation: http://inside.mines.edu/mio
20Friday, August 12, 11
Original Mio Compute Nodes
Have since gone to 12 core nodes and 1 Tbytes disk
• Penguin
• 2 x Dual Intel Xeon X5570 Quad Core 2.93GHz 8MB max RAM speed 1333MHz
• up to 2 x 48GB DDR3-800 REG, ECC (24 x 4GB)
• 2 x 160GB, SATA, 7200RPM
• 2 x Intel Xeon Dual Socket Motherboard with Integrated Infiniband DDR/CX4 Connections
• Half the size of RA nodes
• More efficient in power and computation
• More memory
• Faster Clock
21Friday, August 12, 11
August 2011
22
616 Intel cores - 7.22 Tflop GPU: 2304 cores - 6.82TflopStorage: 86.36 Tbytes = 94,952,697,323,520 bytes
Friday, August 12, 11
Node Summary
Node Cores RAM GB DISK Free (/scratch)
GB0 8 48 1281 8 48 1282 8 48 1283 8 48 1284 8 192 1285 8 48 128
6 to 31 8 24 12832-47,49-52 12 24 849
48 960 GPU 16 -53 1344 GPU 18 -
23Friday, August 12, 11
Mio: Installing a 200 Tbyte File system
• Panasas system
• Used by BP
• Donated to CSM
• Panasas gave us a one shot “deal” on software
• Was working but we need to rebuild after a switch upgrade
• Available in parallel to all nodes
24Friday, August 12, 11
Mio: Installing a 200 Tbyte File system
• Panasas system
• Used by BP
• Donated to CSM
• Panasas gave us a one shot “deal” on software
• Was working but we need to rebuild after a switch upgrade
• Available in parallel to all nodes
24Friday, August 12, 11
File Systems on Mio
• Four partitions
• $HOME - should be kept very small, having only start up scripts and other simple scripts
• $DATA - Should contain programs you have built for use in your research, small data sets and run scripts.
• $SCRATCH - The main area for running applications. Output from parallel runs should be done to this directory.
• $SETS - This area is for large data sets, mainly read only, that will be use over a number of months.
25Friday, August 12, 11
What’s in a Name?
• The name Mio is a play on words.
• It is a Spanish translation of the word “mine” as in belongs to me, not the hole in the ground.
• The phrase “The computer is mine.” can be translated as “El ordenador es mío.”
26Friday, August 12, 11
Mio.mines.edu
Compute Nodes
Panasasfile
system
Head Node
Tux
Networkswitch
27Friday, August 12, 11
• GECO main page
• http://geco.mines.edu/
• User guide
• http://geco.mines.edu/guide
• Quick Start Guide (Run a job on RA via Copy/Paste)
• http://geco.mines.edu/guide/quickstart/
“If I had read the web pages I could have saved us both a lot of time.”
28Friday, August 12, 11
• RA User agreement
• http://geco.mines.edu/guide/agreement.shtml
• Software (compilers, libraries, debuggers, profilers)
• http://geco.mines.edu/software
• Library examples
• http://geco.mines.edu/software/mkl/index.shtml
• I am starting from scratch thinking there might be a LAPACK routine that can help me. What do I do?
• http://geco.mines.edu/software/mkl/casestudy.html
• Ra electronic notebook (blog)
• http://inside.mines.edu/~tkaiser/data/books/rabook.html
29Friday, August 12, 11
• Job information on RA
• http://ra.mines.edu/ganglia/
• http://ra.mines.edu/ganglia/addons/rocks/top.php
• http://ra.mines.edu/jobs/
• Job information on Mio
• http://mio.mines.edu/ganglia/
• http://mio.mines.edu/inuse/
• http://mio.mines.edu/jobs/
30Friday, August 12, 11
• Mio HOME
• http://inside.mines.edu/mio/index.html
• User Guide
• http://inside.mines.edu/mio/page3.html
• File System
• http://inside.mines.edu/mio/page4.html
• GPU nodes
• http://inside.mines.edu/mio/page6.html
31Friday, August 12, 11