National Center for Supercomputing Applications

University of Illinois at Urbana-Champaign

GPU Clusters for HPC

Edward Seidel

Director

Bill Kramer

Blue Waters Project Director

National Center for

Supercomputing Applications

University of Illinois at Urbana-

Champaign

WHAT IS NCSA

National Center for Supercomputing Applications

Imaginations unbound

National Center for Supercomputing Applications: 30 years of leadership

• NCSA

• R&D unit of the University of Illinois at Urbana-Champaign

• One of original five NSF-funded supercomputing centers

• Mission: Provide state-of-the-art computing capabilities (hardware, software, hpc

expertise) to nation’s scientists and engineers

• The Numbers

• Approximately 200 staff (160+ technical/professional staff)

• Approximately 15 graduate students (+ new SPIN program), 15 undergrad students

• Two major facilities (NCSA Building, NPCF)

• Operating NSF’s most powerful computing system: Blue Waters

• Managing NSF’s national cyberinfrastructure: XSEDE

Source: Thom Dunning

Petascale Computing Facility: Home to Blue

Waters

• Modern Data Center

• 90,000+ ft2 total

• 30,000 ft2 raised floor

20,000 ft2 machine room gallery

• Energy Efficiency

• LEED certified Gold

• Power Utilization Efficiency = 1.1–1.2

• Blue Waters

• 13PF, 1500TB,

300PB

• >1PF On real apps

• NAMD, MILC,

WRF, PPM,

NWChem, etc

Source: Thom Dunning

Data Intensive Computing

Source: Thom Dunning

LSST, DES Personalized Medicine w/ Mayo

NCSA’s Industrial Partners

Source: Thom Dunning

PARTNERSHIP FOR

INNOVATION

NCSA and NVIDIA

Imaginations unbound

NCSA, NVIDIA and GPUs

• NCSA and NVIDIA have been partners for over a

decade, building the expertise, experience and

technology.

• The efforts were at first exploratory and small scale, but

have now blossomed into providing the largest GPU

production resource in the US academic cyber-

infrastructure

• Today, we are focusing on helping world class science

and engineering teams decrease their time to insight for

some of the world’s most important and challenging

computational and data analytical problems

Imaginations unbound

Innovative Systems Lab 1.0 Research

• Evaluation of emerging

computing architectures

• Reconfigurable computing

• Many-core (GPU) architecture

• Heterogeneous clusters

• Systems software research

and development

• Run-time systems

• GPU accelerator cluster

management

• Tools and utilities: GPU

memory test, power

profiling, etc.

• Power Measurement and

efficiency

Imaginations unbound

QP: first GPU cluster at NCSA

16 HP xw9400 workstations

32 Quadro Plex Computing Servers

How might this be used for Science??

• Focus on Application

development for emerging

computing architectures – not

benchmarking

• Cuda Center of Excellence in

the Coordinated Systems

Laboratory

• Computer Science efforts in

GPUs

• A variety of discipline

departments

• Computational chemistry

(electronic structure, MD)

• Computational physics (QCD)

• Cosmology and astronomy

• Data mining

Imaginations unbound

MD Simulations and Computational

Microscope; see John Stone!

Turbulence simulations; Paul

Woodward

FROM PIONEERING TO

PRODUCTION

Imaginations unbound

Blue Waters Computing System

Sonexion: 26 usable PB

>1 TB/sec

100 GB/sec

10/40/100 Gb Ethernet Switch

Spectra Logic: 300 usable PB

120+ Gb/sec

100-300 Gbps WAN

IB Switch

External Servers

Aggregate Memory – 1.6 PB

Imaginations unbound

Original Blue Waters Goals

• Deploy a computing system capable of sustaining more than one

petaflops or more for a broad range of applications • Cray system achieves this goal using a well defined metrics

• Enable the Science Teams to take full advantage of the sustained

petascale computing system • Blue Waters Team has established strong partnership with Science Teams, helping them to

improve the performance and scalability of their applications

• Enhance the operation and use of the sustained petascale system • Blue Waters Team is developing tools, libraries and other system software to aid in operation of

the system and to help scientists and engineers make effective use of the system

• Provide a world-class computing environment for the petascale

computing system • The NPCF is a modern, energy-efficient data center with a rich WAN environment (100-400

Gbps) and data archive (>300 PB)

• Exploit advances in innovative computing technology • Proposal anticipated the rise of heterogeneous computing and planned to help the computational

community transition to new modes for computational and data-driven science and engineering

Imaginations unbound

Details of Blue Waters

Imaginations unbound

Production Computation Science with XK

nodes • The Computational Microscope

• PI – Klaus Schulten

• Simulated flexibility of ribosome trigger factor complex at

full length and obtained better starting configuration of

trigger factor model (simulated to 80ns)

• 100ns simulation of cylindrical HIV 'capsule’ of CA proteins

revealed it is stabilized by hydrophobic interactions

between CA hexamers; maturation involves detailed

remodeling rather than

disassembly/re-assembly of CA lattice, as had been

proposed.

• 200ns simulation of CA pentamer surrounded by CA

hexamers suggested interfaces in hexamer-hexamer and

hexamer-pentamer pairings involve different patterns of

interactions

• Simulated photosynthetic membrane of a chromatophore in

bacterium Rps. photometricum for 20 ns -- simulation of a

few hundred nanoseconds will be needed

Images from Klaus Schulten and John Stone, University of Illinois at Urbana-Champaign

Imaginations unbound

XK7 For Visualization on Blue Waters

• Many visualization utilities rely on the OpenGL API for

hardware-accelerated rendering

• Unsupported by default XK7 system software

• Enabling NVIDIA’s OpenGL required that we:

• Change operating mode of the XK7 GPU firmware

• Develop a custom X11 stack

• Work with Cray to acquire alternate driver package from NVIDIA

• Blue Waters is the first Cray to offer this functionality

which has been distributed to other systems now

Imaginations unbound

Impact: VMD

• Molecular dynamics analysis and

visualization tool used by “The

Computational Microscope”

science team (PI Klaus Schulten)

• 10X to 50X rendering speedup in

VMD

• Interactive rate visualization

• Drastic reduction in required time to

fine tune parameters for production

visualization

Imaginations unbound

Summary

• NVIDIA GPUs are critical to the

nation’s most powerful academic

supercomputer

• Partnership with NCSA, Cray,

NVIDIA

• Science, engineering, private sector

partner development all depend on it

• Enable research that cannot be done

in any other way, on any other

system

• See John Stone presentation, Bill

Kramer presentation for more

details

Imaginations unbound