Vijay K. AgarwalaSenior Director, Research Computing and Cyberinfrastructure

The Pennsylvania State UniversityUniversity Park, PA 16802 USAvijay@psu.edu, 814.865.2162

Research Computing: Critical Needs and Opportunities at a

University Based Academic Computing Center

HPC User Forum October 13 -14, 2008 High Performance Computing Center Stuttgart (HLRS),Germany

October 16th, 2008 Imperial College, London

Human Resources

Vice Provost for Information Technology

and CIO

Associate Vice Provost

Financial Services

Marketing and Communication

Research Computing and Cyberinfrastructure (RCC)

Consulting and Support Services (CSS)

Security Operationsand Services (SOS)

Teaching and Learningwith Technology (TLT)

Digital Library Technologies (DLT)

Administrative InformationServices (AIS)

Telecommunications and Networking Services (TNS)

Research

Security

Libraries

Instruction

Communication

Business

Support

ITS Organization

High PerformanceComputing Systems

Domain Expertise and

Consulting Support

Software Developmentand

Programming Support

Visualization andTelecollaborative

Systems

Senior Director

Faculty Advisory Committeeon Research Computing and Cyberinfrastructure

Information Technology Services

Research Computing and Cyberinfrastructure

Meets the high-end computing technology needs of scholars in their research and teaching endeavors.

The group partners with faculty members and collaborates with technology companies and other

research organizations.

Research Computing and Cyberinfrastructure• Provide systems services by researching current practices in operating system, file system,

data storage, job scheduling as well as computational support related to compilers, parallel computations, libraries, and other software support. Also supports visualization of large datasets by innovative means to gain better insight from the results of simulations.

• Enable large-scale computations and data management by building and operating several state-of-the art computational clusters and machines with a variety of architectures.

• Consolidate and thus significantly increase the research computing resources available to each faculty participant. Faculty members can frequently exceed their share of the machine to meet peak computing needs

• Provide support and expertise for using programming languages, libraries, and specialized data and software for several disciplines.

• Investigate emerging visual computing technologies and implement leading-edge solutions in a cost-effective manner to help faculty better integrate data visualization tools and immersive facilities in their research and instruction.

• Investigate emerging architectures for numerically-intensive computations and work with early-stage companies. For example: interconnects, networking, and graphics processors and FPGA for computations.

• Help build inter- and intra-institutional research communities using cyberinfrastructure and grid technologies.

• Maintain close contacts with NSF and DoE funded national centers, and help faculty members with porting and scaling of codes across multiple architectures.

Compute EnginesLION-XJ144 nodes288 quad-core procsMemory-16GB /nodeIB interconnect

LION-XK144 nodes288 quad-core procsMemory-32GB /nodeIB interconnect

Pleiades170 nodes340 processors340 GB RAM35.1 TB storage

LION-XO132 nodes368 processors1280 GB RAMSilverstorm Infiniband interconnect

Hammer/LION-XD16 nodes 64 processorsMemory-128GB /nodeInfiniband interconnect

LION-XB16 nodes128 processors512 GB RAMPathscale InfinipathInfiniband interconnect

LION-XC140 nodes560 processors1664 GB RAM

Unisys ES70003 nodes64 processors192 GB RAMMyrinet interconnect

HPC Storage Farm100 TB of diskneeds to be 1000 TB

Programs, Libraries, and Application Codes in Support of Computational Research

• Compilers and Debuggers: AbsoftProFortran, GNU Pascal, IBM XLF, IBM XLC/C++, Intel Fortran, Intel C/C++, Lahey/Fujitsu Fortran 95 Pro, Portland PGI Compilers and Tools, Java, PathScale EKO compiler suite, TotalView, DDT, Valgrind

• Computational Biology: BLAST, Blastall, Cister, ClustalW, ClustalX, Dotter, FASTA, fastDNAml, GeneMachine, GENSCAN, HMMgene, MrBayes, MZEF, PHRED/PHRAP/Consed, PHYLIP, ReadSeq, RepeatMasker, SEG, sim4, Sputnik, Treetool, wuBlast

• Computational Chemistry and Material Science: FHI98MD, Gamess, Gaussian 03, GaussView, SemiChem, NWChem, Jaguar, Maestro, CHARMM, WIEN2K, VASP, ThermoCalc, Accelrys Material Studio, ADF, tmolex, Amber, Gromacs, NAMD, WxDragon, Molden, CPMD, Rosetta, CCP4

• Finite Element Solvers: ABAQUS, ANSYS, FLUENT, GAMBIT, FIELDVIEW, LS-DYNA, MD/Nastran, OpenFOAM

• Mathematical and Statistical Libraries and Applications: ATLAS, BLAS, ESSL, IMSL, LaPack, ScaLaPack, MASS, GOTO, Intel MKL, AMD ACML, Mathematica, MATLAB, Maple, Distributed MATLAB, PETSc, NAG, StarP, Watson Sparse Matrix Solver

• Solid Modeling: MD/Patran

• Statistics: R, SAS

• Parallel Libraries: MPICH, Optimized versions of MPICH for high-performance cluster interconnects, Parallel IMSL, Distributed MATLAB, StarP, Distributed Maple

• Optimization: GAMS / CPLEX, Csim, Tomlab

• Multiphysics: Comsol

All software installations are driven by faculty. The software stack on every system is customized and entirely shaped by faculty needs.

Joint research and education initiative (NSF, DOE, PSU) focused on understanding molecular issues related to environmental chemical kinetics, geochemical cycling of elements, fate and transport of contaminants, and carbon sequestration. (Dr. Susan L. Brantley, Professor of Geosciences)

Virtual center that integrates genetic, immunological, ecological and other studies to understand how disease processes work, and how they inter-relate across time and length scales. (Dr. Ottar Bjornstad, Associate Professor of Entomology and Biology; Dr. Bryan Grenfell, Alumni Professor of Biology )

Center for Gravitational Wave Physics (NSF, PSU) fosters research of a truly interdisciplinary character linking the highest caliber astrophysics, gravitational wave physics and experimental gravitational wave detection in the pursuit of the scientific understanding of gravity. (Dr. Lee S. Finn, Professor of Physics, Astronomy and Astrophysics)

Interdisciplinary center (NSF, NIH, PSU) focused on identifying issues in statistics, research design, and measurement emerging in the prevention and treatment of problem behaviors, particularly drug abuse. (Dr. Linda M. Collins, Professor of Human Development and Family Studies, Statistics)

Participating Research Centers

Collaborative effort (NSF, PSU, Georgia Tech) aimed to educate the next generation of scientists and engineers in the emerging field of materials design. (Dr. Zi-Kui Liu, Professor of Materials Science and Engineering)

IceCube@Penn StateLarge collaboration (NSF, DOE, PSU, and others) of about 150 scientists working to use the AMANDA and IceCube telescopes to detect ultra-high energy neutrinos. (Dr. Douglas F. Cowen, Professor of Physics, Astronomy and Astrophysics)

Center aimed at describing, modeling, and understanding the Earth's climate system. (Dr. Michael E. Mann, Associate Professor of Meteorology)

A hub to connect experimental and simulation activities through the organization of collaborative projects, short courses and workshops. (Dr. Jorge O. Sofo, Associate Professor of Physics, Astronomy and Astrophysics )

Participating Research Centers

$10,

000

$100

,000

$1,0

00,0

00

$10,

000,

000

$100

,000

,000

$1,0

00,0

00,0

00

$2,0

00,0

00,0

00

$5,0

00,0

00,0

00

$10,

000,

000,

000

B&H

Myopic Filter

Filter-to-End

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

Initial Investment

2-layer ONIOM MethodHF/6-311+G(d,p)/HF3-21G*

Low level High level -90.4 ppm

HPC

Performance of LS-DYNA Case Study – Blast Loading

• Blast load using ConWep algorithm• 450,000 dofs, spherical blast for 11ms• Study of mesh convergence for plastic strain stability

Professor: Ashok D.BelegunduStudent : Vikas ArgodDept of Mechanical and Nuclear Engineering

Visualization Services

Recent support areas include:

• Visualization system design and deployment

• 3D modeling and geometry exchange (e.g. FormZ)

• Visualization development applications and programming toolkits (e.g. OpenDX, VTK )

• VR development and device libraries (e.g. VRPN, VMRL, JAVA3D, OpenGL, OpenSG, CaveLib)

• Domain specific visualization tools (e.g. VMD, SCIRun)

• Telecollaborative tools and facilities support (e.g. Access Grid, VNC)

• Parallel graphics and online visualization (e.g. Paraview, DCV)

• Programming for graphics (e.g. C/C++, JAVA3D, Tcl/Tk, Qt)

Staff members provide consulting, teach seminars, assist faculty and support facilities for visualization and VR.

Visualization Facilities

• Locating facilities strategically across campus for convenient access by targeted disciplines and user communities

• Leveraging existing applications and workflows so that visualization and VR can be natural extensions to existing work habits for the users being served

• Providing outreach seminars, end-user training and ongoing staff support in use of the facilities

• Working on an ongoing basis with academic partners to develop and adapt these resources more precisely to fit their needs

• Helping to identify and pursue funding opportunities for further enhancement of these efforts as they take root

Our goal is to foster more effective use of visualization and VR techniques in research and teaching across colleges and disciplines via strategic deployment of facilities and related support.

Immersive Environments Lab (IEL) in partnership with School of Architecture and Landscape Architecture

208 Stuckeman Family Building

• Focused on teaching and research in the experiential understanding of design spaces by architecture and landscape architecture students

• 3-screen 3D-stereo multi-OS display offers multi-modal immersive environment

• Experiential design review in design studio and digital media courses

• Telecollaborative studio using standard definition video and 3D application sharing with Carleton University, Spring 2007 (lab also supports Access Grid)

• Research into immersive and collaborative tools for design professions

• Develop application and data integration workflows for ARCH/LARCH (Building Information Modeling, energy and structural analyses, land use planning, etc.) in conjunction with ICON lab under internal Bowers support

Immersive Construction Lab (ICon) in partnership with Architectural Engineering

306 Engineering Unit C

• Focused on research and teaching in the use of immersive visualization and VR techniques for planning and management of large construction projects

• 3-screen, 3D-stereo, Windows desktop, immersive information environment

• VR extensions to commercial construction planning applications, custom development of “VR-like” teaching modules

• Industry partners provide real world use cases for studying the practical application of tools under development

• Linked SMART Board allows dynamic VR updates from scheduling applications, etc.

• Laptop display sharing facilitates group collaboration among students

• Building upon ICON lab, IEL and related work, Penn State hosted CONVR 2007 international conference on construction applications of virtual reality

Visualization/VR Lab, 215 Osmondin partnership with Materials Simulation Center

• VR facility for central campus science community (Materials Science, Molecular Biology, Physics and more)

• 8’ x 8’ active stereo display • Tracked devices for user interaction• Linux console workstation• Complement of open source data

visualization tools (VMD, VTK, Paraview - can be built upon in response to user needs)

• Seminars for teaching graduate students on use of VR tools

• Initial users in Materials Science and Molecular Biology

Visualization/VR Lab, 336 IST Building in partnership with Computer Science and Engineering

• Facility targets compute-intensive applications in science, engineering and related disciplines

• Large-format 3D stereo display (6.75 x 9 ft., 1400 x 1050 pixel) for VR applications

• 2 x 2 tiled display (6 x 8 ft., 2800 x 2100 pixel) for high-resolution applications

• Linux console workstations

• Interactive device support• Initial complement of data visualization tools

(VMD, VTK, Paraview, SCIRun) to be built upon in response to user needs

• Opened in October 2007

• Seminars and outreach activity underway

Sports Medicine VR Labassisting a partnership between Kinesiology, Athletics and HMC

Lasch Football Building

• Special purpose lab supports study of perception action disruptions in posture and balance related to mild traumatic brain injury (e.g. concussions), elderly populations, etc.

• Motion in VR display is synchronized with measurement from EEG, postural tracking and force plate instrumentation

• Enhanced two-screen lab (wall and floor) in development for Recreation Building for broader use by faculty in kinesiology, psychology, engineering science and mechanics, SSRI and Hershey Medical Center

Collaborative Tools in Research

• Telecollaboration

– Adobe Connect (site license)

– Vyew

– Access Grid (many-to-many)

• Document / Source Sharing

– Subversion

– Wiki• MediaWiki (public domain)

• Confluence Wiki (commercial)

• Web-Based Science Gateways

– Materials Simulation Center Gateway (with Materials Research Institute)

ACCESS Grid Teleconferencing FacilityRoom 140 Computer Building

• Allows faculty to participate in Access Grid events with international academic community

• Scalable, multi-group telecollaboration (voice, video, application sharing) using multicast internet connections.

• Small number of highly satisfied users• Needs greater awareness and adoption

Ongoing research collaboration: Dean Snow (ANTHY), Craig Cameron (BMB) Research reporting: Richard Alley (GEOSCI), Donald Bryant (BMB), Mark Gahegan (GEOG) Virtual conferences: Genomics and Bioinformatics, SC Global, SC Desktop

Industry Outreach in partnership with

Institute for Computational Science (ICS) and Industrial Research Office (IRO)

" ... to out-compete, we must out-compute .... "

• Putting Pennsylvania based companies at a competitive advantage byhelping meet their large-scale computing needs.

Many small-to-medium sized companies, and even larger ones, do not have enough in-house expertise and resources for large-scale computations and as a result have not been able to use simulation and analysis tools with far greater frequency to help them innovate faster and become more competitive.

• Helping develop stronger relationship between faculty and industry byproviding computational services; may lead to more alignment betweenfaculty research areas and industry needs and positive impact on economic

development along the I-99 corridor and throughout the commonwealth.