high performance computingmorrisriedel.de/wp-content/uploads/2018/03/hpc-lecture-0-hpc-prol… ·...

ADVANCED SCIENTIFIC COMPUTING

Dr. – Ing. Morris RiedelAdjunct Associated ProfessorSchool of Engineering and Natural Sciences, University of IcelandResearch Group Leader, Juelich Supercomputing Centre, Germany

PrologueAugust 22, 2017Room TG-227

High Performance Computing

LECTURE 0

Outline of the Course

1. High Performance Computing

2. Parallelization Fundamentals

3. Parallel Programming with MPI

4. Advanced MPI Techniques

5. Parallel Algorithms & Data Structures

6. Parallel Programming with OpenMP

7. Hybrid Programming & Patterns

8. Debugging & Profiling Techniques

9. Performance Optimization & Tools

10. Scalable HPC Infrastructures & GPUs

Lecture 0 - Prologue

11. Scientific Visualization & Steering

12. Terrestrial Systems & Climate

13. Systems Biology & Bioinformatics

14. Molecular Systems & Libraries

15. Computational Fluid Dynamics

16. Finite Elements Method

17. Machine Learning & Data Mining

18. Epilogue

+ additional practical lectures for our

hands-on exercises in context

2 / 37

Course Information

Lecture 0 - Prologue 3 / 37

Positioning in the Field of High Performance Computing

Consists of techniques for programming large-scale HPC Systems Approach: Get a broad understanding what HPC is and what can be done Goal: Train general HPC techniques and bits of domain-specific applications


High Performance Computing(a field of constant changes)

HPC Course

Domain-specific Science &

Engineering A


Engineering B


Engineering C


Engineering D

4 / 37

Course Motivation

Parallel processing and distributed computing have matured (over the past three decades) Both Emerged as a well developed field in computer science

‘Scientific computing‘ with Maple, Matlab, etc. Performed on small computing machines like Desktop PCs or Laptops Increasing number of cores enables ‘better scientific computing‘ today Good for small & less complex applications, quickly reach memory limits

‘Advanced scientific computing‘with computational simulations Performed on large parallel computers;

often scientific domain-specific approaches Use orders of magnitude more cores & memory Enables ‘simulations of reality‘ often based on

known physical laws and numerical methodsLecture 0 - Prologue 5 / 37

Selected Learning Outcomes

Students understand… Latest developments in parallel processing

and high performance computing (HPC) How to create high-performance clusters What are scalable networks & workloads The importance of domain decomposition Complex aspects of parallel programming HPC environment tools that support programming or analyze behaviour Different abstractions of parallel computing on various levels Foundations and approaches of scientific domain specific applications

Students are able to … Programm and use HPC programming paradigms (i.e. MPI / OpenMP) Take advantage of innovative scientific computing simulations Work with technologies and tools to handle parallelism complexity


Lecturer Morris Riedel (since ~2004 in HPC)

Holds PhD in Computer Science (from Karlsruhe Institute of Tech.) MSc in data visualization and steering of HPC & Grid applications

Over the time several Positions at Juelich Supercomputing Centre OS, Grid divisions; later deputy division leader federated systems and data Currently: Research Group Leader – High Productivity Data Processing

Selected other recent activities Working with CERN & LHC & Grid/Cloud

(Strategic Director of EU Middleware) Architect of Extreme Science and

Engineering Discovery Environment XSEDE (US HPC Infrastructure)

Co-Design of European Data Infrastructure (EUDAT) services, Research Data Alliance Big Data (Analytics) Chair, DEEP-EST HPC Machine Developments

University of IcelandCourses: HPC A / B, Statistical Data Mining, Cloud Computing & Big Data


[1] Morris Riedel Web page

7 / 37

Juelich Supercomputing Centre (JSC)

Part of Research Centre ‘Forschungszentrum Jülich’ in Germany Founded in 1956 as nuclear research facility, staff ~4500 90% Federal Republic of Germany; 10% State of North Rhine-Westphalia Research Areas: Health, Energy & Environment, Information, Key

Competency (e.g. HPC and computational science, big data processing) JSC is part of the Insitute for Advanced Simulation (IAS)

Operate supercomputers ( TOP500), cloud systems, and large storages Enable scientists to solve complex problems via simulations and analytics


[2] Juelich Supercomputing Centre Web page8 / 37

Juelich Supercomputing Centre – Major HPC Resources

JUQUEEN – Juelich BlueGene/Q 28 racks (7 rows à 4 racks); 28.672 nodes (458.752 cores) Nodeboard: 32 compute nodes (1 node 16 cores); main memory: 448 TB Overall peak performance: 5.9 Petaflop/s (Linpack: 5.0 Petaflop/s)

JURECA – Intel XEON Cluster with NVIDEA GPUs 75 nodes equipped with two NVIDIA K80 GPUs 1872 compute nodes; 45216 CPU cores; 79 TB main memory 1.8 (CPU) + 0.44 (GPU) Petaflop per second peak performance


Juelich Supercomputing Centre – HPC & ‘Big Data‘ Ecosystem


Lots of practice experience goes into the course to tacklereal world problems

10 / 37

Lecture 1 – High Performance Computing

What means ‘high performance‘? Four basic building blocks of HPC TOP500 and Performance Benchmarks Relationship to ‘Parallelization‘

HPC Architectures Shared Memory & Distributed Memory Architectures Hybrid and Emerging Architectures Parallel Applications and Infrastructures

HPC Ecosystem Software Environments & Scheduling System Architectures & Data Access Multicore Processor Design Network Topologies Interesting international HPC Projects


Lecture 2 – Parallelization Fundamentals

Parallel Applications Simple first parallel application examples Gradually more complex applications

Common Strategies for Parallelization Moore’s law Parallelization reasons and approaches Various domain decompositions Data parallelism methods Functional parallelism methods

Common goals & problems Speedup Higher throughput Better granularity Load imbalance


Lecture 3 – Parallel Programming with MPI

Message Passing Interface (MPI) Concepts Distributed memory systems Message passing functions Understanding the functionality of MPI collectives Standardization & portability Using MPI rank and communicators

MPI Parallel Programming Basics Environment with libraries & modules Thinking Parallel Basic building blocks of a program Compilations of codes Parallel executions ‘Bad‘ code examples vs.

good code examplesLecture 0 - Prologue

HPC Machine

13 / 37

Lecture 4 – Advanced MPI Techniques

MPI Communication Techniques MPI Communicators Cartesian Communicator Hardware & Communication Issues Network Interconnects Task-Core Mappings Application examples

MPI Parallel I/O Techniques I/O Terminologies & Challenges Parallel Filesystems MPI I/O Techniques Higher-Level I/O Libraries Portable File Formats Application examples


Lecture 5 – Parallel Algorithms & Data Structures


Selected Parallel Algorithms Vector Addition in MPI & OpenMP Matrix – Vector Multiplication in MPI Fast Fourier Transform with MPI Advanced Algorithm Examples Use of MPI collectives in applications

Selected Data Structures Basic MPI Datatypes Arrays &

Multi-dimensional datasets Derived MPI Datatypes Relationships to

Parallel IO & Filesystems

15 / 37

Lecture 6 – Parallel Programming with OpenMP


Shared-Memory Programming Concepts Parallel and Serial Regions Fork & Joins Master and Worker Threads Portability Application Examples Differences to distributed memory

OpenMP Parallel Programming Basics Basic building blocks Local/shared variables & Loops Synchronization & Critical Regions Selected Comparisons with MPI Simple Applications

int main()

{

#pragma omp parallel

printf(„Hello World“);

}

int main()

{

#pragma omp parallel

printf(„Hello World“);

}

mT T0

T1

T2

T3

masterthreadbecomes T0

16 / 37

Lecture 7 – Hybrid Programming & Patterns

Hybrid Programming Motivation and Memory Benefits Programming Hybrid Systems Vector Mode and Task Mode Lessons Learned & Performance Another type of Hybrid Programming Application Examples

Programming Patterns Neareast Neighbour Communication Cartesian Communicator Shifts Stencil-based Iterative Methods Jacobi 2D Application Example Working with Halo Regions Application Examples


Lecture 8 – Debugging & Profiling Techniques

Debugging Techniques Origin and Terminologies Bug Prevention Approaches Review Printf Debugging Advanced Debugging Techniques Selected Debugging Tools

Profiling Techniques Terminologies Understanding Wall-clock time Simple MPI Timing Approaches MPI Profiling Interface Selected Profiling Tools


P1 P2 P3 P4 P5

Shar

ed M

emor

y

T1 T2 T3 T4 T5

x=1 or x=2x =?

x=1x=2

x =?

18 / 37

Lecture 9 – Performance Optimization & Tools

Parallelization Terms & Theory Speedup & Load Imbalance Role of Serial Elements Scalability Metrics & Performance Amdahl’s Law & Performance Analysis

Performance Optimization Terminologies & Metrics Tracing Technique Open Tracing Format Selected Tools & Technologies

Selected Optimization Methods Simple Loops Constructs Improve MPI Function Calls MPI & OpenMP Problem Patterns


Lecture 10 – Scalable HPC Infrastructures & GPUs

Scalable Infrastructures Large Scale HPC Infrastructures e-Science and Grid Computing Cloud Computing Infrastructures Collaborative Data Infrastructures Scientific Workflows Applications

GPGPUs Terminology & Architecture Programming Models Usage Models & Applications Nvidia & CUDA Examples OpenCL Approach Applications


Large-scale HPC centres(e.g. EU HPC centres)

National HPC centres(e.g. German HPC centres)

Topical and Regional HPCCentres (e.g. climate centre)

Servers and small clusters(e.g. universities, institutes)

20 / 37

Lecture 11 – Scientific Visualization & Steering

Scientific Visualization Motivation & Objectives Understanding HPC Simulation Data Selected Visualization Techniques Selected Tools and Technologies Multi-scale Visualization Example Application Examples

Computational Steering Motivation & Terminology Different Methods Iterative Application Examples Selected Tools and Technologies Advanced Interactive Visualization Application Examples


(‘a picture is worth 1000 words numbers‘)

21 / 37

Lecture 12 – Terrestrial Systems & Climate

Terrestrial Systems Numerical Terrestrial Simulations & Models ParFlow Hydrology Parallel Application CLM Land-Surface Model Parallel Application COSMO Weather Model Parallel Application Coupled Models & Other Models & Libraries Application Examples

Climate Numerical Weather Prediction & Forecast Role of Partial Differential Equations (PDEs) WRF Model Parallel Application SAR Weather Project & Business Case Other Models & Libraries Application Examples


Lecture 13 – Systems Biology & Bioinformatics

Systems Biology Motivation & Basic Terminology Selected Scientific Case Protein Folding Role of Monte Carlo Methods SMMP & Neuroscience Parallel Applications Other Models & Libraries Application Examples

Bioinformatics Motivation & Basic Terminology Selected Scientific Case Gene Sequencing Role of Databases and Web-based Portals BLAST Parallel Application Other Tools & Techniques Application Examples


Lecture 14 – Molecular Systems & Libraries

Molecular Systems Terminology & Motivation Ab Initio Calculations Molecular Docking & Dynamics Application Examples

Selected Methods & Libraries NAMD CPMD MP2C AMBER Parallel Interoperability

Application


(force field)

24 / 37

Lecture 15 – Computational Fluid Dynamics

Computational Fluid Dynamics Terminology & Motivation Naviar-Stokes Method Lattice-Boltzmann Method Large Eddy Turbulence Model Modelling Methodology Revisited Application Examples

Selected Libraries and Methods Fire Dynamics Simulator OpenFOAM HemeLB iFluids Application Examples


(classical mechanics solutions are rather trajectories

of positions of a certain particle, here fluid velocity is in focus)

25 / 37

Lecture 16 – Finite Elements Method

Finite Elements Methods Terminology & Motivation Boundary Value Problems Mesh Generation Technique Adaptive Mesh Refinement Application Examples

Selected Libraries and Methods Elmer Code_Aster Salome-Meca Platform Ansys OpenFOAM Revisited Application Examples


Lecture 17 – Machine Learning & Data Mining

Machine Learning & Data Mining Terminology & Motivation Contrast to High Throughput Computing HPC for Classification,

Clustering & Regression Selected Remote Sensing Use Case Parallel Support Vector Machines HBDBSCAN for Clustering Inverse Problems Relationship to

Parallel I/O Application

Examples


[Video] High Performance Computing by Dreamworks


[3] YouTube, Dreamworks

28 / 37

Course Organization


UGLA Tool & Office Hours (!)

Reference course information High Performance Computing Forritun Ofurtölva (A) REI105M, spring 2017

Use it for course communication Every course member requires account Contact other students & discuss topics Recorded lectures & contact lecturer

Find course materials Slides & Handouts Further reading topics

(e.g. papers, practical lectures)

Questions, major difficulties, etc.? Don’t wait long! Use my office hours, send meeting request email to [email protected]


[4] High Performance ComputingUGLA Course Page Online

30 / 37

Overall Course Organization

3 Assignments (40 %) Guided by practical lectures in context with hands-on elements for all HPC configuration & parallel programming projects Influence in the overall grade TBD(all): Create Groups of 2-3 and send the group to [email protected]

Quizzes (10 %) Small quiz from time to time (pre-announced) to check understanding Jointly minor influence in the overall grade – good preparation for exam

Exam (50 %) End of the lecture series (~December) – major part of the overall grade ‘Not knowing everything is key – but understand the important elements‘

Invited Lectures A couple of presentations (e.g. companies, interesting projects, etc.)


Associated Literature

Further bibliography and readings will be provided in context


Introduction to High Performance Computing for Scientists and Engineers,Georg Hager & Gerhard Wellein,Chapman & Hall/CRC Computational Science,ISBN 143981192X, English, ~330 pages, 2010

[5] Introduction to High Performance Computing, 2010

32 / 37

Jötunn Cluster – HI HPC Service

HPC Service @ University of Iceland Course usage: Small to medium sized computation Every member of the course requires an account

Cluster Setup Nodes: 4 cpu: 2x Intel Xeon

CPU E5-2690 v3 @ 2.60GHz (2.6 GHz, 12 core) Memory: 128GB DDR4 Interconnect: 10 Gb/s ethernet

Software Setup Scheduling: SLURM Monitoring: Ganglia Libraries: MPI, OpenMP


Every course member receives an account from the HI HPC Service

Service can be asked when serious problems with the scheduling may be experienced:[email protected][6] Jotunn Cluster Online

33 / 37

[Video] PRACE – What is High Performance Computing


[7] PRACE, High Performance Computing

34 / 37

Lecture Bibliography


Lecture Bibliography

[1] Morris Riedel Web page, Online: http://www.morrisriedel.de

[2] Juelich Supercomputing Centre Web page, Online: http://www.fz-juelich.de/ias/jsc/EN/Home/home_node.html

[3] YouTube Video, Dreamworks, High Performance Computing, Online: http://www.youtube.com/watch?v=TGSRvV9u32M

[4] High Performance Computing UGLA Course Page, Online: https://ugla.hi.is/kv/index2.php?sid=219&namsknr=70067120176&kennsluvefur_efnisatridi=1

[5] Introduction to High Performance Computing for Scientists and Engineers, Georg Hager & Gerhard Wellein, Chapman & Hall/CRC Computational Science, ISBN 143981192X, English, ~330 pages, 2010, Online:http://www.amazon.de/Introduction-Performance-Computing-Scientists-Computational/dp/143981192X

[6] Jötunn HPC Cluster, Online: http://ihpc.is/jotunn/ [7] PRACE – Dare to Think the Impossible, YouTube Video,

Online: http://www.youtube.com/watch?v=fgy-ZkJyom0


high performance computingmorrisriedel.de/wp-content/uploads/2018/03/hpc-lecture-0-hpc-prol… ·...

Documents