Executing Message-Passing Programs

Mitesh Meswani

Presentation Outline

Introduction to Top Gun (eServer pSeries 690)

MPI on Top Gun (AIX/Linux) Itanium2 (Linux) Cluster Sun (Solaris) Workstation Cluster

Environment Set-up Compilation Execution

Contacts and Web Sites

Introduction to Top Gun

System Summary

Architecture IBM POWER4

OS AIX/LINUX

CPUs 16

Peak GFLOPS 83.2

Memory 32GB

Disk Storage 2TB

Top Gun is an IBM eServer pSeries 690 (p690) multiprocessor

Some features of the p690*

Logical Partitioning Processor – IBM POWER4 dual-core, 1GHz

processor with a fast shared L2 cache Self Management – Autonomic computing

tools for self correction and detection of errors

AIX Operating System – supports large files, system partitioning and large main memory

*Source : http://www-1.ibm.com/servers/eserver/pseries/hardware/tour/690_text.html

POWER4 Processor Layout*

Memory

I/O Bus

Processor local busL3 Cache

>1GHz CPU

L3 Controller L3 Directory

Shared L2 Cache

FABRIC CONTROLLERDISTRIBUTED SWITCH

DD

II>1GHz CPU

Description:•Two 64-bit processors per chip•64KB Icache and 32KB Dcache per processor• 1.44MB L2 unified cache with a 125GB/s transfer rate per chip

Some POWER4 Features:• Supports speculative execution• Supports out-of-order execution• Each processor is superscalar with 8 functional units: two floating-point units, two integer units, two load/store units, one branch prediction unit and one logical unit

*Source : http://www-1.ibm.com/servers/eserver/pseries/hardware/whitepapers/power4.html

Multi-Chip Module Layout

• Each MCM has• four POWER4 chips, a total of 8 processors.• a shared 128MB L3 unified cache with max transfer rate of

13.8GB/s• Max of four MCMs per p690 server/cabinet • Multiple p690 servers can be connected to form a cluster

PP

L2

PP

L2

PP

L2

PP

L2

L3

32-Processor p690 Configuration

P P

PP

PP

P

L2

L2

L2

L2

L3

MCM 0

P

P P

PP

PP

P

L2

L2

L2

L2

L3

MCM 2

P

P P

PP

PP

P

L2

L2

L2

L2

L3

MCM 1

P

P P

PP

PP

P

L2

L2

L2

L2

L3

MCM 3

P

Projects using Top Gun - 1

Dr. Stephen Aley, Department of Biological Sciences: programs for the analysis and data mining of DNA sequences, analysis of micro array data, and proteomic analysis by LC/MS-MS.

Robert Jeffrey Bruntz, Department of Physics: Analyzing the output of magneto hydrodynamics (MHD) simulations via visualization enabled by OpenDX.

Dr. Michael Huerta, Department of Mechanical and Industrial Engineering: Modeling shaped charges and other shock physics problems via hydro code calculations.

Projects using Top Gun - 2 Dr. Ramon Ravelo, Department of Physics:

Performing multi-million atom simulations of material response employing parallel molecular dynamics algorithms.

Dr. Ming-Ying Leung, Department of Mathematical Sciences : Developing efficient algorithms for pseudo knot prediction in long RNA sequences.

Dr. Patricia Teller, Diana Villa, and Research Assistants, Department of Computer Science: Developing performance-counter based metrics and techniques to tune application performance, mathematical models to steer HPC applications, memory access models of large complex application to explore sources of performance degradation, and techniques to dynamically adapt the Linux operating system.

Software on Top Gun

System Software: OS - AIX 5.3 Compilers: XL, Visual Age, GCC Parallel Programming Libraries: MPI, OpenMP, SHMEM Parallel Run Time Environment: Parallel Operating

Environment, POE Other Software

OpenGL 5.2 Loadleveler for AIX 2.0 PAPI XL Fortran Compiler Perl 5 Java 1.4.1

Complete list available at: http://research.utep.edu/Default.aspx?tabid=20686

MPI on Top Gun

Sample hello world MPI program:

#include <stdio.h>#include <mpi.h>

main(argc, argv)int argc;char *argv[];{ char name[BUFSIZ]; int length;

MPI_Init(&argc, &argv); MPI_Get_processor_name(name, &length); printf("%s: hello world\n", name); MPI_Finalize();}

MPI on Top GunEnvironment Set-up1. Create a file .rhosts and type topgun.utep.edu and

save the file in your home directory.

2. Create a host file .hf containing eight lines with the string topgun.utep.edu. Save this file in your home directory. Then set the environment variable MP_HOSTFILE to the absolute path of your .hf file.

     Example:       %setenv MP_HOSTFILE /home/mitesh/.hf

3. Define an environment variable called MP_PROCS to have a default value for number of processors.

Example:%setenv MP_PROCS 4

MPI on Top GunProgram Compilation Use mpcc Example: %mpcc HelloWorldMPI.c -o hello This creates an executable called hello. Other MPI compilers:

Fortran77: mpxlf C++: mpCC

Complete list available at: http://research.utep.edu/Default.aspx?tabid=20687

MPI on Top GunProgram Execution - 1 Use poe utility to execute parallel programs.

The poe command invokes the Parallel Operating Environment (POE) for loading and executing programs on remote nodes.

The flags associated with poe: -hostfile: a file with a list of hosts can be

specified – This will override the .hf file. -procs: number of MPI tasks to create – This

flag will ignore the MP_PROCS environment variable.

MPI on Top GunProgram Execution - 2 Example with two tasks:

%poe hello -procs 2%topgun.utep.edu: hello world%topgun.utep.edu: hello world

Example with a user specified host file:%poe hello -hostfile ./host.txt -procs 3%topgun.utep.edu: hello world%topgun.utep.edu: hello world%topgun.utep.edu: hello world

Link for information on IBM’s POE ( ParallelOperating Environment): http://www.llnl.gov/computing/tutorials/ibm_sp/#POE

MPI on Itanium2 Cluster

Vampyre Cluster: 8-processor Intel Itanium2 Cluster, with dual (SMP) 900MHz Itanium2 processors per node

Network Features: Externally accessible by 100 Mbps Ethernet; internal network runs at 1Gbps

OS: Linux kernel 2.4.18-e.25 MPICH

MPI on Itanium2 ClusterEnvironment Set-up In your home directory create an .rhosts file

that contains the following four lines: it01.vampyre.cs.utep.edu <account-name> it04.vampyre.cs.utep.edu <account-name> it03.vampyre.cs.utep.edu <account-name> it02.vampyre.cs.utep.edu <account-name>*

• it02 is down and cannot be used for MPI at the moment.• Additionally only two of the three remaining nodes are active

because of routing problems caused by it02 being down.

MPI on Itanium2 ClusterProgram Compilation Use mpicc <progname> <prog related

flags> -o <object filename> to compile program

Example:

%mpicc HelloWorldMPI.c -o hello

MPI on Itanium2 ClusterProgram Execution - 1 Use mpirun <options> to execute your program Some mpirun options:

-machinefile <machine-file name> : Create the list of possible machines on which to execute from the file <machine-file name>

-np <np>: specify the number of processors on which to execute

-nolocal: avoids executing MPI tasks on local host Example:

%mpirun -np 2 ./hello%sabina.vampyre.cs.utep.edu: hello world%clarimonde.cs.utep.edu: hello world

MPI on Itanium2 ClusterProgram Execution - 2 Example: Create one task while logged into

host sabina and use host clarimonde to execute the MPI task.

%mpirun –nolocal –np 1 ./hello%clarimonde.cs.utep.edu: hello world

MPI on Sun Workstation ClusterEnvironment Set-up Create an .rhosts file with hostname

<username> Example:

station10 mitesh

station11 mitesh

.

.

station20 mitesh

Copy the latest .cshrc file for mpi %cp /usr/local/cshrc/.cshrc.mpi .cshrc

MPI on Sun Workstation ClusterProgram Compilation Use mpicc <progname> <prog related

flags> -o <object filename> to compile program

Example:

%mpicc HelloWorldMPI.c -o hello

MPI on Sun Workstation ClusterProgram Execution - 1 Use mpirun <options> to execute a program Some mpirun options:

-machinefile <machine-file name>: Create the list of possible machines on which to execute from the file <machine-file name>. The format of machinefile list is nodename

Example: station10station11

-np <np>: specify the number of processors to run on -nolocal: avoids executing MPI tasks on local host

Example: %mpirun -np 3 ./hello%station11.: hello world%station13.: hello world%station12.: hello world

MPI on Sun Workstation ClusterProgram Execution - 2

Example: Create two tasks while logged into station11 and use only station12 and station13 to execute the MPI tasks.

%mpirun –nolocal –np 2 ./hello

%station12.: hello world

%station13.: hello world Refer to this MPICH Web Site for complete list of mpirun options:

http://www-unix.mcs.anl.gov/mpi/www/www1/mpirun.html

Contacts and Websites

System Administrators: Jose Hernandez (jose@cs.utep.edu) for Top Gun and

Sun Workstation Cluster Leopoldo Hernandez (leo@cs.utep.edu) for Itanium2

Cluster System Web Sites:

Top Gun: http://research.utep.edu/topgun Itanium2 Cluster:

http://www.cs.utep.edu/~bdauriol/courses/ParallelAndConcurrentProgramming/vampyre.html

MPI Links: http://www.llnl.gov/computing/tutorials/workshops/work

shop/mpi/MAIN.html#References http://www-unix.mcs.anl.gov/mpi/

Questions?