MPI and High Performance Computing:Systems and Programming

Barry Britt, Systems Administrator Department of Computer Science

Iowa State University

Purpose

To give you: … an overview of some new system-level MPI

functions … access to tools that you need to compile and

run MPI jobs … some instruction in the creation and use of

Makefiles … some instruction on how to tell time in C

programs.

Makefiles

Makefiles

GNU Make Enables the end user to build and install a

package without worrying about the details. Automatically figures out which files it needs

to update based on which source files have changed.

Not language dependent Not limited to building a package; can be used

to install or uninstall

Makefile Rules

A rule tells Make how to execute a series of commands in order to build a target from source files.

Specifies a list of dependancies Dependancies should include ALL files that are

dependancies for a target

target:dependancies ....

commands...

Example Makefile for C Source

CC=gccCFLAGS=-WallINCLUDES=BINARIES=rand test

.SUFFIXES: .c .o

.c.o: $(CC) $(CFLAGS) -c $*.call: $(BINARIES)rand.o: rand.ctest.o: test.crand: rand.o $(CC) $(CFLAGS) -o rand rand.otest: test.o $(CC) $(CFLAGS) -o test test.oclean: rm -f a.out core *.o $(BINARIES)

Example Makefile for C SourceCC=gccCFLAGS=-WallINCLUDES=BINARIES=rand test

Variables CC is set to use the GCC compiler

For MPI programs, set it to mpicc, not gcc CFLAGS:

-c: compile -Wall: set warnings to all

Example Makefile for C Source

Target “clean”. Use by typing make clean

Rule states: In my current directory, run:

rm -f a.out core *.o $(BINARIES) rm -f a.out core *.o rand test

clean:rm -f a.out core *.o $(BINARIES)

Example Makefile for C Source

Makefile instruction on how to handle .c files and turn them into object (.o) files

Compile using $(CC) value with $(CFLAGS) Compile each individual file into its

appropriate .o file

.SUFFIXES: .c .o

.c.o: $(CC) $(CFLAGS) -c $*.c

Example Makefile for C Source

Target: rand or test Run $(CC) $(CFLAGS) -o rand rand.o gcc -Wall -o rand rand.o

If you were going to include external libraries to link, they would be linked at the end of the rule.

rand.o:rand.ctest.o: test.crand: rand.o

$(CC) $(CFLAGS) -o rand rand.otest: test.o

$(CC) $(CFLAGS) -o test test.o

Random Matrix Generation

Random Generator for Matrices

Rand -f: filename to which to write the matrix -c: number of matrix columns -r: number of matrix rows -h: help documentation -s: seed -m: max integer in matrix cells

Random Generator for Matrices

Completely random generation for an m by n matrix

Uses a random seed to create the matrix Output file

First line contains the number of rows and the number of columns

Subsequent lines contain matrix cell values, one per line.

Random Generator for Matrices

For a Matrix with row length m, cell A[i,j] is on line:

m * i + j + 2 Lines are not zero-indexed for the purpose of

this calculation. Therefore, for a 5 x 5 matrix (zero-indexed):

A[0, 0] is on line 2 A[0, 1] is on line 3 A[4, 4] is on line 26 A[2, 3] is on line 15

Calculating Run Time in C

Calculating Running Time in C

#include <stdio.h>#include <sys/time.h>

int main() { struct timeval begin, end; double time;

gettimeofday(&begin, NULL); sleep(10); gettimeofday(&end, NULL);

time = (end.tv_sec - begin.tv_sec) +((end.tv_usec - begin.tv_usec) / 1000000.0); printf("This program ran for %f seconds\n", time);

return 0;}

C Time

Includes seconds and microseconds Used by the gettimeofday() system call gettimeofday()

Returns the number of seconds (and microseconds) since the UNIX Epoch

Is this completely accurate? No, but it's VERY close (within a few

microseconds).

C Time

You MUST use the timeval struct for the gettimeofday() call

On UNIX systems, you need to include sys/time.h to use this.

Calculation of time is:(end seconds – begin seconds) +

((end microseconds – begin microseconds) / 1000000)

You can calculate: Program run time Algorithm execution time

Using the PBS Job Submission System

PBS (Torque/Maui)

hpc-class job submission system qsub All queues are managed by the scheduler. PBS scripts can be created at:

http://hpcgroup.public.iastate.edu/HPC/hpc-class/hpc-class_script_writer.html

Example script

#!/bin/csh

#PBS -o BATCH_OUTPUT #PBS -e BATCH_ERRORS

#PBS -lvmem=256Mb,pmem=256Mb,mem=256Mb,nodes=16:ppn=2,cput=2:00:00,walltime=1:00:00

# Change to directory from which qsub was executed cd $PBS_O_WORKDIR

time mpirun -np 32 <program>

PBS Variables

-l (resources) vmem: total virtual memory pmem: per task memory mem: total aggregate memory nodes – total number of nodes ppn – processors per node cput – CPU time walltime – total time for all CPUs

PBS Variables

vmem = pmem = mem total CPUs = nodes * ppn cput = walltime * ppn

PBS (Torque/Maui)

Based on the previous script BATCH_OUTPUT contains the output from the

batch job BATCH_ERRORS contains the error

information from the batch job

Some other important information

Max CPU – 32 for classwork Max memory – 2.0 GB Max swap – 2.0 GB Short queue -

4 nodes per job; 16 total CPUs 1 hour per job 2 total jobs per user

MPI Blocking vs. Non-Blocking Communication

MPI Communication

Blocking Communication: MPI_Send MPI_Recv

MPI_Send → Basic blocking send operation. Routine returns only after the application buffer in the sending task is free for reuse.

MPI_Recv → Receive a message and block until the requested data is available in the application buffer in the receiving task.

MPI Communication

Non-blocking Communication

MPI_Isend | MPI_Irecv MPI_Wait | MPI_Test

MPI_Isend → Identifies an area in memory to serve as a send buffer. Processing continues without waiting for the message to be copied out from the buffer.

MPI_Irecv → Identifies an area in memory to serve as a receive buffer. Processing continues immediately without waiting for the message to be received and copied into the the buffer.

MPI_Test → check the status of a non-blocking send or receive

MPI_Wait → block until a specified non-blocking send or receive operation has completed

Why non-blocking communication?

In some cases, it can increase performance. If there is an expensive operation you need to

do, it helps speed up the program Disk I/O Heavy processing on already received data

BE CAREFUL!!! If you try to access a buffer when it isn't there,

your program WILL fail.

int main (int argc, char **argv) { int myRank;

MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &myRank);

if (myRank == 0) master(); else slave();

MPI_Finalize(); return 0;}

int master() { int i, size, my_answer = 0, their_work = 0; MPI_Status status;

MPI_Comm_size(MPI_COMM_WORLD, &size);

for (i = 1; i < size; i++) { MPI_Recv ( &their_work, 1, MPI_INT, i, TAG, MPI_COMM_WORLD, &status); my_answer += their_work; } printf("The answer is: %d\n", my_answer);

return 0;}

int slave() { int i, myRank, size, namelength, work = 0; char name[MPI_MAX_PROCESSOR_NAME];

MPI_Comm_rank(MPI_COMM_WORLD, &myRank); MPI_Comm_size(MPI_COMM_WORLD, &size); MPI_Get_processor_name(name, &namelength);

printf("[%s]: Adding the nubmers %d to %d = ", name, (100 / (size-1)) * (myRank-1) + 1 , (100 / (size-1)) * myRank);

for (i = (100 / (size-1)) * (myRank-1) + 1; i <= myRank * (100 / (size-1)); i++) { work = work + i; } printf("%d\n", work);

MPI_Send(&work, 1, MPI_INT, 0, TAG, MPI_COMM_WORLD);

return 0;}