introduzione al message passing interface (mpi)enrico/ingegneriadel... · processes identification...

Introduzione al Message Passing

Interface (MPI)Interface (MPI)

Andrea Clematis

IMATI – CNR

[email protected]

Ack. & riferimenti

• An Introduction to MPI Parallel Programming with the

Message Passing InterfaceWilliam Gropp ,Ewing Lusk Argonne National Laboratory

• Online examples available at• Online examples available at

http://www.mcs.anl.gov/mpi/tutorials/perf

• ftp://ftp.mcs.anl.gov/mpi/mpiexmpl.tar.gz contains source code and

run scripts that allows you to evaluate your own MPI implementation

Sommario

�Comunicazione tra due processi

�Aspetti generali di MPI �Aspetti generali di MPI

�Specificazione della comunicazione in MPI

The Message-Passing Model

• A process is (traditionally) a program counter and address

space.

• Processes may have multiple threads (program counters

and associated stacks) sharing a single address space. MPI and associated stacks) sharing a single address space. MPI

is for communication among processes, which have

separate address spaces.

• Interprocess communication consists of

– Synchronization

– Movement of data from one process’s address space to

another’s.

Cooperative Operations for Communication

• The message-passing approach makes the exchange of data

cooperative.

• Data is explicitly sent by one process and received by

another.

• An advantage is that any change in the receiving process’s

memory is made with the receiver’s explicit participation.

• Communication and synchronization are combined.

Process 0 Process 1

Send(data)

Receive(data)

One-Sided Operations for Communication

• One-sided operations between processes include remote memory reads

and writes

• Only one process needs to explicitly participate.

• An advantage is that communication and synchronization are

decoupleddecoupled

• One-sided operations are part of MPI-2.

Process 0 Process 1

Put(data)

(memory)

(memory)

Get(data)

What is message passing?

• Data transfer plus synchronization

DataProcess 0 May I Send? DataData

DataData

• Requires cooperation of sender and receiver

• Cooperation not always apparent in code

Process 1 Yes

Data

Data

Data

Time

Sommario




What is MPI?

• A message-passing library specification

– extended message-passing model

– not a language or compiler specification

– not a specific implementation or product– not a specific implementation or product

• For parallel computers, clusters, and heterogeneous networks

• Full-featured

• Designed to provide access to advanced parallel hardware for

– end users

– library writers

– tool developers

MPI Sources

• The Standard itself:

– at http://www.mpi-forum.org

– All MPI official releases, in both postscript and HTML

• Books:

– Using MPI: Portable Parallel Programming with the Message-Passing Interface, by Gropp, Lusk, and Skjellum, MIT Press, 1994.

– MPI: The Complete Reference, by Snir, Otto, Huss-Lederman, Walker, and – MPI: The Complete Reference, by Snir, Otto, Huss-Lederman, Walker, and Dongarra, MIT Press, 1996.

– Designing and Building Parallel Programs, by Ian Foster, Addison-Wesley, 1995.

– Parallel Programming with MPI, by Peter Pacheco, Morgan-Kaufmann, 1997.

– MPI: The Complete Reference Vol 1 and 2,MIT Press, 1998(Fall).

• Other information on Web:

– at http://www.mcs.anl.gov/mpi

– pointers to lots of stuff, including other talks and tutorials, a FAQ, other MPI pages

Why Use MPI?

• MPI provides a powerful, efficient, and portable

way to express parallel programs

• MPI was explicitly designed to enable libraries…

• … which may eliminate the need for many users

to learn (much of) MPI

A Minimal MPI Program (C)

#include "mpi.h"

#include <stdio.h>

int main( int argc, char *argv[] )int main( int argc, char *argv[] )

{

MPI_Init( &argc, &argv );

printf( "Hello, world!\n" );

MPI_Finalize();

return 0;

}

A Minimal MPI Program (Fortran)

program main

use MPI

integer ierr

call MPI_INIT( ierr )

print *, 'Hello, world!'

call MPI_FINALIZE( ierr )

end

• #include "mpi.h" provides basic MPI

definitions and types

• MPI_Init starts MPI• MPI_Init starts MPI

• MPI_Finalize exits MPI

• Note that all non MPI routines are local

thus the printf run on each process the actual

behaviour depends on implementation

Notes on C and Fortran

• C and Fortran bindings correspond closely

• In C:

– mpi.h must be #included

– MPI functions return error codes or MPI_SUCCESS

• In Fortran:

– mpif.h must be included, or use MPI module (MPI-2)

– All MPI calls are to subroutines, with a place for the

return code in the last argument.

• C++ bindings, and Fortran-90 issues, are part of MPI-2.

Error Handling

• By default, an error causes all processes to abort.

• The user can cause routines to return (with an error code)

instead.

– In C++, exceptions are thrown (MPI-2)– In C++, exceptions are thrown (MPI-2)

• A user can also write and install custom error handlers.

• Libraries might want to handle errors differently from

applications.

Running MPI Programs

• The MPI-1 Standard does not specify how to run an MPI program, just

as the Fortran standard does not specify how to run a Fortran program.

• In general, starting an MPI program is dependent on the

implementation of MPI you are using, and might require various implementation of MPI you are using, and might require various

scripts, program arguments, and/or environment variables.

• mpiexec <args> is part of MPI-2, as a recommendation, but not

a requirement

Finding Out About the Environment

• Two important questions that arise early in a parallel program

are:

– How many processes are participating in this computation?

– Which one am I?

• MPI provides functions to answer these questions:

– MPI_Comm_size reports the number of processes.

– MPI_Comm_rank reports the rank, a number between 0

and size-1, identifying the calling process

Better Hello (C)

#include "mpi.h"

#include <stdio.h>

int main( int argc, char *argv[] )

{{

int rank, size;

MPI_Init( &argc, &argv );

MPI_Comm_rank( MPI_COMM_WORLD, &rank );

MPI_Comm_size( MPI_COMM_WORLD, &size );

printf( "I am %d of %d\n", rank, size );

MPI_Finalize();

return 0;

}

Sommario




MPI Basic Send/Receive

• We need to fill in the details in

Process 0 Process 1

Send(data)

Receive(data)

• Things that need specifying:

– How will “data” be described?

– How will processes be identified?

– How will the receiver recognize/screen messages?

– What will it mean for these operations to complete?

Receive(data)

Message passing before MPI

A typical send looks like:

send (dest, type, address, length);

where:

dest : is an integer identifier representing the process to

receive the message;

type: is a non negative integer that the destination may

use to screen the messages;

address, length: describe a contiguosu area in memory

containing the message to be sent

Message passing before MPI

A typical global operation looks like:

broadcast (type, address, length);

All of these specifications are a good match to hardware,

ease to understand but too unflexible

Message passing before MPI – the buffer

Sending and receiving only a contiguous array of bytes:

• hides the real data structure from hardware which might be

able to handle it directly;

•requires pre-packing dispersed data

•rows of a matrix stored columnwise;

•general collections of structures

•prevents communication between machines with different

representations (even lengths) for same data types

Message passing in MPI – generalizing the buffer

Specified in MPI by starting address, datatype and count, where datatype is:

• elementary (all C and Fortran datatypes)

•contiguous array of datatypes;

•strided blocks of datatypes;

•indexed array of blocks of datatypes;

•general structures•general structures

Datatypes are constructed recursively;

Specification of elementary datatypes allow heterogeneous communications;

Elimination of length in favour of count is clearer;

Specifying application oriented layout of data allows maximal use of special

hardware

Why Datatypes?

• Since all data is labeled by type, an MPI implementation can support

communication between processes on machines with very different

memory representations and lengths of elementary datatypes

(heterogeneous communication).

• Specifying application-oriented layout of data in memory

– reduces memory-to-memory copies in the implementation

– allows the use of special hardware (scatter/gather) when available

MPI Datatypes

• The data in a message to sent or received is described by a triple

(address, count, datatype), where

• An MPI datatype is recursively defined as:

– predefined, corresponding to a data type from the language (e.g.,

MPI_INT, MPI_DOUBLE_PRECISION)MPI_INT, MPI_DOUBLE_PRECISION)

– a contiguous array of MPI datatypes

– a strided block of datatypes

– an indexed array of blocks of datatypes

– an arbitrary structure of datatypes

• There are MPI functions to construct custom datatypes, such an array

of (int, float) pairs, or a row of a matrix stored columnwise.

The type of messages and MPI Tag

• A separate communication context for each family of messages used for queueing and matching

• This has often been simulated in the past by

overloading the tag fieldoverloading the tag field

• No wild cards allowed for security

• Allocated by the system for security

• Types tags in MPI retained for normal use wildcards OK

MPI Tags

• Messages are sent with an accompanying user-defined

integer tag, to assist the receiving process in identifying

the message.

• Messages can be screened at the receiving end by • Messages can be screened at the receiving end by

specifying a specific tag, or not screened by specifying

MPI_ANY_TAG as the tag in a receive.

• Some non-MPI message-passing systems have called tags

“message types”. MPI calls them tags to avoid confusion

with datatypes.

Processes Identification: the library problem

Sub1 and Sub2 are from different libraries

The code:

Sub1();

The example is due to Marc Snir

Sub2();

is executed in parallel by Procees0, Process1, and

Process2;

The library problem

Correct execution of Correct execution of communication

The library problem

Incorrect execution of Incorrect execution of communication

Processes Identification: the library problem

Sub1a and Sub1b are from the same library

The code:

Sub1a();

Sub2();

The example is due to Marc Snir

Sub2();

Sub1b();

is executed in parallel by Procees0, Process1, and

Process2 as shown in next slide;

The library problem

Correct execution of Correct execution of communication (one pending)

The library problem

Incorrect execution of Incorrect execution of communication (one pending)

Processes identification in MPI

• Processes can be collected into groups.

• Each message is sent in a context, and must be received in

the same context.

• A group and context together form a communicator.• A group and context together form a communicator.

• A process is identified by its rank in the group associated

with a communicator.

• There is a default communicator whose group contains all

initial processes, called MPI_COMM_WORLD.

Tags and Contexts

• Separation of messages used to be accomplished by use of tags, but

– this requires libraries to be aware of tags used by other libraries.

– this can be defeated by use of “wild card” tags.

• Contexts are different from tags

– no wild cards allowed

– allocated dynamically by the system when a library sets up a

communicator for its own use.

• User-defined tags still provided in MPI for user convenience in

organizing application

• Use MPI_Comm_split to create new communicators

MPI Basic (Blocking) Send

MPI_SEND (start, count, datatype, dest, tag, comm)

• The message buffer is described by (start, count, • The message buffer is described by (start, count, datatype).

• The target process is specified by dest, which is the rank of

the target process in the communicator specified by comm.

• When this function returns, the data has been delivered to the

system and the buffer can be reused. The message may not

have been received by the target process.

MPI Basic (Blocking) Receive

MPI_RECV(start, count, datatype, source, tag, comm, status)

• Waits until a matching (on source and tag) message is received• Waits until a matching (on source and tag) message is received

from the system, and the buffer can be used.

• source is rank in communicator specified by comm, or

MPI_ANY_SOURCE.

• status contains further information

• Receiving fewer than count occurrences of datatype is OK,

but receiving more is an error.

Retrieving Further Information

• Status is a data structure allocated in the user’s program.

• In C:

int recvd_tag, recvd_from, recvd_count;

MPI_Status status;

MPI_Recv(..., MPI_ANY_SOURCE, MPI_ANY_TAG, ..., &status )

recvd_tag = status.MPI_TAG;recvd_tag = status.MPI_TAG;

recvd_from = status.MPI_SOURCE;

MPI_Get_count( &status, datatype, &recvd_count );

• In Fortran:

integer recvd_tag, recvd_from, recvd_count

integer status(MPI_STATUS_SIZE)

call MPI_RECV(..., MPI_ANY_SOURCE, MPI_ANY_TAG, .. status, ierr)

tag_recvd = status(MPI_TAG)

recvd_from = status(MPI_SOURCE)

call MPI_GET_COUNT(status, datatype, recvd_count, ierr)

Simple Fortran Example - 1

program main

use MPI

integer rank, size, to, from, tag, count, i, ierr

integer src, dest

integer st_source, st_tag, st_count

integer status(MPI_STATUS_SIZE)

double precision data(10)

call MPI_INIT( ierr )

call MPI_COMM_RANK( MPI_COMM_WORLD, rank, ierr )

call MPI_COMM_SIZE( MPI_COMM_WORLD, size, ierr )

print *, 'Process ', rank, ' of ', size, ' is alive'

dest = size - 1

src = 0


if (rank .eq. 0) then

do 10, i=1, 10

data(i) = i

10 continue

call MPI_SEND( data, 10, MPI_DOUBLE_PRECISION,call MPI_SEND( data, 10, MPI_DOUBLE_PRECISION,

+ dest, 2001, MPI_COMM_WORLD, ierr)

else if (rank .eq. dest) then

tag = MPI_ANY_TAG

source = MPI_ANY_SOURCE

call MPI_RECV( data, 10, MPI_DOUBLE_PRECISION,

+ source, tag, MPI_COMM_WORLD,

+ status, ierr)


call MPI_GET_COUNT( status,MPI_DOUBLE_PRECISION,

st_count, ierr )

st_source = status( MPI_SOURCE )

st_tag = status( MPI_TAG )st_tag = status( MPI_TAG )

print *, 'status info: source = ', st_source,

+ ' tag = ', st_tag, 'count = ', st_count

endif

call MPI_FINALIZE( ierr )

end

MPI is Simple

• Many parallel programs can be written using just these six functions, only two of which are non-trivial:

– MPI_INIT

– MPI_FINALIZE– MPI_FINALIZE

– MPI_COMM_SIZE

– MPI_COMM_RANK

– MPI_SEND

– MPI_RECV

• Point-to-point (send/recv) isn’t the only way...

main(int argc, char **argv)

{

int me, size;

int SOME_TAG = 0;

...

MPI_Init(&argc, &argv);

MPI_Comm_rank(MPI_COMM_WORLD, &me); /* local */

MPI_Comm_size(MPI_COMM_WORLD, &size); /* local */

if ((me % 2) == 0)

{ /* send unless highest-numbered process */

if ((me + 1) < size)

MPI_Send(..., me + 1, SOME_TAG, MPI_COMM_WORLD);

}else

MPI_Recv(..., me - 1, SOME_TAG, MPI_COMM_WORLD);

...

MPI_Finalize();

}


{

int me, size;

int SOME_TAG = 0;

...




if ((me % 2) == 1) if ((me % 2) == 1)


MPI_Send(..., me - 1, SOME_TAG, MPI_COMM_WORLD);

}else


MPI_Recv(..., me + 1, SOME_TAG, MPI_COMM_WORLD);

...

MPI_Finalize();

}


{

int me, size;

int SOME_TAG = 0;

...




if ((me % 2) == 0)



MPI_Send(..., me + 1, SOME_TAG, MPI_COMM_WORLD); MPI_Send(..., me + 1, SOME_TAG, MPI_COMM_WORLD);

if (me != 0)

MPI_Recv(..., me - 1, SOME_TAG, MPI_COMM_WORLD);

}else

{ MPI_Recv(..., me - 1, SOME_TAG, MPI_COMM_WORLD);


MPI_Send(..., me + 1, SOME_TAG, MPI_COMM_WORLD);

}

...

MPI_Finalize();

}

Types of Point-to-Point Operations:

• There are different types of send and receive routines used for different purposes. For example: – Synchronous send

– Blocking send / blocking receive

– Non-blocking send / non-blocking receive – Non-blocking send / non-blocking receive

– Buffered send

– Combined send/receive

– "Ready" send

• Any type of send routine can be paired with any type of receive routine.

• MPI also provides several routines associated with send - receive operations, such as those used to wait for a message's arrival or probe to find out if a message has arrived.

Data Movement in point to point communication

Buffering

• In a perfect world, every send operation would be perfectly synchronized with its matching receive. This is rarely the case. Somehow or other, the MPI implementation must be able to deal with storing data when the two tasks are out of sync.

• Consider the following two cases:

• A send operation occurs 5 seconds before the receive is ready - where is the message while the receive is pending?

• Multiple sends arrive at the same receiving task which can only accept one send at a time - what • Multiple sends arrive at the same receiving task which can only accept one send at a time - what happens to the messages that are "backing up"?

• The MPI implementation (not the MPI standard) decides what happens to data in these types of cases. Typically, a system buffer area is reserved to hold data in transit. For example:

Buffering

• System buffer space is:

– Opaque to the programmer and managed entirely by the MPI

library

– A finite resource that can be easy to exhaust – A finite resource that can be easy to exhaust

– Often mysterious and not well documented

– Able to exist on the sending side, the receiving side, or both

– Something that may improve program performance because it

allows send - receive operations to be asynchronous.

• User managed address space (i.e. your program variables) is called the

application buffer. MPI also provides for a user managed send buffer.

Blocking vs. Non-blocking

• Most of the MPI point-to-point routines can be used in either blocking or non-blocking mode.

• Blocking: – A blocking send routine will only "return" after it is safe to modify the

application buffer (your send data) for reuse. Safe means that modifications will not affect the data intended for the receive task. Safe modifications will not affect the data intended for the receive task. Safe does not imply that the data was actually received - it may very well be sitting in a system buffer.

– A blocking send can be synchronous which means there is handshaking occurring with the receive task to confirm a safe send.

– A blocking send can be asynchronous if a system buffer is used to hold the data for eventual delivery to the receive.

– A blocking receive only "returns" after the data has arrived and is ready for use by the program.

Blocking vs. Non-blocking

• Non-blocking: – Non-blocking send and receive routines behave similarly - they will return

almost immediately. They do not wait for any communication events to complete, such as message copying from user memory to system buffer space or the actual arrival of message.

– Non-blocking operations simply "request" the MPI library to perform the – Non-blocking operations simply "request" the MPI library to perform the operation when it is able. The user can not predict when that will happen.

– It is unsafe to modify the application buffer (your variable space) until you know for a fact the requested non-blocking operation was actually performed by the library. There are "wait" routines used to do this.

– Non-blocking communications are primarily used to overlap computation with communication and exploit possible performance gains.

Order and Fairness

• Order: – MPI guarantees that messages will not overtake each other.

– If a sender sends two messages (Message 1 and Message 2) in succession to the same destination, and both match the same receive, the receive operation will receive Message 1 before Message 2.

– If a receiver posts two receives (Receive 1 and Receive 2), in succession, and both are looking for the same message, Receive 1 will receive the message before Receive 2.

– Order rules do not apply if there are multiple threads participating in the communication operations.

– Order rules do not apply if there are multiple threads participating in the communication operations.

• Fairness: – MPI does not guarantee fairness - it's up to the programmer to prevent "operation starvation".

– Example: task 0 sends a message to task 2. However, task 1 sends a competing message that matches task 2's receive. Only one of the sends will complete.

introduzione al message passing interface (mpi)enrico/ingegneriadel... · processes identification...

Documents