message-passing programming with mpi · concepts if/then/else languages java fortran struct python...
TRANSCRIPT
![Page 1: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/1.jpg)
Message-Passing
Programming with MPIMessage-Passing Concepts
![Page 2: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/2.jpg)
Reusing this material
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
http://creativecommons.org/licenses/by-nc-sa/4.0/
This means you are free to copy and redistribute the material and adapt and build on the material under the following terms: You must give appropriate credit, provide a link to the license and indicate if changes were made. If you adapt or build on the material you must
distribute your work under the same license as the original.
Acknowledge EPCC as follows: “© EPCC, The University of Edinburgh, www.epcc.ed.ac.uk”
Note that this presentation contains images owned by others. Please seek their permission before reusing these images.
3
![Page 3: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/3.jpg)
Overview
• This lecture will cover
- message passing model
- SPMD
- communication modes
- collective communications
4
![Page 4: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/4.jpg)
Programming Models
5
Control flow Variables
Arrays
Human-readable
Serial Programming
Concepts
if/then/else
Languages
Java Fortran
struct
PythonC/C++
Subroutines
Implementations
icc
pgcc -fast
crayftn
gcc -O3
OO
Processes
SPMD
Concepts
Libraries
Implementations
Intel MPI
Message-Passing Parallel Programming
Groups
Send/Receive
Collectives
javac
MPI
MPICH2
OpenMPI
Cray MPI
IBM MPIcraycc
MPI_Init()
![Page 5: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/5.jpg)
Message Passing Model
• The message passing model is based on the notion of
processes
- can think of a process as an instance of a running program,
together with the program’s data
• In the message passing model, parallelism is achieved by
having many processes co-operate on the same task
• Each process has access only to its own data
- ie all variables are private
• Processes communicate with each other by sending and
receiving messages
- typically library calls from a conventional sequential language
6
![Page 6: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/6.jpg)
M
P
Memory
Processor
Process
Sequential Paradigm
7
![Page 7: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/7.jpg)
Processes
Message Passing Interface
Communication Network
0 1 2 3
Parallel Paradigm
8
![Page 8: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/8.jpg)
P MP MP M
P M
P M P M
P M
P MInterconnect
Distributed-Memory Architectures
9
![Page 9: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/9.jpg)
a=23 Recv(1,b)
Process 1 Process 2
23
23
24
23
Program
Data
Send(2,a) a=b+1
Process Communication
10
![Page 10: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/10.jpg)
SPMD
• Most message passing programs use the Single-
Program-Multiple-Data (SPMD) model
• All processes run (their own copy of) the same program
• Each process has a separate copy of the data
• To make this useful, each process has a unique identifier
• Processes can follow different control paths through the
program, depending on their process ID
• Usually run one process per processor / core
11
![Page 11: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/11.jpg)
Emulating General Message Passing (C)
main (int argc, char **argv)
{
if (controller_process)
{
Controller( /* Arguments */ );
}
else
{
Worker ( /* Arguments */ );
}
}
12
![Page 12: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/12.jpg)
Emulating General Message Passing (F)
PROGRAM SPMD
IF (controller_process) THEN
CALL CONTROLLER ( ! Arguments ! )
ELSE
CALL WORKER ( ! Arguments ! )
ENDIF
END PROGRAM SPMD
13
![Page 13: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/13.jpg)
Messages
• A message transfers a number of data items of a certain
type from the memory of one process to the memory of
another process
• A message typically contains
- the ID of the sending processor
- the ID of the receiving processor
- the type of the data items
- the number of data items
- the data itself
- a message type identifier
14
![Page 14: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/14.jpg)
Communication modes
• Sending a message can either be synchronous or
asynchronous
• A synchronous send is not completed until the message
has started to be received
• An asynchronous send completes as soon as the
message has gone
• Receives are usually synchronous - the receiving process
must wait until the message arrives
15
![Page 15: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/15.jpg)
Synchronous send
• Analogy with faxing a letter.
• Know when letter has started to be received.
16
![Page 16: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/16.jpg)
Asynchronous send
• Analogy with posting a letter.
• Only know when letter has been posted, not when it has
been received.
17
![Page 17: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/17.jpg)
Point-to-Point Communications
• We have considered two processes
- one sender
- one receiver
• This is called point-to-point communication
- simplest form of message passing
- relies on matching send and receive
• Close analogy to sending personal emails
18
![Page 18: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/18.jpg)
Collective Communications
• A simple message communicates between two processes
• There are many instances where communication between
groups of processes is required
• Can be built from simple messages, but often
implemented separately, for efficiency
19
![Page 19: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/19.jpg)
Barrier
• Global synchronisation
20
Barrier
Barrier
Barrier
![Page 20: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/20.jpg)
Broadcast
• One to all communication
21
![Page 21: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/21.jpg)
Broadcast
• From one process to all others
22
8
8 8
8
8
8
![Page 22: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/22.jpg)
Scatter
• Information scattered to many processes
23
0 1 2 3 4 5
0
1
3
4
5
2
![Page 23: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/23.jpg)
Gather
• Information gathered onto one process
24
0 1 2 3 4 5
0
1
3
4
5
2
![Page 24: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/24.jpg)
Reduction Operations
• Combine data from several processes to form a single
result
25
Strike?
![Page 25: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/25.jpg)
Reduction
• Form a global sum, product, max, min, etc.
26
0
1
3
4
5
2
15
![Page 26: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/26.jpg)
Launching a Message-Passing Program
• Write a single piece of source code• with calls to message-passing functions such as send / receive
• Compile with a standard compiler and link to a message-passing library provided for you• both open-source and vendor-supplied libraries exist
• Run multiple copies of same executable on parallel machine• each copy is a separate process
• each has its own private data completely distinct from others
• each copy can be at a completely different line in the program
• Running is usually done via a launcher program• “please run N copies of my executable called program.exe”
27
![Page 27: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/27.jpg)
Issues
• Sends and receives must match
- danger of deadlock
- program will stall (forever!)
• Possible to write very complicated programs, but …
- most scientific codes have a simple structure
- often results in simple communications patterns
• Use collective communications where possible
- may be implemented in efficient ways
28
![Page 28: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/28.jpg)
Summary (i)
• Messages are the only form of communication
- all communication is therefore explicit
• Most systems use the SPMD model
- all processes run exactly the same code
- each has a unique ID
- processes can take different branches in the same codes
• Basic communications form is point-to-point
- collective communications implement more complicated patterns
that often occur in many codes
29
![Page 29: Message-Passing Programming with MPI · Concepts if/then/else Languages Java Fortran struct Python C/C++ Subroutines Implementations icc pgcc -fast crayftn gcc -O3 OO Processes SPMD](https://reader030.vdocuments.net/reader030/viewer/2022040400/5e6e6158d3a32d04da34fca8/html5/thumbnails/29.jpg)
Summary (ii)• Message-Passing is a programming model
- that is implemented by MPI
- the Message-Passing Interface is a library of function/subroutine calls
• Essential to understand the basic concepts
- private variables
- explicit communications
- SPMD
• Major difficulty is understanding the Message-Passing model
- a very different model to sequential programming
30
if (x < 0)
print(“Error”);
exit;