Parallel and Distributed Parallel and Distributed ProgrammingProgramming

Kashif BilalKashif Bilal

Parallel and Distributed Parallel and Distributed Programming.Programming.

• Generally We have two types of Generally We have two types of Distributed and parallel Systems.Distributed and parallel Systems.– Multiprocessors ( Shared memory Multiprocessors ( Shared memory

System)System)•Having single memory accessible to every Having single memory accessible to every

processorprocessor

– Multicomputers ( Message Passing Multicomputers ( Message Passing System)System)•Every Processor has its own local memory.Every Processor has its own local memory.

Parallel Programming Parallel Programming ArchitecturesArchitectures

• Generally we have two basic Generally we have two basic architectures for parallel architectures for parallel programmingprogramming

– Supervisor Worker Structure (Master Supervisor Worker Structure (Master Slave model)Slave model)

– Hierarchy Structure (Tree Model )Hierarchy Structure (Tree Model )

Supervisor Worker ModelSupervisor Worker Model

• There is only one level of hierarchy in this structure

• one supervisor and many workers.• Supervisor

– It normally interacts with the user– activates the workers – assigns work to the workers– collects results from the workers.

• Workers perform calculations and send result back to supervisor

Hierarchy StructureHierarchy Structure

•The hierarchy structure allows the workers to create new levels of workers.

•The top-level supervisor is the initiating task, which creates a set of workers at the second level.

•These workers may create other sets of workers.

Process …..Process …..

• A program (task) in execution.A program (task) in execution.

•A process is a set of executable instructions (program) which runs on a processor.

• Process is basic entity to achieve Process is basic entity to achieve parallelism in both multiprocessors parallelism in both multiprocessors and multicomputers.and multicomputers.

Distributes System Distributes System ClassificationClassification• John Flynn classified computers in four

categories.

– SISD – Single Instruction stream/Single Data stream

– SIMD – Single Instruction stream/Multiple Data stream ORSPMD – Single Program/Multiple Data stream

– MISD – Multiple Instruction stream/Single Data stream (No real application)

– MIMD – Multiple Instruction stream/Multiple Data stream.

Message PassingMessage Passing

•The method by which data from one processor’s memory is copied to the memory of another processor.

• In distributed memory systems, data is generally sent as packets of information over a network from one processor to another.

Thumb Recognition Thumb Recognition ExampleExample• Suppose we have a database of 10 lack Suppose we have a database of 10 lack

thumb impressions and their related data.thumb impressions and their related data.

• A user comes, system takes ones impression A user comes, system takes ones impression and searches the database for ones and searches the database for ones information.information.

• Suppose one database match take 1/100 Suppose one database match take 1/100 seconds.seconds.

• To search the complete database, system will To search the complete database, system will take approx. 10000 seconds. i.e. 2.7 hours.take approx. 10000 seconds. i.e. 2.7 hours.

Algorithm for Thumb Algorithm for Thumb RecognitionRecognition

Main()Main(){{Picture =Capture the thumb impression to be matchedPicture =Capture the thumb impression to be matchedDetails = Match( picture )Details = Match( picture )}}Match(Picture)Match(Picture){{Pick record from database one by one and match it with Pick record from database one by one and match it with

PicturePictureIf (matched)If (matched)

return Details of that recordreturn Details of that record}}

Distributed Algorithm…Distributed Algorithm…

Main()Main(){{Receive impression, start record number and end record number Receive impression, start record number and end record number

from master.from master.Details = Match( picture, start_no, end_no )Details = Match( picture, start_no, end_no )Send details to Master or supervisor code.Send details to Master or supervisor code.}}Match( Picture , start_no, end_no)Match( Picture , start_no, end_no){{Pick record from database one by one and match it with Picture from Pick record from database one by one and match it with Picture from

record no start_no till end_no.record no start_no till end_no.If (matched)If (matched)

return Details of that recordreturn Details of that record}}

Problem…Problem…

• How to receive data from supervisor How to receive data from supervisor to workers.to workers.

• How send details back to supervisor How send details back to supervisor process.process.

• How supervisor allocate and How supervisor allocate and communicate with workers.communicate with workers.

• Etc……Etc……

Possible Solutions Possible Solutions

• Make a new programming language for Make a new programming language for programming in distributed and programming in distributed and parallel.parallel.

• Change existing languages.Change existing languages.• Build libraries and API’s( Set of Build libraries and API’s( Set of

functions) to perform all tasks related to functions) to perform all tasks related to Distributed and parallel programming Distributed and parallel programming like remote spawning to task, like remote spawning to task, communication, synchronization etc.communication, synchronization etc.

PVM and MPIPVM and MPI

• PVM and MPI are two famous libraries used PVM and MPI are two famous libraries used for parallel and distributed programming.for parallel and distributed programming.

• PVM is a bit older than MPI.PVM is a bit older than MPI.

• MPI is now considered as standard for MPI is now considered as standard for building parallel and distributed programs.building parallel and distributed programs.

• Both libraries provide different functions to Both libraries provide different functions to perform different tasks required in parallel perform different tasks required in parallel programs. programs.

PVM (Parallel Virtual PVM (Parallel Virtual Machine)Machine)•Started as a research project in 1989•The Parallel Virtual Machine (PVM)

was originally developed at Oak Ridge National Laboratory and the University of Tennessee.

•The PVM offers a powerful set of process control and dynamic resource management functions.

PVM…..PVM…..• It provides programmers with a library of

routines for– Initiation and termination of tasks.– Synchronization.– Alteration of the virtual machine configuration.– Facilitates message passing via a number of

simple constructs.– Interoperability among different heterogeneous

computers.

• Programs written for some architecture can be copied to another architecture, compiled and executed without modification.

PVM…PVM…•PVM has two components:

– A library of PVM routines– A daemon that should reside on all the

hosts in the virtual machine.

•The pvmd serves as a message router and controller. It provides a point of contact, authentication, process control, and fault detection.

•One task (process) normally instantiated on one machine (Supervisor), and other tasks as instantiated automatically by supervisor.

Task Creation…Task Creation…• A task in PVM can be started manually or

can be spawned from another task.• The initiating task is always activated

manually by simply running its executable code on one of the hosts.

• Other PVM tasks can be created dynamically from within other tasks.

• The function pvm_spawn() is used for dynamic task creation.

• The task• that calls the function pvm_spawn() is

referred to as the parent and the newly created

• tasks are called children.

To Create a child, you must To Create a child, you must specify..specify..

1. The machine on which the child will be started

2. A path to the executable file on the specified machine

3. The number of copies of the child to be created

4. An array of arguments to the child tasks

Pvm_spawn()…Pvm_spawn()…

•Num=pvm_spawn (Child, Arguments, Flag, Where, How Many, &Tids)

•This function has six parameters and returns the actual number of the successfully created tasks in the variable num.– Child: Name of task (process) to be Child: Name of task (process) to be

executed.executed.– Arguments : Same as command line Arguments : Same as command line

arguments.arguments.– Flag : Flag : A flag value decides what machine

will run the spawned task.

• Flag valuesFlag values– PvmTaskDefault PvmTaskDefault

• 0 PVM can choose any machine to start task 0 PVM can choose any machine to start task

– PvmTaskHost PvmTaskHost •1 1 wherewhere specifies a particular host specifies a particular host

– PvmTaskArch PvmTaskArch •2 2 wherewhere specifies a type of architecture specifies a type of architecture

• Where : Depends on value of flag.Where : Depends on value of flag.• How Many : Number of tasks to be How Many : Number of tasks to be

spawned.spawned.• Tids: Array to store Tid’s of spawned Tids: Array to store Tid’s of spawned

tasks.tasks.• Return : Total number of tasks Return : Total number of tasks

created.created.

•n1 = pvm_spawn(“simple_pvm”, 0, 1, “mynode11”, 1, &tid1)– Will create 1 tasks and run executables Will create 1 tasks and run executables

named simple_pvm on node mynode.named simple_pvm on node mynode.

• numt = pvm_spawn( “simple_pvm”, numt = pvm_spawn( “simple_pvm”, 0, PvmTaskArch, “LINUX", 5, &tids);0, PvmTaskArch, “LINUX", 5, &tids);– Will create 5 tasks running simple_pvm Will create 5 tasks running simple_pvm

onspecific architecture i.e. Linux.onspecific architecture i.e. Linux.

• res = pvm_spawn(“simple_pvm”, res = pvm_spawn(“simple_pvm”, NULL, PvmTaskDefault, "", 5, NULL, PvmTaskDefault, "", 5, children);children);– Will ask pvm to choose nodes itself.Will ask pvm to choose nodes itself.

Task Id’sTask Id’s

• All PVM tasks are identified by an All PVM tasks are identified by an integer task identifierinteger task identifier

• When a task is created it is assigned a When a task is created it is assigned a unique identifier (TID)unique identifier (TID)

• Task identifiers can be used to Task identifiers can be used to identify senders and receivers during identify senders and receivers during communication. communication.

• It can also be used to assign functions It can also be used to assign functions to different tasks based on their TIDsto different tasks based on their TIDs

Retrieval Of TidRetrieval Of Tid

• Task’s TID Task’s TID pvm_mytid() pvm_mytid()Mytid = pvm_mytid;Mytid = pvm_mytid;

• Child’s TID Child’s TID pvm_spawn() pvm_spawn()pvm_spawn(…,…,…,…,…, &tid);pvm_spawn(…,…,…,…,…, &tid);

• Parent’s TID Parent’s TID pvm_parent() pvm_parent()my_parent_tid = pvm_parent();my_parent_tid = pvm_parent();

• Daemon’s TID Daemon’s TID pvm_tidtohost() pvm_tidtohost()daemon_tid = pvm_tidtohost(id);daemon_tid = pvm_tidtohost(id);

Communication among Communication among TasksTasks• Communication among PVM tasks is

performed using the message passing approach,

• Achieved using a library of routines and a daemon.

• User program communicates with the PVM daemon

• Daemon Daemon determines the destination of each message.

• Communication is generally asynchronous.

User

applicationLibrary

Daemon

1

2 3

4

User

applicationLibrary

Daemon

5

67

8

Sending Task Receiving Task

How to send a messageHow to send a message

• Sending a message requires 3 steps.Sending a message requires 3 steps.1.1. A send buffer must be initialized.A send buffer must be initialized.2.2. The message is packed into the buffer.The message is packed into the buffer.3.3. The completed message is sent to its The completed message is sent to its

destination(s).destination(s).

• Receiving of message requires 2 stepsReceiving of message requires 2 steps1.1. The message is received.The message is received.2.2. The received items are unpacked.The received items are unpacked.

Message Sending…Message Sending…

• Buffer Creation (before packing)Buffer Creation (before packing)Bufid = pvm_initsend(encoding_option)Bufid = pvm_initsend(encoding_option)

Bufid = pvm_mkbuf(encoding_option)Bufid = pvm_mkbuf(encoding_option)

Encoding optionEncoding option MeaningMeaning

00 XDRXDR

11 No encodingNo encoding

22 Leave data in placeLeave data in place

Message Sending…Message Sending…

•Data PackingData Packingpvm_pk*()pvm_pk*()– pvm_pkstr() – one argumentpvm_pkstr() – one argument

pvm_pkstr(“This is my data”);pvm_pkstr(“This is my data”);– Others – three argumentsOthers – three arguments

1.1.Pointer to the first itemPointer to the first item

2.2.Number of items to be packedNumber of items to be packed

3.3.StrideStride

pvm_pkint(my_array, n, pvm_pkint(my_array, n, 1);1);

Message Sending…Message Sending…• Point to point (one receiver)Point to point (one receiver)

info = pvm_send(tid, tag)info = pvm_send(tid, tag)

• broadcast (multiple receivers)broadcast (multiple receivers)info = pvm_mcast(tids, n, tag)info = pvm_mcast(tids, n, tag)info = pvm_bcast(group_name, tag)info = pvm_bcast(group_name, tag)

• Pack and Send (one step)Pack and Send (one step)info = pvm_psend(tid, tag, my_array, length, data info = pvm_psend(tid, tag, my_array, length, data type)type)

• The call returns integer status code The call returns integer status code info. A negative value of info indicates info. A negative value of info indicates an error.an error.

Receiving a MessageReceiving a Message• PVM supports three types of message PVM supports three types of message

receiving functions: blocking, nonblocking, receiving functions: blocking, nonblocking, and timeout.and timeout.

• BlockingBlockingbufid = pvm_recv (tid, tag)bufid = pvm_recv (tid, tag)

-1 -1 wild card in either tid or tag wild card in either tid or tag

• NonblockingNonblocking bufid = pvm_nrecv (tid, tag)bufid = pvm_nrecv (tid, tag)

bufid = 0 (no message was received)bufid = 0 (no message was received)

• TimeoutTimeout bufid = pvm_trecv (tid, tag, timeout)bufid = pvm_trecv (tid, tag, timeout)

bufid = 0 (no message was received)bufid = 0 (no message was received)

Data UnpackingData Unpacking

pvm_upk*()pvm_upk*()– pvm_upkstr() – one argumentpvm_upkstr() – one argument

pvm_upkstr(string);pvm_upkstr(string);

– Others – three argumentsOthers – three arguments1.1.Pointer to the first itemPointer to the first item2.2.Number of items to be unpackedNumber of items to be unpacked3.3.StrideStride

pvm_upkint(my_array, n, 1);pvm_upkint(my_array, n, 1);

Data UnpackingData Unpacking• pvm_upk*()pvm_upk*()

– pvm_upkstr() – one argumentpvm_upkstr() – one argument

pvm_upkstr(string);pvm_upkstr(string);

– Others – three argumentsOthers – three arguments1.1.Pointer to the first itemPointer to the first item

2.2.Number of items to be unpackedNumber of items to be unpacked

3.3.StrideStride

pvm_upkint(my_array, n, 1);pvm_upkint(my_array, n, 1);

• Receiving and unpacking in single stepReceiving and unpacking in single step– Info = pvm_precv(tid, tag, my_array, len,

datatype, &src,&atag, &alen)

Work Assignment (different Work Assignment (different programs)programs)

info1 = pvm_spawn(“worker1”, 0, 1, info1 = pvm_spawn(“worker1”, 0, 1, “lpc01”, 1, &tid1)“lpc01”, 1, &tid1)

info2 = pvm_spawn(“worker2”, 0, 1, info2 = pvm_spawn(“worker2”, 0, 1, “lpc02”, 1, &tid2)“lpc02”, 1, &tid2)

info3 = pvm_spawn(“worker3”, 0, 1, info3 = pvm_spawn(“worker3”, 0, 1, “lpc03”, 1, &tid3)“lpc03”, 1, &tid3)

info4 = pvm_spawn(“worker4”, 0, 1, info4 = pvm_spawn(“worker4”, 0, 1, “lpc04”, 1, &tid4)“lpc04”, 1, &tid4)

Distributed Algorithm…Distributed Algorithm…Main()Main(){{Receive impression, start record number and end Receive impression, start record number and end

record number from master.record number from master.Details = Match( picture, start_no, end_no )Details = Match( picture, start_no, end_no )Send details to Master or supervisor code.Send details to Master or supervisor code.}}Match( Picture , start_no, end_no)Match( Picture , start_no, end_no){{Pick record from database one by one and match it Pick record from database one by one and match it

with Picture from record no start_no till end_no.with Picture from record no start_no till end_no.If (matched)If (matched)

return Details of that recordreturn Details of that record}}

Distributed Algorithm using Distributed Algorithm using PVMPVM• MasterMaster

Main()Main(){{

Input thumb Impression…Input thumb Impression…Int arr[2];Int arr[2];Int start=1;Int start=1;Pvm_spawn(“slave”,0,0,””,100,children);Pvm_spawn(“slave”,0,0,””,100,children);Pvm_init_send(XDR);Pvm_init_send(XDR);for(i=0;i<100;i++)for(i=0;i<100;i++){{arr[0]=start; arr[0]=start; arr[1]=start+999;arr[1]=start+999;pvm_pkint(arr,2,1);pvm_pkint(arr,2,1);pvm_send(children[i],)pvm_send(children[i],)Start+=1000;Start+=1000;Pvm_send(picture);Pvm_send(picture);} } Pvm_recv(-1,-1);Pvm_recv(-1,-1);}}

Distributed Algorithm using Distributed Algorithm using PVMPVM• SlaveSlave

Main()Main(){{Int arr[2];Int arr[2];Pvm_recv(pvm_parent(),1);Pvm_recv(pvm_parent(),1);Pvm_upkint(arr,2,1);Pvm_upkint(arr,2,1);Pvm_recv(pvm_parent(),-1);Pvm_recv(pvm_parent(),-1);Match(picture,arr[0],arr[1]);Match(picture,arr[0],arr[1]);}}

Match( Picture , start_no, end_no)Match( Picture , start_no, end_no){{Pick record from database one by one and match Pick record from database one by one and match

it with Picture from record no start_no till it with Picture from record no start_no till end_no.end_no.

If (matched)If (matched)return Details of that record// return Details of that record// pvm_send(pvm_parent,2);pvm_send(pvm_parent,2);

}}

Distributed Algorithm using Distributed Algorithm using PVMPVM• Master and slave both in single programMaster and slave both in single program

Main()Main(){{

int tid = pvm_parent();int tid = pvm_parent();if(tid=PvmNoParent)if(tid=PvmNoParent)

{{Write supervisor code Here..Write supervisor code Here..

}}ElseElse{{Write worker code here..Write worker code here..}}}}

PVM Master ProgramPVM Master Program#include <stdio.h>#include <stdio.h>#include <pvm3.h>#include <pvm3.h>

int main()int main(){{ int myTID;int myTID; int x = 12;int x = 12; int children[10];int children[10]; int res;int res;

myTID = pvm_mytid();myTID = pvm_mytid(); printf("Master: TID is %d\n", myTID);printf("Master: TID is %d\n", myTID);

res = pvm_spawn("slave", NULL, PvmTaskDefault, "", 1, res = pvm_spawn("slave", NULL, PvmTaskDefault, "", 1, children);children);

if (res<1) {if (res<1) { printf("Master: pvm_spawn error\n");printf("Master: pvm_spawn error\n"); pvm_exit();pvm_exit(); exit(1);exit(1); }}

pvm_initsend(PvmDataDefault);pvm_initsend(PvmDataDefault); pvm_pkint(&x, 1, 1);pvm_pkint(&x, 1, 1); pvm_send(children[0], 1);pvm_send(children[0], 1); pvm_recv(-1, -1);pvm_recv(-1, -1); pvm_upkint(&x, 1, 1);pvm_upkint(&x, 1, 1);

printf("Master has received x=%d\n", printf("Master has received x=%d\n", x);x);

pvm_exit();pvm_exit();

return 0;return 0;}}

How to Compile and How to Compile and Execute?Execute?• gcc -I /opt/pvm3/include myprogram.c gcc -I /opt/pvm3/include myprogram.c

-L /opt/pvm3/lib/LINUX/ -lpvm3 -o -L /opt/pvm3/lib/LINUX/ -lpvm3 -o myprogramexe myprogramexe

• Illustration: Illustration: • gcc = C Compiler gcc = C Compiler • -I = Include -I = Include • opt/pvm3/include = Path (include files) opt/pvm3/include = Path (include files) • myprogram = Name of source code file myprogram = Name of source code file • -L = search path info as well as the pm3 -L = search path info as well as the pm3

lib lib • -o = output file-o = output file