inter process communication -...

Inter Process Communication

Prof. Ananthanarayana V.S.

Dept. Of Information Technology

N.I.T.K., Surathkal

Objectives of This Module

• This module discusses the fundamental issue pertaining to types of communications.

• The application layer problems in communication are addressed.

• Different semantics of send and receive are discussed.

• Should give general idea for the use of inter‐process communications in real life applications.

Road Map

• Message Passing Communication

• Request – Reply Communication

• Transaction Communication

• Group Communication

Message Passing Communication

Message Passing Communication

• Basic communication constructs/ primitives– Send(destination, message)– Receive(source, message)– Here, source/destination is process name or link or mailbox or port

• Issues in message passing communication– Direct Vs. Indirect– Blocking Vs. Non‐blocking – Reliable Vs. Unreliable– Buffered Vs. Unbuffered

Direct Communication – Verities

• Unique process name– This can be done by concatenating host name and process id to identify the source/destination

– Note that the format for both send and receive primitives are symmetric

– There is a bidirectional communication between source and destination. That means, for every send primitive there is a receive primitive

A B

Direct Communication – Unique process name

.

.

.Send(p2@machine2, buffer[])...

p1@machine1 p2@machine2

.

.

.Recv(p1@machine1, buff[])...

Format is Symmetric, Communication is bidirectional (for every Send/Recv, there is Recv/Send

command)

Direct Communication – Verities • Receive message from unknown source

– Source in Receive primitive is a variable which gets the value (sender id) when the message is received.

– Asymmetric sender and receive primitive. [Only sender need to specify the receiver id and receiver may receive the message from any sender]

– Unidirectional path between sender and receiver [No waiting for specific sender by receiver]

A

BC

Direct Communication – Receive messages from unknown source

p1@machine1

p3@machine3

p2@machine2

.Send(p_x@machine_n, buffer[]).



.Recv( variable, buffer[]).

p_x@machine_n

variable = p1@machine1, on receipt of message from p1@machine1variable = p2@machine2 on receipt of message from p2@machine2variable = p3@machine3 on receipt of message from p3@machine3

Direct Communication – Verities

• Connections or links– This introduces concept of multiple paths or links between processes for direct communication

– Each link is identified by a unique number called link number

– Links are similar to virtual circuits in Computer Networks; unidirectional

A B

Indirect Communication

• This type of communication is used when sending processes are not concerned with the identity of receiving processes and receiving processes are interested only in the message and not the identity of sending processes.

• Example: Multiple clients may request services from one of the multiple servers.

Clients Servers

Any client request may be served by any available server. So sender (client) and receiver (server) concerned only with data and not id of each other

Network

Indirect Communication –Implementation using mailbox

• Multipoint communication

• Multipath communication

MAIL BOX

MAIL BOX

MAIL BOX

Indirect Communication –Implementation using mailbox

• Multipoint communication

• Multipath communication

MAIL BOX

MAIL BOX

MAIL BOX

Mail box is a global data structure shared by sender (producer) and receiver (consumer) processes. Producer produces messages and put them into mail box, consumer receives messages from the mail box

Message Synchronization & Buffering

Process

Kernel

Network

Message Synchronization & Buffering

Process

Kernel

Network

When sending the message from source to destination process, the message is passed to sender kernel, which is then pass over communication network, then to receiver kernel; finally to the destination process

Synchronization points

• While sending the message, synchronization can take place at one or more of the following points– Between Sender and Sending kernel– Between Sending kernel and Receiving kernel– Between Receiving kernel and Receiver

• Based on above synchronization points, we have different kind of semantics under send and receive primitives.

• Buffering space is present in the sender’s kernel, receiver’s kernel and in the communication network. These can be logically combined into a single buffer.

Semantics of Receive Primitive

• Blocking receive:– Receive has only one semantic, that is the process has to wait till a message is received by the process

.Recv(......)..

Send

Semantics of Send Primitive

• Non blocking:– The sender process is released after the message has been composed and copied onto sender’s kernel

– This is asynchronous send

– Buffer space is assumed to be unbound

– The sender process is blocked if the sending kernel is not ready to accept the message

• Blocking send:– Sender process is released after the message has been

transmitted into the communication network• Reliable blocking send:

– Sender process is released after the message has been received by the destination kernel

• Explicit blocking send:– Sender process is released after the message has been received

by the destination (receiver) process• Request and reply:

– Sender process is released after the message has been received by the receiver and response is returned to the sender

– Also called Client‐Server communication

Semantics of Send Primitive ‐ Contd

Semantics of Send PrimitiveSending Process

Sending Kernel

Network

Receiving Process

Receiving Kernel

SENDER IS BLOCKED


Sending Kernel

Network

NON BLOCKING SEND (till message reaches S kernel)

Receiving Process

Receiving Kernel

SENDER IS BLOCKED


Sending Kernel

Network


Receiving Process

Receiving Kernel

BLOCKING SEND (till message reaches n/w)

SENDER IS BLOCKED


Sending Kernel

Network


Receiving Process

Receiving Kernel

BLOCKING SEND (till message reaches n/w)RELIABLE BLOCKING SEND (till message reaches R kernel)

SENDER IS BLOCKED


Sending Kernel

Network


Receiving Process

Receiving Kernel

BLOCKING SEND (till message reaches n/w)RELIABLE BLOCKING SEND (till message reaches R kernel)EXPLICIT BLOCKING SEND (till message reaches receiver)

SENDER IS BLOCKED


Sending Kernel

Network


Receiving Process

Receiving Kernel

BLOCKING SEND (till message reaches n/w)RELIABLE BLOCKING SEND (till message reaches R kernel)EXPLICIT BLOCKING SEND (till message reaches receiver)REQUEST AND REPLY (till response from receiver reaches back sender)

SENDER IS BLOCKED

Semantics of Send and Receive Primitives

• Depending on the need of applications, the algorithms in distributed environment consider any of the send primitives

• If buffer space is unbound, then the send primitive is non blocking (asynchronous send).

• If there is no buffer in any component then, the sender and receiver must be synchronized to perform the message transfer (i.e., request –reply type of send)

• If there exists buffer in some components then, blocking or reliable or explicit blocking send primitive is applicable

Pipes

• Used to provide message passing communication at higher level

• Implemented with finite size FIFO byte stream buffer• Created by pipe system call in Unix which provides one way flow of data

• Format:– int pipe(int *pipe_descriptor)– On successful, this system call returns two pipe descriptors

• pipe_descriptor[0] – for reading and • pipe_descriptor[1] – for writing

Pipes ‐ Contd

• Normal mode of using the pipe is to provide one way communication between parent and child(s)

• Procedure:– Parent process first call pipe(int * des) system call and then fork() system call

– Close des[0] (i.e. read descriptor) at parent side and close des[1] (i.e. write descriptor) at child side

Pipe ‐ Contd

des[0] – (close)

des[1] ‐√ (keep it open)

Parent process executes pipe(des) first

des[1] – (close)

des[0] ‐√ (keep it open)

Child process

fork()

pipe (FIFO byte stream buffer)

Kernel

Data flow ‐ unidirectional

Execution of ‘who|sort|lpr’ in Unix environment

des1[0] – (close)

des1[1] ‐ √ (keep it open)

des1[0] – √ ( keep it open)

des2[1] ‐ √ (keep it open)

des2[1] – (close)

des2[0] ‐√ (keep it open)

who process sort process lpr process

pipe 1

Data flow ‐ unidirectional

pipe 2

Kernel

Note: shell forks three processes to execute who, sort and lpr at command level. Two pipes are then created and are linked as shown in the flow diagram

Use of Pipes for Bidirectional Communication

• Bidirectional communication between parent and a child process• Procedure:

– Create pipe1() and pipe2() at parent– fork() at parent– Parent closes read descriptor of pipe 1(say des1[0])– Note write descriptor of pipe1(say des1[1]) is open at parent– Parent closes write descriptor of pipe 2( say des2[1]– Note read descriptor of pipe2 (say des2[0]) is open at parent– Child closes write descriptor of pipe1(say des1[1])– Note read descriptor of pipe1(say des1[ 0]) is open at child– Child closes read descriptor of pipe2(say des2[0])– Note write descriptor of pipe2(say des2[1]) is open at child

• Now there will be two way communication between parent and its child

Use of Pipes for Bidirectional Communication

des2[0] –√(keep it open)

des1[1] ‐√(keep it open)

Parent process

des2[1] – √(keep it open)

des1[0] –√(keep it open)

Child process

fork()

pipe 1

pipe 2

Named Pipe

• In pipe based communication, the pipe descriptor is shared. The possible communication is between parent and child processes (related processes)

• To provide communication between unrelated processes within the same file system, FIFO FILES are used between the processes, called named pipes.

• mknod – unix command is used to create FIFO file which can be written by one process and read by another process. Normal file based operations ‐fopen(), fread(), fwrite() and fclose() system calls are used.

Sockets

• It is the communication endpoint of communication link and is managed by the transport services.

• Sockets are used for communication between processes in heterogeneous domain.

• It is established by socket system call, which returns a descriptor and informs the kernel about the protocol (TCP/UDP) the process is going to use.

• The system call is described by– (protocol, local address, local process/port, foreign address, foreign process/port)

Socket for Connection Oriented Protocol

Socket

Bind()

Listen()

Accept()

Read()

Write()

Protocol

Local address + port

Ready to accept connection

Block until connection from client

Process request

SERVER CLIENT

Socket

Connect()

Protocol

To the Local address + port given by the system

Write()

Read()

Data (Request)

Data (Reply)

Socket for Connectionless Protocol

Socket

Bind()

Receive From()

Send To()

SERVER

Protocol Socket

Bind()

Receive From()

Send To()

Block until data received from the client

CLIENT

Group Communication

Group Communication

• Idea is group of processes receive messages from source– If group size = N, communication is broadcasting– Else (group size < N) communication is multicasting– N : number of processes in a DCS

• The source may belong to the same group of recipients – Example: Updating the balance of account in the banking application where the accounts are replicated

• The source may not belong to the same group of recipients. – Example: Remote request at Mangalore for booking the ticket from Bangalore to Chennai in a Railway reservation system

Software Architecture for Broadcast Protocol

Broadcast Server

Application process

Application process

Application process

Application process

Application process

Application process

Application process

NetworkNETWORKBroadcast Server

Broadcast Server

Broadcast Server

Software Architecture for Broadcast Protocol

Broadcast Server

Application process

Application process

Application process

Application process

Application process

Application process

Application process

NetworkNETWORKBroadcast Server

Broadcast Server

Broadcast Server

Application process (AP): The process who generates and sends messages to the connected broadcast server or the process who receives the message from the connected broadcast serverBroadcast server (BS): The server responsible to satisfy the broadcast protocol requirements before delivering the messages to application process

• Unanimity:– A message is delivered either to all processes in the group or not at all

(atomic property – by undo/redo)• Termination:

– The result of a message submission to the broadcast server is known to the submitted broadcast server within a finite time period

• Uniformity:– If a message is delivered to a process in a group, then it should be

delivered to all the processes in that group– Enforce atomic property by redo operations

• Order:– BS will deliver the messages to APs in the group based on any one of the

following ordering requirements:• FIFO order• Casual order• Total order

Requirements of Broadcast Protocol

Ordering of messages• Broadcast server delivers its messages to the AP in any of the

following order:– FIFO : If an AP broadcasts message p and then q, all APs receives q

after p• Ex : In banking, money transfer w.r.t an account: If message M1 (credit money) and message M2 (debit money) w.r.t. an account are sent to do needful action by an AP, then it has to be received and hence processed in the same order at all APs where that account information is replicated.

– CAUSAL : If ∃SEND(p) → SEND(q) then p is received before q by all APs• Ex : Bus reservation: If two customer (X and Y) try to reserve the same seat in the same bus, then the observed order is X ‐> Y, then same order is preserved at all APs.

– TOTAL : If an AP receives p before q, then all other APs should receive p before q

• Ex : DME algorithm: In Lamport’s DME algorithm, if processes requests to enter their CSs, then order in which they can enter will be observed in same way by all the processes

Rules to Ensure the Broadcast Protocol Requirements

• Agreed Delivery Rule: – BS before delivering a message to its APs , it ensures that all previous messages are delivered to them as per order requirements.

• Safe Delivery Rule:– Before a BS delivers a message to its APs, it ensures that all other BSs have received that message.

Performance Criteria of Broadcast Protocol

• Delivery time:– It is the time gap between a message is en‐queued for broadcasting at a BS and that message is actually delivered to all BSs by the BS.

• Stability time:– It is the time gap between a message is en‐queued for broadcasting at a BS and all the BSs knows that the message is delivered to all other BSs.

• Throughput:– Average number of messages from APs broadcasted by the BS.

Atomic Broadcast Protocol

• A broadcast protocol is said to be atomic, if it satisfies all the requirements of Broadcast protocol and ordering requirement is total.

A General Overview of Atomic Broadcast Protocol

• 1. All messages which are to be delivered to APs are ordered globally– This can be achieved by assigning a unique sequence number

(USN) globally to each messages to be delivered. The possible ways are:a. Using centralized server to generate USN and every BS has to send

their message to the centralized server and this server will broadcast after attaching UCN– Problem: Traffic near the centralized server + single point of failure

b. Use centralized server to get USN and the message delivery is by individual BS– Problem: Single point of failure

c. BSs are arranged in the form of a ring with a token (which carries USN) circulating in this structure. The BS need to grab the token and update the USN value and attach the same to the message to be broadcasted.

d. The token is available on demand using DME protocol and this token is used to obtain USN.

A General Overview of Atomic Broadcast Protocol – Contd.

• 2. Transfer the messages to each of the BSs.– This can be achieved as follows:

• In case of 1a, Centralized server do the broadcasting

• In case of 1b, the individual BS do the broadcasting

• In case of 1c, the message can be appended to the token itself

• In case of 1d, the individual BS do the broadcasting

A General Overview of Atomic Broadcast Protocol – Contd.

• 3. At each BS, determine the fact that the message is stable.– This means that every BS knows that the message is delivered to every other BSs. This can be achieved as follows:

• In case of ring network with token, two full circulation of token in the network guarantees the stability. (WHY?)

• When the message is broadcasted, each BS will broadcast either +iveor –ive acknowledgement. In +ive acknowledgement scheme, the stability time will be low but the # of messages will be more. In case of –ive acknowledgement scheme, the stability time will be high but the # of messages will be less. (WHY?)

• 4. Once the message is stable at all BSs, deliver it to their APs provided all their previous messages has been delivered. (BSS Protocol)

Remote Procedure Call


• Types of procedure calls:– Local procedure call from the same program

main(){….y = sqrt(x);…

}Float sqrt(int a){…}


– Local procedure call (LPC) from library using static linking (i.e. linking during compilation)

• Ex: $ gcc test.c –lm – o test– Here math library is linked at compilation time since test.c uses some function defined in math library.

– Local procedure call from the library using dynamic linking (i.e., linking at run time)

• Ex: libm.so.1. The functions from these libraries are linked to the program at run time using dynamic linkers

– S ld –linux.so.1

– System call• These are call to the functions which are executed in kernel mode.

– Ex: int n

Remote Procedure Call ‐ RPC

• In RPC, the calling process (client) sends the request along with parameters to the remote called process (server) for execution. After execution, the called process returns the result to the calling process.

• Typically, RPC is request – reply communication with local procedure call

• Typically RPC is based on client stub and server stub. Stubs are used to hide distribution and heterogeneity (platforms and programming languages)

Design Issues in RPCA. STUB STRUCTURE: Widely used structure for RPC

ClientReturn Callreply request

Client StubMessage to Parameter toparameter message

Transport layerReceive Send

ServerCall Returnrequest reply

Server StubMessage to Parameter toparameter message

Transport layerReceive Send

Network

1

2

3 4

5

6 7

8

910

11

12

Procedure execution

Note: 1 to 12 indicates flow of data /information in that order

B. RPC Compilation:– The compilation of RPC programs requires three major components in RPC package:

• An interface specification file – specifies interface aspect of client and server side

• A RPC generator which take the interface specification file as input and produces client and server stub source code as output

• A run time library for supporting the execution of a RPC including the support for binding, data conversion and communication.

Design Issues in RPC ‐ Contd

– Server stub code contains registering of server at the binding server as initialization part. It also contains code for message to parameter and parameter to message conversion as per interface specification file.

– Client stub code contains how does client locate the server based on the service required as initialization part. It also contains code for message to parameter and parameter to message conversion as per interface specification file.

– Binding server maintains registry of servers. On request by client, it returns the address of the server


Client side Client Stub

Program Procedure

LPC(…)

Return

From

Local call

Query the binding

server

Pack &

Transmit to the server

identified

Unpack Result

Receive Query

Return server address

Unpack

LPC(…)

Pack Result

Execute proc

Return

Server Stub Remote

Procedure Procedure

Register services (Initialize)

CLIENT SERVER

BINDING SERVER

CLIENT SIDE SERVER SIDE

Wait for the replay

Working of RPC

2 3

4

1

5 6

789


Wait

Working of RPC

• STEPS (1 – 9):1. Registering of service provider’s (server) services2. Client stub receives the LPC request from client3. Querying the Registry server for the service provider address4. Binding server returns the address of service provider5. Client stub will convert the parameter into message, pack it

and transfer to the identified service provider6. Service provider will unpack it and invoke the procedure with

the parameter at service provider side7. Server stub receives the result from service provider after the

execution of the procedure, convert parameter into message8. Packed message will be sent to the client stub9. Client stub will unpack the message, convert the message into

parameters, send the parameters to client

Handling of Heterogeneity at Stubs

• Naïve approach– Generate a pair (client stub, server stub) for every client and server (2 X n X m stubs)

• Use of generic internal representation1

m

1

n

G

N

E

R

I

C

.

.

.CLIENTS

.

.

.SERVERS

n + m stubs

• Static binding ‐ binding at compile timeHandle of the server where it resides is hard‐codedSimple and efficientClient and server are tightly coupledNo load balancing

• Dynamic binding – binding at run timeLoad balancingClient and server are decoupled


C. BINDING – how does client locate server?

D. PARAMETER AND RESULT PASSING– Different parameter passing methods are:

• Call by value: wherein data is passed by value– Ex:

• Call by name:– Arguments to the function are directly substituted within the function body

and will be evaluated at run time • Call by reference:

– The reference to the variable is passed to the function rather than the value. So change of referenced value is visible outside the function

• Call by copy/restore:– It is just like call by value at the point of entry and call by reference upon exit

In RPC, only call by value and call by copy/restore are generally implemented.


call_by_value(int a){a:= a+10}

main(){int x;call_by_value(x) }

E. DATA COVERSION:– Client and server machine may use different data format. The solution to handle this are:

• Send the data along with a code identifying the format. At the remote machine translate to the local code if necessary

• Using common format for transferring data. These common formats are generally based on abstract syntax notation one (ASN.1). Examples are – SUN’s XDR (eXtended Data Representation), DEC’s IDL (Interface Definition Language)


F. EXCEPTION HANDLING:– Exceptions in server procedures such as overflow/underflow/protection violation need to be informed to the client

– Client may want to stop the execution of the server procedure


G. FAILURE HANDLING:– Case 1: Client crashes after invoking a RPC

• In this case, there is no client to receive the result of RPC and the execution procedure becomes an orphan

– Case 2: Client may invoke a RPC more than once due to (i) slow server (ii) server crash

– In view of these, following RPC semantics are possible:1. At least once: If RPC succeeds, then at least one execution of

remote procedure. Zero/ partial/ one/ more execution of the procedure otherwise

2. Exactly once: If RPC succeeds, then exactly one execution of remote procedure. Zero/ partial/ one/ more execution of the procedure otherwise

3. At most once: Same as exactly once semantics. But on failure, there is no side effects. This is also called “zero or one” semantics


Transaction Communication

Transaction

• A process which changes or accesses the database• It has statements to read data, write data and perform

operations on them.– Ex: Transferring Rs. 5000 from an account 10451234 to an

account 10175123• Further, more than one transactions may be executed in an

interleaved fashion to improve the efficiency of the system.• Transaction should be executed in atomic fashion. i.e., it

should be executed to its entirety or not at all.• This is ensured by Transaction Management. It contains (i)

concurrency control mechanism / scheduler (like 2PL), which guarantees serializable execution of transactions (ii) recovery management (like deferred recovery) which guarantees the recovery from failures

Transaction Management Model

Transaction Manager

Scheduler

Recovery Manager

Cache Manager

Data Manager

DB

TnT2T1

Distributed Transactions

• They are transactions in distributed environment.• Three types:

– Local transaction: where the data is available in the site where the transaction originates

– Remote transaction: where the location of data and location of transaction from where it is originated are different

– Global transaction: the data requested by the transaction is not available in one site. So, the transaction manager from where the transaction is originated need to decompose the transaction into sub transactions and submit to the sites where the data is available. Later, there is a need to integrate the results.

• Examples: Railway reservation system (Depicted in next slide)– Here, there are four databases pertaining to trains scheduled from four

location viz., Mangalore, Mumbai, Delhi and Chennai. These databases are having same schema but distributed data.

Railway Reservation System with Distributed Databases

NETWORK

DBSDBS

DBS DBS

L R

MUMBAI

DELHIMANGALORE

CHENNAI

DBS: Database SystemL: Local transactionR: Remote transactionG: Global transaction

G

• Local transaction (L):– A customer at Mangalore is requesting to book a ticket from Mangalore to

Mumbai.– This is a local transaction because, this can be handled by the site (database)

from where it originates (i.e., Mangalore).• Remote transaction (R):

– A customer at Mangalore is requesting to book a ticket from Chennai to Mumbai.

– Here, the local site (Mangalore) need to transfer the request to Chennai site to do the required processing and is transparent to the customer

• Global transaction (G):– A customer at Mangalore is requesting to book tickets from Chennai to

Mumbai and from Mumbai to Delhi.– Here, the local site (Mangalore) need to decompose the request into two (sub

transactions), one is request to book the ticket from Chennai to Mumbai, transfer it to Chennai site and the other is request to book the ticket from Mumbai to Delhi , transfer it to Mumbai site. Later, if both sub‐transactions are successful, result need to be integrated at Mangalore site and provide to the customer.

Railway Reservation System with Distributed Databases

TP Monitors

• Support the execution of distributed transactions.

• A transaction from a client may invoke more than one procedure calls (sub Transactions) and should not be treated as independent one. Example, Global transactions explained in previous slide.

• Example: Withdrawal of money from account A at Server1 and deposit it at account B at Server2

TP Monitors

• Transaction T is set of procedure calls placed between BOT and EOT

• Each procedure call is a sub transaction executed at a Server (RPC)

• Transaction Manager ensures the atomicity of T using 2PC protocol (discussed in followingslides using flow diagram).

‐Registering Transactions‐Informing completion of Transaction‐ Final result of T based on 2PC

TP Monitor Overview

BOTSub transaction ST1Sub transaction ST2Sub transaction ST3…Sub transaction STnEOT

TRANSACTION, T

Client Stub Server StubLPCi

TP Manager‐Maintains the status of live transaction (i.e. status of each of STi )‐On completion of each STi , invoke 2Phase Commit Protocol (2PC)

‐Registering of service‐Participation in 2PC

Server/Procedure provider

START

Write Prepare‐Commit T Log

START

Is Ready to Commit?

Write ABORT T in Log

WAITNO ‐T

PREPARE

Write READY T in Log

N

YYES‐T

Any NO? or Un‐received Message?

Write ABORT T in LogY

WAITGLOBAL‐ABORT

Write COMMIT T in Log

GLOBAL‐COMMITN

COMMIT

Type of Message

Write COMMIT T in Log

GLOBAL‐COMMIT

Write ABORT T in Log

GLOBAL‐ABORT

ABORT

COORDINATOR PARTICIPANT

2PC PROTOCOL

Summary

• In this module, different mode of communication that may exists between two processes are discussed.

• Semantics of ‘send’ and ‘receive’ are discussed.

• Requirements, Architecture and Overview of Atomic Broadcast Protocol is highlighted.

• RPC and its variations, Distributed transactions and issues related to that are outlined.

References

• UNIX Network Programming (Volume 2) – W. Richard Stevens, Prentice Hall

• Distributed Operating Systems and Algorithms– Randy Chaw and T Johnson, Addison Wesley

• Distributed Operating Systems– A S Tanenbaum, Prentice Hall

• Distributed Systems Concepts and Design– G Coulouris and J Dollimore, Addison Wesley

inter process communication -...

Documents