Distributed (Operating) Systems -Communication in Distributed Systems-

Fall 2011Kocaeli University Computer

Engineering Department

Communication inDistributed Systems

• Communication is done through message passing• Expressing communication through message passing is

harder than using primitives based on shared memory

• Remote Procedure Calls– Transparency but poor for passing references– Ideal for client server applications

• Message-oriented Communication• Stream-oriented Communication

– Continuous media

Communication Between Processes

• Unstructured communication– Use shared memory or shared data structures

• Structured communication– Use explicit messages (IPCs)

• Distributed Systems: both need low-level communication support (why?)

Types of Communication -1

• Transient Communication– A message is stored by the communication system only as

long as the sending and the receiving application are executing

– Typically all transport level communication is transient communication

• Persistent Communication– A message that has been submitted for transmission is kept

by middleware as long as it takes to deliver it to the receiver– Receiving application don’t need to be executing when the

message when the message is submitted

Types of Communication -2

• Asynchronous Communication– Sender continues immediately after it has submitted its

message for transmission– It means, message is temporarily stored in middleware

upon submission• Synchronous Communication

– Sender is blocked until its request is known to be accepted– Three points

• Sender informs message is delivered• Middleware informs message is delivered• Sender informs with the response after processing

Types of Communication -3

• Connection-oriented (telephone)– TCP

• Connectionless (mailbox – dropping a letter)– IP, UDP

Persistence and Synchronicity in Communication

Persistence

• Persistent communication– Messages are stored until (next) receiver is ready– Examples: email, pony express

Transient Communication

• Transient communication– Message is stored only so long as

sending/receiving application are executing– Discard message if it can’t be delivered to next

server/receiver– Example: transport-level communication services

offer transient communication– Example: Typical network router – discard

message if it can’t be delivered next router or destination

Synchronicity

• Asynchronous communication– Sender continues immediately after it has submitted

the message– Need a local buffer at the sending host

• Synchronous communication– Sender blocks until message is stored in a local buffer

at the receiving host or actually delivered to sending– Variant: block until receiver processes the message

• Six combinations of persistence and synchronicity

Persistence and Synchronicity Combinations

a) Persistent asynchronous communication (e.g., email)b) Persistent synchronous communication

Persistence and Synchronicity Combinations

c) Transient asynchronous communication (e.g., UDP)d) Receipt-based transient synchronous communication

Persistence and Synchronicity Combinations

e) Delivery-based transient synchronous communication at message delivery (e.g., asynchronous RPC)

f) Response-based transient synchronous communication (RPC)

Layered Protocols 1

• Communication messaging rules are defined in protocols

• Due to the absence of shared memory, all communication in distributed systems is based on sending and receiving (low-level) messages

• Many different agreements– IBM’s EBCDIC and ASCII– How many volts should be used to signal 0– How does the receiver know which is the last bit– How can the receiver detect if a message has been damaged

Layered Protocols 2

• ISO/OSI never widely used and are essentially dead.

• In the OSI model communication is divided up into 7 layers

• OSI layers were never popular. In contrast, protocols developed for the internet, such as TCO and IP, are mostly used

Communication Protocols• There are some rules that communicating processes must adhere to –known as

protocols• Protocols are agreements/rules on communication• Protocols could be connection-oriented or connectionless

Layered Protocols

• A typical message as it appears on the network.

Layers -1

• Physical layer– Transmitting 0s and 1s– How many bits per sec transfer– Can transmission take place in both direction

• Data Link Layer– Puts special bit pattern on the start and end of each

frame– Computing checksum

• Network Layer– Routing

Layers -2

• Transport Protocol– Turns the underlying network into something than an application

developer can use– Messages from the application is broken into small pieces-packets– Which packets have been sent, which have been received, which

should be retransmitted– Connection-oriented: messages arrive in the same order (as in TCP)– Connectionless: message can arrive in different order (as in UDP)– Another example of transport protocol is RTP– The combination of TCP/IP is now used as a de facto standard for

network communication

Client-Server TCP

Layers -3

• Session Layer– Enhanced version if transport layer– Keeps track of which party is currently talking and

it provides a synchronization– Puts checking points – in case of crash going to the

last check point• Presentation Layer

– Concerned with the meaning of the bits

Middleware Protocols• Middleware: layer that resides between an OS and an application

– May implement general-purpose protocols that warrant their own layers

OSI Model summary

• In this model, clear distinction between applications, application-specific protocols and general purpose protocols.

• Application-specific protocol– FTP– HTTP

• General-purpose protocols– Useful to many applications but cannot be qualified as

transport protocols. These protocols fall into the category of middleware protocols.

Middleware Protocols 1

• Some services are not tied to any specific applications, but instead can be integrated into a middleware system as a general service– Authentication protocols

• Proof of claimed identity

– Atomicity protocols• Widely applied in transactions

– Distributed locking protocols• Resources can be protected against simultaneous

accesses

Middleware Protocols 2

• Also supports high-level communication services– Remote Procedure Call (RPC) – Message Oriented Middleware (MOM)– RTP: Streams for transferring real-time data, such

as needed for multimedia applications

Remote Procedure Calls

• Goal: Make distributed computing look like centralized computing

• Allow remote services to be called as procedures– Transparency with regard to location,

implementation, language• Issues

– How to pass parameters– Bindings– Semantics in face of errors

Example of an RPCNo message passing at all is visible to the programmer.

Divide programs up and add communication protocols

blah, blah, blah

bar = add(i,j);

blah, blah, blah

blah, blah, blah

bar = add(i,j);

blah, blah, blah

Int add(int x, int y ) { if (x>100)

return(y-2); else if (x>10)

return(y-x); else

return(x+y);}

Int add(int x, int y ) { if (x>100)

return(y-2); else if (x>10)

return(y-x); else

return(x+y);}

ClientServer

protocol

RPC Semantics• Principle of RPC between a client and server program [Birrell&Nelson 1984]

Other RPC Models

• Asynchronous RPC– Request-reply behavior often not needed– Server can reply as soon as request is received and execute

procedure later• Deferred-synchronous RPC

– Use two asynchronous RPCs– Client needs a reply but can’t wait for it; server sends reply via

another asynchronous RPC• One-way RPC

– Client does not even wait for an ACK from the server– Limitation: reliability not guaranteed (Client does not know if

procedure was executed by the server).

Asynchronous RPC

a) The interconnection between client and server in a traditional RPCb) The interaction using asynchronous RPC

Deferred Synchronous RPC

• A client and server interacting through two asynchronous RPCs

Conventional Procedure Calla) Parameter passing in a local

procedure call: the stack before the call to read

b) The stack while the called procedure is active

Count = read(fd, buf, nbytes)

Possible Issues

• Calling and called procedures run on different machines

• They execute in different address spaces• Parameters and results have to be passed, it can be

complicated when the machines are not identical.– How do you represent integers – big-endian little-endian

• Either or both machines can crash and each of the possible failures causes different problems.

Parameter Passing

• Local procedure parameter passing– Call-by-value– Call-by-reference: arrays, complex data structures

• No sense BUT in RMI it is OK

• Remote procedure calls simulate this through:– Stubs – proxies– Flattening – marshalling

• Related issue: global variables are not allowed in RPCs

Client and Server Stubs

• Client makes procedure call (just like a local procedure call) to the client stub

• Server is written as a standard procedure• Stubs take care of packaging arguments and

sending messages• Packaging parameters is called marshalling• Stub compiler generates stub automatically from

specs in an Interface Definition Language (IDL)– Simplifies programmer task

Steps of a Remote Procedure Call

1. Client procedure calls client stub in normal way2. Client stub builds message, calls local OS3. Client's OS sends message to remote OS4. Remote OS gives message to server stub5. Server stub unpacks parameters, calls server6. Server does work, returns result to the stub7. Server stub packs it in message, calls local OS8. Server's OS sends message to client's OS9. Client's OS gives message to client stub10. Stub unpacks result, returns to client

Marshalling

• Problem: different machines have different data formats– Intel: little endian, SPARC: big endian

• Solution: use a standard representation– Example: external data representation (XDR)

Binding

• Problem: how does a client locate a server?– Use Bindings

• Server– Export server interface during initialization– Send name, version no, unique identifier, handle (address)

to binder• Client

– First RPC: send message to binder to import server interface– Binder: check to see if server has exported interface

• Return handle and unique identifier to client

Binder: Port Mapper• Server start-up: create port• Server stub calls svc_register to register prog #, version # with local port mapper• Port mapper stores prog #, version #, and port• Client start-up: call clnt_create to locate server port• Upon return, client can call procedures at the server

Case Study: SUNRPC• One of the most widely used RPC systems• Developed for use with NFS• Built on top of UDP or TCP• Multiple arguments marshaled into a single structure• At-least-once semantics if reply received, at-least-zero semantics if

no reply. With UDP tries at-most-once• Use SUN’s eXternal Data Representation (XDR)

– Big endian order for 32 bit integers, handle arbitrarily large data structures• XDR was originally designed for specifying eXternal Data

Representation• XDR has been extended to become Sun RPC IDL• An interface contains a program number, version number,

procedure definition and required type definitions

Rpcgen: generating stubs

• Q_xdr.c: do XDR conversion• Detailed example: later in this course