reliable distributed systems agenda for march 23 rd, 2006 group membership services use of multicast...

Reliable Distributed Systems

Agenda for March 23rd, 2006 Group Membership Services Use of Multicast to support replications Virtual Synchrony Model Applications - Examples

All based on Ken Birman’s slide set

Architecture

Membership Agreement, “join/leave” and “P seems to be unresponsive”

3PC-like protocols use membership changes instead of failure notification

Applications use replicated data for Applications use replicated data for high availabilityhigh availability

Issues? How to “detect” failures

Can use timeout Or could use other system monitoring tools

and interfaces Sometimes can exploit hardware

Tracking membership Basically, need a new replicated service System membership “lists” are the data it

manages We’ll say it takes join/leave requests as input

and produces “views” as output

Architecture

GMS

A

B

C

D

join leave

join

A seems to have failed

{A}

{A,B,D}

{A,D}

{A,D,C}

{D,C}

X Y Z

Application processes

GMS processes

membership views

Issues Group membership service (GMS) has

just a small number of members This core set will tracks membership for a

large number of system processes Internally it runs a group membership

protocol (GMP) Full system membership list is just

replicated data managed by GMS members, updated using multicast

GMP design What protocol should we use to

track the membership of GMS Must avoid split-brain problem Desire continuous availability

We’ll see that a version of 3PC can be used

But can’t “always” guarantee liveness

Multicast Primitives

To support replication

Ordering properties: FIFO

Fifo or sender ordered multicast: fbcast

Messages are delivered in the order they were sent (by any single sender)

p

q

r

s

a e

Ordering properties: FIFO

Fifo or sender ordered multicast: fbcast

Messages are delivered in the order they were sent (by any single sender)

p

q

r

s

a

b c d

e

delivery of c to p is delayed until after b is delivered

Implementing FIFO order Basic reliable multicast algorithm has

this property Without failures all we need is to run it on

FIFO channels (like TCP, except “wired” to our GMS

With failures need to be careful about the order in which things are done but problem is simple

Multithreaded applications: must carefully use locking or order can be lost as soon as delivery occurs!

Ordering properties: Causal Causal or happens-before ordering: cbcast

If send(a) send(b) then deliver(a) occurs before deliver(b) at common destinations

p

q

r

s

a

b

Ordering properties: Causal Causal or happens-before ordering: cbcast

If send(a) send(b) then deliver(a) occurs before deliver(b) at common destinations

p

q

r

s

a

b cdelivery of c to p is delayed until after b is delivered

Ordering properties: Causal Causal or happens-before ordering: cbcast

If send(a) send(b) then deliver(a) occurs before deliver(b) at common destinations

p

q

r

s

a

b c

e

delivery of c to p is delayed until after b is deliverede is sent (causally) after b

Ordering properties: Causal Causal or happens-before ordering: cbcast

If send(a) send(b) then deliver(a) occurs before deliver(b) at common destinations

p

q

r

s

a

b c d

e

delivery of c to p is delayed until after b is delivereddelivery of e to r is delayed until after b&c are delivered

Insights about c/fbcast These two primitives are asynchronous:

Sender doesn’t get blocked and can deliver a copy to itself without “stopping” to learn a safe delivery order

If used this way, the multicast can seem to sit in the output buffers a long time, leading to surprising behavior

But this also gives the system a chance to concatenate multiple small messages into one larger one.

Concatenation

Application sends 3 asynchronous cbcasts

Multicast Subsystem

Message layer of multicast system combines them in a single packet

State Machine Concept Sometimes, we want a replicated

object or service that advances through a series of “state machine transitions”

Clearly will need all copies to make the same transitions

Leads to a need for totally ordered multicast

Ordering properties: Total

Total or locally total multicast: abcast

Messages are delivered in same order to all recipients (including the sender)

p

q

r

s

a

b c d

e

all deliver a, b, c, d, then e

Ordering properties: Total

Can visualize as “closely synchronous” Real delivery is less synchronous, as on the

previous slide

p

q

r

s

a

b c d

e

all deliver a, b, c, d, then e

Often conceive of causal order as a form of total order!

Point is that causal order is totally ordered for any single causal chain.

We’ll use this observation later

p

q

r

s

a

b c d

e

all receive a, b, c, d, then e

Implementing Total Order Many ways have been proposed

Just have a token that moves around Token has a sequence number When you hold the token you can send the next

burst of multicasts Extends to a cbcast-based version

We use this when there are multiple concurrent threads sending messages

Transis and Totem extend VT causal order to a total order

But their scheme involves delaying messages and sometimes sends extra multicasts

How to think about order? Usually, we think in terms of state machines and

total order But all the total order approaches are costly

There is always someone who may need to wait for a token or may need to delay delivery

Loses benefit of asynchronous execution Could be several orders of magnitude slower!

So often we prefer to find sneaky ways to use fbcast or cbcast instead

Reliable Distributed Systems

Virtual Synchrony

Virtual Synchrony

A powerful programming model! Called virtual synchrony It offers

Process groups with state transfer, automated fault detection and membership reporting

Ordered reliable multicast, in several flavors

Extremely good performance

Why “virtual” synchrony?

What would a synchronous execution look like?

In what ways is a “virtual” synchrony execution not the same thing?

A synchronous execution

p

q

r

s

t

u

With true synchrony executions run in genuine lock-step.

Virtual Synchrony at a glance

With virtual synchrony executions only look “lock step” to the application

p

q

r

s

t

u

What about membership changes? Virtual synchrony model synchronizes

membership change with multicasts Idea is that:

Between any pair of successive group membership views…

… same set of multicasts are delivered to all members

If you implement code this makes algorithms much simpler for you!

Process groups with joins, failures

crash

G0={p,q} G1={p,q,r,s} G2={q,r,s} G3={q,r,s,t}

p

q

r

s

tr, s request to join

r,s added; state xfer

t added, state xfer

t requests to join

p fails

Implementation? When membership view is changing, we

also need to terminate any pending multicasts Involves wiring the fault-tolerance

mechanism of the multicast to the view change notification

Tricky but not particularly hard to do Resulting scheme performs well if

implemented carefully

Virtual Synchrony at a glance

p

q

r

s

t

u

We use the weakest (hence fastest) form of communication possible

Chances to “weaken” ordering Suppose that any conflicting updates are

synchronized using some form of locking Multicast sender will have mutual exclusion Hence simply because we used locks, cbcast

delivers conflicting updates in order they were performed!

If our system ever does see concurrent multicasts… they must not have conflicted. So it won’t matter if cbcast delivers them in different orders at different recipients!

Causally ordered updates Each thread corresponds to a different

lock

In effect: red “events” never conflict with green ones!

p

r

s

t1

2

3

4

5

1

2

In general?

Replace “safe” (dynamic uniformity) with a standard multicast when possible

Replace abcast with cbcast Replace cbcast with fbcast

Unless replies are needed, don’t wait for replies to a multicast

Why “virtual” synchrony?

The user sees what looks like a synchronous execution Simplifies the developer’s task

But the actual execution is rather concurrent and asynchronous Maximizes performance Reduces risk that lock-step execution

will trigger correlated failures

Correlated failures Why do we claim that virtual synchrony

makes these less likely? Recall that many programs are buggy Often these are Heisenbugs (order sensitive)

With lock-step execution each group member sees group events in identical order So all die in unison

With virtual synchrony orders differ So an order-sensitive bug might only kill one

group member!

Programming with groups Many systems just have one group

E.g. replicated bank servers Cluster mimics one highly reliable server

But we can also use groups at finer granularity E.g. to replicate a shared data structure Now one process might belong to many groups

A further reason that different processes might see different inputs and event orders

Embedding groups into “tools”

We can design a groups API: pg_join(), pg_leave(), cbcast()…

But we can also use groups to build other higher level mechanisms Distributed algorithms, like snapshot Fault-tolerant request execution Publish-subscribe

Distributed algorithms

Processes that might participate join an appropriate group

Now the group view gives a simple leader election rule Everyone sees the same members, in

the same order, ranked by when they joined

Leader can be, e.g., the “oldest” process

Distributed algorithms

A group can easily solve consensus Leader multicasts: “what’s your

input”? All reply: “Mine is 0. Mine is 1” Initiator picks the most common value

and multicasts that: the “decision value”

If the leader fails, the new leader just restarts the algorithm

Distributed algorithms

A group can easily do consistent snapshot algorithm Either use cbcast throughout system,

or build the algorithm over gbcast Two phases:

Start snapshot: a first cbcast Finished: a second cbcast, collect process

states and channel logs

Distributed algorithms: Summary

Leader election Consensus and other forms of

agreement like voting Snapshots, hence deadlock

detection, auditing, load balancing

More tools: fault-tolerance Suppose that we want to offer clients

“fault-tolerant request execution” We can replace a traditional service with a

group of members Each request is assigned to a primary (ideally,

spread the work around) and a backup Primary sends a “cc” of the response to the request

to the backup Backup keeps a copy of the request and steps

in only if the primary crashes before replying Sometimes called “coordinator/cohort”

just to distinguish from “primary/backup”

Publish / Subscribe

Goal is to support a simple API: Publish(“topic”, message) Subscribe(“topic”, event_hander)

We can just create a group for each topic Publish multicasts to the group Subscribers are the members

Scalability warnings! Many existing group communication

systems don’t scale incredibly well E.g. JGroups, Ensemble, Spread Group sizes limited to perhaps 50-75 members And individual processes limited to joining

perhaps 50-75 groups (Spread: see next slide) Overheads soar as these sizes increase

Each group runs protocols oblivious of the others, and this creates huge inefficiency

Other “toolkit” ideas We could embed group

communication into a framework in a “transparent” way Example: CORBA fault-tolerance

specification does lock-step replication of deterministic components

The client simply can’t see failures But the determinism assumption is painful,

and users have been unenthusiastic And exposed to correlated crashes

Other similar ideas

There was some work on embedding groups into programming languages But many applications want to use them

to link programs coded in different languages and systems

Hence an interesting curiosity but just a curiosity

More work is needed on the whole issue

Existing toolkits: challenges

Tensions between threading and ordering We need concurrency (threads) for

perf. Yet we need to preserve the order in

which “events” are delivered This poses a difficult balance for

the developers

Preserving order

Group Communication Subsystem: A library linked to theapplication, perhaps with its own daemon processes

G1={p,q}

m3 m4 G2={p,q,r}

Time application

p

q

r

m1 m2

m3 m4

Features of major virtual synchrony platforms

Isis: First and no longer widely used But was perhaps the most successful;

has major roles in NYSE, Swiss Exchange, French Air Traffic Control system (two major subsystems of it), US AEGIS Naval warship

Also was first to offer a publish-subscribe interface that mapped topics to groups

Features of major virtual synchrony platforms

Totem and Transis Sibling projects, shortly after Isis Totem (UCSB) went on to become

Eternal and was the basis of the CORBA fault-tolerance standard

Transis (Hebrew University) became a specialist in tolerating partitioning failures, then explored link between vsync and FLP

Features of major virtual synchrony platforms Horus, JGroups and Ensemble

All were developed at Cornell: successors to Isis These focus on flexible protocol stack linked directly

into application address space A stack is a pile of micro-protocols Can assemble an optimized solution fitted to specific

needs of the application by plugging together “properties this application requires”, lego-style

The system is optimized to reduce overheads of this compositional style of protocol stack

JGroups is very popular. Ensemble is somewhat popular and supported by a

user community. Horus works well but is not widely used.

JGroups (part of JBoss) Developed by Bela Ban

Implements group multicast tools Virtual synchrony was on their “to do” list But they have group views, multicast,

weaker forms of reliability Impressive performance! Very popular for Java community

Downloads from http://www.JGroups.org

Spread Toolkit Developed at John Hopkins

Focused on a sort of “RISC” approach Very simple architecture and system Fairly fast, easy to use, rather popular

Supports one large group within which user sees many small “lightweight” subgroups that seem to be free-standing

Protocols implemented by Spread “agents” that relay messages to apps

Summary? Role of a toolkit is to package commonly

used, popular functionality into simple API and programming model

Group communication systems have been more popular when offered in toolkits If groups are embedded into programming

languages, we limit interoperability If groups are used to transparently replicate

deterministic objects, we’re too inflexible Many modern systems let you match the

protocol to your application’s requirements

Reliable Distributed Systems

Applications

Applications of GCS Over the past three weeks we’ve heard

about group communication Process groups Membership tracking and reporting “new views” Reliable multicast, ordered in various ways Dynamic uniformity (safety), quorum protocols

So we know how to build group multicast… but what good are these things?

Applications of GCS

Today, we’ll review some practical applications of the mechanisms we’ve studied Each is representative of a class Goal is to illustrate the wide scope of

these mechanisms, their power, and the ways you might use them in your own work

Sample Applications Wrappers and

Toolkits Distributed Program-

ming Languages Wrapping a Simple

RPC server Wrapping a Web Site

Hardening Other Aspects of the Web

Unbreakable Stream Connections

Reliable Distributed Shared Memory

What should the user “see”? Presentation of group communication

tools to end users has been a controversial topic for decades!

Some schools of thought: Direct interface for creating and using groups Hide in a familiar abstraction like publish-

subscribe or Windows event notification Use inside something else, like a cluster mgt.

platform a new programming language Each approach has pros and cons

Toolkits

Most systems that offer group communication directly have toolkit interfaces User sees a library with various calls

and callbacks These are organized into “tools”

Style of coding? User writes a program in Java, C, C++,

C#... The program declares “handlers” for

events like new views, arriving messages Then it joins groups and can

send/receive multicasts Normally, it would also use threads to

interact with users via a GUI or do other useful things

Toolkit approach: Isis Join a group, state transfer:

Gid = pg_join(“group-name”,PG_INIT, init_func, PG_NEWVIEW, got_newview,

XFER_IN, rcv_state, XFER_OUT, snd_state, … 0); Multicast to a group:

nr = abcast(gid, REQ, “%s,%d”, “a string”, 1234, ALL, “%f”, &fltvec);

Register a callback handler for incoming messagesisis_entry(REQ, got_msg);

Receive a multicast:void got_msg(message *mp) {

Msg_scan(“%s,%d”, &astring, &anint);Reply(mp, “%f”, 123.45);

}

A group is created when a join is first issued. In this case the group initializer

function is called. The user needs to code that function. Here the “new view”

function, also supplied by the user, gets called when the group membership

changes

If the group already exists, a leader is automatically selected and its

XFER_OUT routine is called. It calls xfer_out repeatedly to send state.

Each call results in a message delivered to the XFER_IN routine, which extracts the state from the message

To send a multicast (here, a totally ordered one), you specify the group identifier from

a join or lookup, a request code (an integer), and then the message. This

multicast builds a message using a C-style format string. This abcast wants a reply

from all members; the replies are floating point numbers and the set of replies is

stored in a vector specified by the caller. Abcast tells the caller how many replies it

actually got (nr)

This is how an application registers a callback handler. In this case the

application is saying that messages with the specified request code should be passed to the procedure “got_msg”

Here’s got_msg. It gets invoked when a multicast arrived with the matching request code. This particular procedure extracts a

string and an integer from the message and sends a reply. Abcast will collect all of those

replies into a vector, set the caller’s pointer to point to that vector, and return the number of

replies it received (namely, the number of members in the current view)

Threading A tricky topic in Isis

The user needs threads, e.g. to deal with I/O from the client while also listening for incoming messages, or to accept new requests while waiting for replies to an RPC or multicast

But the user also needs to know that messages and new views are delivered in order, hence concurrent threads pose issues

Solution? Isis acts like a “monitor” with threads, but running them one at a time unless the user explicitly “exits” the monitor

A tricky model to work with!

We have… Threads, which many people find tricky Virtual synchrony, including choices of

ordering A new distributed “abstraction” (groups)

Developers will be making lots of choices, some with big performance implications, and this is a negative

Examples of tools in toolkit Group join, state

xfer Leader selection Holding a “token” Checkpointing a

group

Data replication Locking Primary-backup Load-balancing Distributed

snapshot

How toolkits work They offer a programmer API

More procedures, e.g. Create_replicated_data(“name”, type) Lock_replica(“name”) Update_replica(“name”, value) V = (type)Read_replica(“name”)

Internally, these use groups & multicast Perhaps, asynchronous cbcast as discussed last

week… Toolkit builder optimizes extensively, etc…

How programmers use toolkits

Two main styles Replicating a data structure

For example, “air traffic sector D-5” Consists of all the data associated with that

structure… could be quite elaborate Processes sharing the structure could be

very different (maybe not even the same language)

Replicating a service For high availability, load-balancing

Experience is mixed…. Note that many systems use group communication

but don’t offer “toolkits” to developers/end users Major toolkit successes include New York and Swiss

Stock Exchange, French Air Traffic Control System, US AEGIS warship, various VLSI Fab systems, etc

But building them demanded special programmer expertise and knowledge of a large, complex platform

Not every tool works in every situation! Performance surprises & idiosyncratic behavior common. Toolkits never caught on the way that transactions became standard

But there are several popular toolkits, like JGroups, Spread and Ensemble. Many people do use them

Leads to notion of “wrappers”

Suppose that we could have a magic wand and wave it at some system component “Replicatum transparentus!”

Could we “automate” the use of tools and hide the details from programmers?

Wrapper examples Transparently…

Take an existing service and “wrap” it so as to replicate inputs, making it fault-tolerant

Take a file or database and “wrap” it so that it will be replicated for high availability

Take a communication channel and “wrap” it so that instead of connecting to a single server, it connects to a group

Experience with wrappers? Transparency isn’t always a good thing

CORBA has a fault-tolerance wrapper In CORBA, programs are “active objects” The wrapper requires that these be deterministic

objects with no GUI (e.g. servers) CORBA replaces the object with a group, and

uses abcast to send requests to the group. Members do the same thing, “state machine” style So replies are identical. Give the client the first one

Why CORBA f.tol. was a flop Users find the determinism assumption too

constraining Prevents use of threads, shared memory, system

clock, timers, multiple I/O channels… Real programs sometimes use these sorts of

things unknown to the programmer Who knows how the .NET I/O library was programmed

by Microsoft? Could it have threads inside, or timers? Moreover, costs were high

Twice as much hardware… slower performance! Also, had to purchase the technology separately

from your basic ORB (and for almost same price)

Files and databases?

Here, issue is that there are other ways to solve the same problem A file, for example, could be put on a

RAID file server This provides high speed and high

capacity and fault-tolerance too Software replication can’t easily

compete

How about “TCP to a group?” This is a neat application and very

interesting to discuss. We saw it before. Let’s look at it again, carefully

Goals: Client system runs standard, unchanged TCP Server replaced by a group… leader owns the

TCP endpoint but if it crashes, someone else takes over and client sees no disruption at all!

How would this work? Revisit idea from before: Reminder: TCP is a kind of state

machine Events occur (incoming IP packets,

timeouts, read/write requests from app) These trigger “actions” (sending data

packets, acks, nacks, retransmission) We can potentially checkpoint the state of a

TCP connection or even replicate it in realtime!

How to “move” a TCP connection

We need to move the IP address We know that in the modern internet, IP

addresses do move, all the time NATs and firewalls do this, why can’t

we? We would also need to move the TCP

connection “state” Depending on how TCP was

implemented this may actually be easy!

Migrating a TCP connection

client

Initial Server

New Server

Client “knows” the server by its TCP endpoint: an IP address and port that speak TCP and have the state of this

connection

The server-side state consists of the contents of the TCP window (on the server), the socket to which the IP

address and port are bound, and timeouts or ACK/NACK “pending actions”

We can write this into a checkpoint record

TCP state

TCP state

We transmit the TCP state (with any other tasks we migrate) to the new server. It opens a

socket, binds to the SAME IP address, initializes its TCP stack out of the checkpoint received

from the old server

The client never even notices that the channel endpoint was moved!

The old server discards its connection endpoint

TCP connection state Includes:

The IP address, port # used by the client and the IP address and port on the server

Best to think of the server as temporarily exhibiting a “virtual address”

That address can be moved Contents of the TCP “window”

We can write this down and move it too ACK/NACK state, timeouts

Generalizing the idea

Create a process group Use multicasts when each event

occurs (abcast) All replicas can track state of the

leader Now if a new view shows that the

leader has failed, a replica can take over by binding to the IP address

Fault-tolerant TCP connection

client

Initial Server

New Server

With replication technology we could continuously replicate the connection state (as well as any “per task” state needed by the

server)

Fault-tolerant TCP connection

client

Initial Server

New Server

After a failure, the new server could take over, masking the

fault. The client doesn’t notice anything

What’s new?

Before we didn’t know much about multicast… now we do

This lets us ask how costly the solution would be

In particular Which multicast should be used? When would a delay be incurred?

Choice of multicast

We need to be sure that everyone sees events in the identical order Sounds like abcast

But in fact there is only a single sender at a time, namely the leader Fbcast is actually adequate! Advantage: leader doesn’t need to

multicast to itself, only to the replicas

Timeline picture

client

leader

replica

An IP packet generated by TCP

Leader fbcasts the “event description”

Leader bound to IP address

replica binds to IP address, now it owns the TCP stack

Leader doesn’t need to wait (to “sync”) here

because the client can’t see any evidence of the

leader’s TCP protocol stack state

Leader does need to wait before sending this IP packet

to the client, (to “sync”) to be sure that if he crashes, client TCP stack will be in the same

state as his was

Asynchronous multicast This term is used when we can send a

multicast without waiting for replies Our example uses asynchronous fbcast

An especially cheap protocol: often just sends a UDP packet

Acks and so forth can happen later and be amortized over many multicasts

“Sync” is slower: must wait for an ack But often occurs in background while leader

is processing the request, “hiding” the cost!

Sources of delay? Building event messages to represent

TCP state, sending them But this can occur concurrently with handing

data to the application and letting it do whatever work is required

Unless TCP data is huge, delay is very small Synchronization before sending packets

of any kind to client Must be certain that replica is in the identical

state

Using our solution?

Now we can wrap a web site or some other service Run one copy on each of two or more

machines Use our replicated TCP

Application sees identical inputs and produces identical outputs…

Repeat of CORBA f.tol. idea? Not exactly…

We do need determinism with respect to the TCP inputs

But in fact we don’t need to legislate that “the application must be a deterministic object”

Users could, for example, use threads as long as they ensure that identical TCP inputs result in identical replies

Would users accept this?

Unknown: This style of wrapping has never been explored in commercial products

But the idea seems appealing… perhaps someone in the class will find out…

Recap of today’s lecture

… we’ve looked at each of these topics and seen that with a group multicast platform, the problem isn’t hard to solve

Wrappers and Toolkits

Distributed Program-ming Languages

Wrapping a Simple RPC server

Wrapping a Web Site

Hardening Other Aspects of the Web

Unbreakable Stream Connections

Reliable Distributed Shared Memory [skipped]