eec 688/788 secure and dependable computing lecture 13 wenbing zhao department of electrical and...

EEC 688/788EEC 688/788Secure and Dependable ComputingSecure and Dependable Computing

Lecture 13Lecture 13

Wenbing ZhaoWenbing ZhaoDepartment of Electrical and Computer EngineeringDepartment of Electrical and Computer Engineering

Cleveland State UniversityCleveland State University

[email protected]@ieee.org

22

04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable

ComputingComputing Wenbing ZhaoWenbing Zhao

OutlineOutline• Fault tolerance and replication• Event ordering• Group communication systems– Ordered multicast– Techniques to implement ordered multicast

• Reference: – Reliable distributed systems, by K. P. Birman, Springer;

Chapter 14-16

33



RedundancyRedundancy

• To tolerant fault, some form of redundancy must be used– Replication in time (transaction processing)– Replication in space– Redundancy in software design (n-version

programming)

• The three types of replication techniques are complimentary to each other

44



Replication in TimeReplication in Time

• Replication in time– From the present state, apply one or more operations – If the system fails before completion, abort and rollback

to the original state– Start all over again

• Example: transaction processing– Essence: roll-backward recovery

55



Replication in SpaceReplication in Space

• Replication in space: – Run multiple instances (replicas) of the systems so that

if one fails, one or more replicas can take over– Must assume independent faults– Must ensure consistency among the replicas

66



N-version ProgrammingN-version Programming

• N-version programming (redundancy in software design)– The system (or component) has n-different designs and

implementations– In case of permanent software bugs, a different version

is used for each replica, or for repeated executions

77



Replication is not a Trivial TaskReplication is not a Trivial Task

• Suppose we want to replicate a server using the most popular (an inexpensive) approach– We run two servers on separate computers– The primary sends a log (its state, and/or

logged incoming messages) to the backup – If primary crashes, the backup soon catches

up and can take over

88



Split Brain SyndromeSplit Brain Syndrome

primary

backup

Clients initially connected to primary, which keeps thebackup up to date. Backup collects the log and updates its state

log

99




Transient problem causes some links to break but not all.Backup thinks it is now primary, primary thinks backup is down

primary

backup

1010




Some clients still connected to primary, but one has switchedto backup and one is completely disconnected from both

primary

backup

1111



Replica ConsistencyReplica Consistency

• Replicas must be coordinated appropriately so that we can achieve strong replica consistency:– At the end of each execution step, all replicas must

have the same state– The outputs from every replica must be consistent all

the time

1212



Replica ConsistencyReplica Consistency

• Techniques to achieve strong replica consistency– As a prerequisite, we need to have an agreement on

the membership of the replica group– Ensure that the same set of inputs is delivered to all

replicas and the inputs must be in the same order– Ensure deterministic execution due to each input at

every replica (applicable to active and semi-active replication)

1313



One Copy ImageOne Copy Image• When a component or system is replicated, we

must ensure that the replicated unit appears as a single copy to external components/systems that interact with it

• Need a gateway in between the replicated and non-replicated units

• Alternatively, the gateway module can be integrated with the non-replicated unit

• Main tasks of the gateway– Multicast requests/replies to the replicas– Detect and suppress duplicated replies/requests

1414



Replication StylesReplication Styles• Active replication

– Every input (request) is executed by every replica– Every replica generates the outputs (replies)– Voting is needed to cope with non-fail-stop faults

• Passive replication– One of the replicas is designated as the primary replica– Only the primary replica executes requests– The state of the primary replica is transferred to the backups

periodically or after every request processing

• Semi-active replication– One of the replicas is designated as the leader (or primary)– The leader determines the order of execution– Every input is executed by every replica per the leader’s instruction

1515



DuplicateInvocationSuppressed

DuplicateResponsesSuppressed

Active ReplicationActive ReplicationActively Replicated

Client Object AActively Replicated

Server Object B

RM RM RM RM RM

1616



Active Replication with VotingActive Replication with Voting

Question: to cope with f number of faults (non-malicious), how many replicas are needed?

1717



State Transfer

State State

Response

Invocation

Passive ReplicationPassive ReplicationPassively Replicated

Client Object APassively Replicated

Server Object B

PrimaryReplica

PrimaryReplica

RMRM RM RMRM

Question: can passive replication tolerate non-fail-stop faults?

1818



Ordering info

Ordering info Ordering info

Response

Invocation

Semi-Active ReplicationSemi-Active ReplicationSemi-Actively Replicated

Client Object ASemi-Actively Replicated

Server Object B

PrimaryReplica

PrimaryReplica

RMRM RM RMRM

1919



Ensuring Strong Replica ConsistencyEnsuring Strong Replica Consistency

• Possible strategies– For active replication, use a group

communication system that guarantees total ordering of all messages (plus deterministic processing in each replica)

– Passive replication with systematic checkpointing

– Semi-active replication– Use two-phase commit

2020



Total Ordering of MessagesTotal Ordering of Messages• What is total ordering of messages?

– All replicas receive the same set of messages in the same order– Atomic multicast – If a message is delivered to one replica, it is also

delivered to all correct replicas

• With replication, we need to ensure total ordering of messages sent by a group of replicas to another group of replicas– FIFO ordering between one sender and a group is not sufficient

m1

m2m1

m1m1

m1

m2m2

m1

2121



Potential Sources of Non-determinismsPotential Sources of Non-determinisms

• Multithreading– The order of accesses of shared data by different threads might

not be the same at different replicas

• System calls/library calls– A call at one replica might succeed while the same call might fail

at another replica. E.g., memory allocation, file access

• Host/process specific information– Host name, process id, etc.– Local clocks - gettimeofday()

• Interrupts– Delivered and handled asynchronously – big problem

2222



Event OrderingEvent Ordering• “Time, Clocks, and the Ordering of Events in

a Distributed System”, by Leslie Lamport, Communications of the ACM, July 1978, Volume 21, Number 7, pp.558-565– What usually matters is not that all processes agree

on exactly what time it is, but rather, that they agree on the order in which events occur

2323



Happens-Before RelationHappens-Before Relation

• Assumptions:– The system is composed of a collection of processes,

each process consists of a sequence of events– The events of a process form a sequence, where a

occurs before b in this sequence if a happens before b– The sending or receiving of a message is an event in a

process

2424



Happens-Before RelationHappens-Before Relation• The happens-before relation “→” on the set of

events of a system is the relation satisfying the following three conditions:– If a and b are events in the same process, and a

comes before b, then a → b– If a is the sending of a message by one process and

b is the receipt of the same message by another process, then a → b

– If a → b and b → c, then a → c

2525



Partial OrderingPartial Ordering• Not all events have the happens-before relationship• Two distinct events a and b are said to be

concurrent if a → b and b → a– Neither event can causally affect the other– This introduces a partial ordering of events in a system

with concurrently operating processes

• “a happens before b” means that information can flow from a to b

• “a is concurrent with b” means that there is no information flow between a and b

2626



How to Capture the Partial Ordering?How to Capture the Partial Ordering?

• Use logical clocks to capture the partial ordering– Define a clock Ci for each process Pi. Assign a

number Ci(a) to any event a in that process

– The entire system of clocks is represented by the function C which assigns to any event b the number C(b), where C(b) =Cj(b) if b is an event in process Pj

– The clocks Ci are logical clocks rather than physical clocks

2727



Lamport ClockLamport Clock

• A Lamport logical clock is a monotonically increasing software counter

• Each process Pi keeps its own logical clock Ci to apply Lamport timestamps to events

• To capture the happens-before relation →, processes must do the following:– Before each event at Pi: Ci := Ci+1

– When Pi sends a message m, it piggybacks t = Ci

– When Pj receives (m,t): Cj := max(Cj,t) + 1

e → e’ C(e) < C(e’)

2828



Lamport Clock: An ExampleLamport Clock: An Example

a b

c d

e f

m1

m2

21

3 4

51

p1

p2

p3

Physical time

2929



Group Communication SystemGroup Communication System

• Services provided by the GCS– Membership service: who is up and who is down

• Deals with failure detection and more

– Reliable, ordered, multicast service• FIFO, causal, total

– Virtual synchrony service• Virtual synchrony synchronizes membership change with

multicasts

• GCS is often used to build fault tolerant systems

3030



Reliable MulticastReliable Multicast• Reliable multicast – the message is targeted to

multiple receivers, and all receivers receive the message reliably– Positive or negative acknowledgement– Need to avoid ack/nack implosion

• Distinguish receiving from delivery!

Application

Middleware

Receiving

Delivering

3131



Ordered Reliable MulticastOrdered Reliable Multicast• Ordered reliable multicast – if many messages are

multicast by many senders, in what order the messages are delivered at the receivers?– First in first out (FIFO)– Causal – the causal relationship among msgs preserved– Total – all msgs are delivered at all receivers in the same

order

3232



FIFO Ordered MulticastFIFO Ordered Multicast

• FIFO or sender ordered multicast:Messages are delivered in the order they were sent (by any single sender)

p

q

r

s

a

b c d

e

delivery of c to p is delayed until after b is delivered

3333



Causally Ordered MulticastCausally Ordered Multicast

• Causal or happens-before ordering:If send(a) send(b) then deliver(a) occurs before deliver(b) at common destinations

p

q

r

s

a

b

3434





p

q

r

s

a

b cdelivery of c to p is delayed until after b is delivered

3535





p

q

r

s

a

b c

e

delivery of c to p is delayed until after b is deliverede is sent (causally) after b

3636





p

q

r

s

a

b c d

e

delivery of c to p is delayed until after b is delivereddelivery of e to r is delayed until after b&c are delivered

3737



Totally Ordered MulticastTotally Ordered Multicast

• Total ordering:Messages are delivered in same order to all recipients (including the sender)

p

q

r

s

a

b c d

e

all deliver a, b, c, d, then e

3838



Implementing Total OrderingImplementing Total Ordering

• Use a token that moves around– Token has a sequence number– When you hold the token you can send the next burst

of multicasts

• Use a sequencer to order all multicast– Message is first multicast to all, including the

sequencer; then the sequencer determines the order for the message and informs all

– Or send to the sequencer and the sequencer multicast with total order information

– Each sender can take turn to serve as the sequencer

eec 688/788 secure and dependable computing lecture 13 wenbing zhao department of electrical and...

Documents

dependable computing

primary thinks backup

primary crashes

switchedto backup

different version

separate computersthe

software designthe system

present state