ubi529 distributed algorithms global state of distributed systems
TRANSCRIPT
UBI529 Distributed Algorithms
Global State of Distributed Systems
2
Motivation
Goal: Take a snapshot of the global computation A snapshot of local states on n processes taken at exactly the
same time Two terms “global state” and “global snapshot”
Useful for debugging
Useful for backup/check-pointing
Useful for calculating global predicate E.g., Exactly how much currency do we have in the country
(notice that money flows among people constantly)?
Deadlock DetectionRollback RecoveryTermination Detection
4
The mystery of the missing dollars
A B
$400 $300
Picture taken at A - $400A sends $100 to BPicture taken at B - $400Total is $800
Send $100
5
Global Snapshot Problem
Determine the global system state (e.g. the total money )Each process records its own state No shared clock/memory
Group of photographers taking snaps of different portions and trying to combine to get the overall picture.
6
Consistent cut
Given computation (E,!) and F µ E is a cut iff
F is a consistent cut (global snapshot) iff
7
Consistent and inconsistent cuts
8
Consistent cut
(a consistent cut C) (b happened before a) b C
a b c d g
m e f
k i h jCut 1 Cut 2
A cut is a set of events.
(Not consistent)(Consistent)
P1
P2
P3
9
Consistent snapshot
The set of states immediately following a consistent cut forms
a consistent snapshot of a distributed system.
A snapshot that is of practical interest is the most recent one. Let C1
and C2 be two consistent cuts and C1 C2. Then C2 is more recent than
C1.
Analyze why certain cuts in the one-dollar bank are inconsistent.
10
Consistent snapshot
How to record a consistent snapshot? Note that
1. The recording must be non-invasive
2. Recording must be done on-the-fly.
You cannot stop the system.
11
Chandy Lamport Algorithm
Assumes
FIFO and Unidirectional channels
A bidirectional channel is modelled as two unidirectional channels
Each process has an associated color. All processes are initially white.
A process records it local state just before turning red
On turning red the process sends out a marker on all outgoing channels
On receiving a marker a white process turns red
12
Chandy-Lamport Algorithm
Works on a
(1) strongly connected graph
(2) each channel is FIFO.
An initiator initiates the algorithm
by sending out a marker ( )
13
White and red processes
Initially every process is white. When a
process receives a marker, it turns red
if it has not already done so.
Every action by a process, and every
message sent by a process gets the
color of that process.
14
Two steps
Step 1. In one atomic action, the initiator (a) Turns red (b) Records its own state (c) sends a marker along all outgoing channels
Step 2. Every other process, upon receiving a marker for the first time (and before doing anything else) (a) Turns red (b) Records its own state (c) sends markers along all outgoing channels
The algorithm terminates when (1) every process turns red, and (2) Every process has received a marker through each incoming channel.
15
Why does it work?
Lemma 1. No red message is received in a white action.
16
Why does it work?
Theorem. The global state recorded by Chandy-Lamport algorithm is equivalent to the ideal snapshot state SSS.
Hint. A pair of actions (a, b) can be scheduled in any order, if there is no causal order between them, so (a; b) is equivalent to (b; a)
SSSEasy conceptualization of the snapshot state
All white All red
17
Why does it work?
Let an observer observe the following actions:
w[i] w[k] r[k] w[j] r[i] w[l] r[j] r[l] … w[i] w[k] w[j] r[k] r[i] w[l] r[j] r[l] … [Lemma 1]w[i] w[k] w[j] r[k] w[l] r[i] r[j] r[l] … [Lemma 1]w[i] w[k] w[j] w[l] r[k] r[i] r[j] r[l] … [done!]
Recorded state
18
Example 1. Count the tokens
Let us verify that Chandy-Lamport snapshot algorithm correctly counts
the tokens circulating in the system
A
B
C
D
How to account for the channel states? Use sent and received variables for each process.
19
Chandy Lamport Algorithm
20
Algorithmpublic class RecvCamera extends Process implements Camera { . . . public RecvCamera(Linker initComm, CamUser app) { . . . for (int i = 0; i < N; i++) if (isNeighbor(i)) { closed[i] = false; chan[i] = new LinkedList(); } else closed[i] = true; } public synchronized void globalState() { myColor = red; app.localState(); // record local State; sendToNeighbors("marker", myId); // send Markers } public synchronized void handleMsg(Msg m, int src, String tag) { if (tag.equals("marker")) { if (myColor == white) globalState(); closed[src] = true; if (isDone()){
----- Display channel state (transit messages) chan[] ---- } } else { // application message if ((myColor == red) && (!closed[src])) chan[src].add(m); app.handleMsg(m, src, tag); // give it to app } } boolean isDone() { if (myColor == white) return false; for (int i = 0; i < N; i++) if (!closed[i]) return false; return true;} }
21
Lai Yang Algorithm
LY1. The initiator records its own state. When it needs to send a message m to another process, it sends a message (m, red).
LY2. When a process receives a message (m, red), it records its state if it has not already done so, and then accepts the message m.
22
Another example of distributed snapshot: Communicating State Machines
ch1
ch2
i j
up
down
up
state machine i state machine
j
send M
send M'
down
global state i ch1 j ch2 S0 down down S1 up M down S2 up M up M' S3 down M up
receive M'
receive M
23
Something unusual
Let machine i start Chandy-lamport snapshot before it has sent M along ch1. Also, let machine j receive the marker after it sends out M’ along ch2. Observe that the snapshot state is
down up M’
Doesn’t this appear strange? This state was never reached during the computation!
24
Understanding snapshot
S0
S1
i sends M j sends M'
j receives M j sends M' i receives M'
S1'
S2
S2' i sends M
j receives M i receives M'
i receives M' j receives M
S3
S0
S3'
recorded state SSS
25
Understanding snapshot
The observed state is a feasible state that is reachable
from the initial configuration. It may not actually be visited
during a specific execution.
The final state of the original computation is always
reachable from the observed state.
26
Discussions
What good is a snapshot if that state has never been visited by the system?
- It is relevant for the detection of stable predicates.- Useful for checkpointing.
27
Discussions
What if the channels are not FIFO? Study how Lai-Yang algorithm works. It does not use any marker
LY1. The initiator records its own state. When it needs to send a message m to another process, it sends a message (m, red).
LY2. When a process receives a message (m, red), it records its state if it has not already done so, and then accepts the message m.Question 1. Why will it work? Question 1 Are there any limitations of this approach?
28
Global state collection
Some applications- computing network topology- termination detection- deadlock detection
Chandy Lamport algorithm does a partial job. Each process collects a fragment of the global state, but these pieces have to be stitched together to form a global state.
29
A simple exercise
Once the pieces of a
consistent global state become
available, consider collecting the
global state via all-to-all broadcast
At the end, each process
will compute a set V, where
V= {s(i): 0 ≤ i ≤ N-1 }
i
k
j
l
s(i) s(j)
s(k) s(l)
30
All-to-all broadcast
Program broadcast (for process i}
define V.i, W.i : set of values;
initially V.i={s(i)}, W.i =
andevery channel is empty
do V.i ≠ W.i send (V.i \ W.i) to every
outgoing channel; W.i := V.i
¬ empty (k, i) receive X from
channel(k, i); V.i := V.i X
od
V.iW.i
V.kW.k
(i,k)
Acts like a “pump”
Assume that the topology is strongly connected graph
31
Proof
Lemma. empty (i. k) W.i V.k.
(Upon termination) i: V.i = W.i,
and all channels are empty.
So, V.i V.k.
On a cyclic path, V.i = V.k must be
true. Since s(i) V.i, s(i) V.k
V.iW.i
V.kW.k
(i,k)
32
Acknowledgements
This part is heavily dependent on Dr. Sukumar Ghosh Iowa University Distributed Systems course 22C:166
33
Termination Detection and Deadlocks
35
Termination detection
During the progress of a distributed computation,processes may periodically turn active or passive.
A distributed computation termination when:
(a) every process is passive, (b) all channels are empty, and (c) the global state satisfies the desired postcondition
36
Visualizing diffusing computation
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
(a) (b) (c)
Notice how one process engages another process. Eventually all processes turn white, and no message is in transit -this signals termination. How to develop a signaling mechanism to detect termination?
passive
active
initiator
37
Dijkstra-Scholten algorithm
An initiator initiates termination detectionby sending signals (messages) down theedges via which it engages other nodes.
At a “suitable time,” the recipient sends anack back.
When the initiator receives ack from everynode that it engaged, it detects termination.
Node j engages node k.
j k
j k
j k
The basic scheme
signal
ack
38
Dijkstra-Scholten algorithm
Deficit (e) = # of signals on edge e - # of ack on edge e
For any node, C = total deficit along incoming edges
and D = total deficit along outgoing edges
For the initiator, by definition, C = 0
Dijkstra-Scholten algorithm used the following twoInvariants to develop their algorithm:
Invariant 1. (C ≥ 0) (D ≥ 0)Invariant 2. (C > 0) (D = 0)
0
1
2
4
3
5
39
Dijkstra-Scholten algorithm
The invariants must hold when an interim node sends an ack. So, acks will be sent when
(C-1 ≥ 0) (C-1 > 0 D=0) {follows from INV1 and INV2}= (C > 1) (C ≥1 D=0)= (C > 1) (C =1 D=0)
0
1
2
4
3
5
40
Dijkstra-Scholten algorithm
program detect {for an internal node i}
initially C=0, D=0, parent = i
do
- m = signal (C=0) C:=1; state:= active; parent :=
sender{this node can send out messages
to engage other nodes, or turn passive}
- m = ack D:= D-1
- (C=1 D=0) state = passive send ack to parent; C:= 0; parent := i
- m = signal (C=1) send ack to the sender;
od
0
1
2
4
3
5
Note that the engaged nodes induce a spanning tree
41
Distributed deadlock
Assume each process owns a few resources, and review how resources are allocated.
Why deadlocks occur?
- Exclusive (i.e not shared) resources- Non-preemptive scheduling- Circular waiting by all or a subset of processes
42
Distributed deadlock
Three aspects of deadlock
deadlock detection deadlock prevention deadlock recovery
43
Distributed deadlock
May occur due to bad designs/bad strategy
[Sometimes prevention is more expensive than detection and recovery. So designs may not care about deadlocks, particularly if it is rare.]
Caused by failures or perturbations in the system
44
Wait-for Graph (WFG)
Represents who waits for whom.
No single process can see the WFG.
Review how the WFG is formed.
45
Another classification
Resource deadlock
[R1 AND R2 AND R3 …] also known as AND deadlock
Communication deadlock[R1 OR R2 OR R3 …] also known as OR deadlock
46
Detection of resource deadlock
Notations
w(j) = true (j is waiting)
depend [j,i] = true j succn(i) (n>0)
P(i,s,k) is a probe (i=initiator, s= sender, r=receiver)
3
2
4
1
P(4,4,3)
initiator
47
Detection of resource deadlock
{Program for process k}
do
P(i,s,k) received w[k] (k ≠
i) ¬ depend[k, i]
send P(i,k,j) to each successor j;
depend[k, i]:= true
P(i,s, k) received w[k] (k = i)
process k is deadlocked
od
1
2 3
4
0
48
Observations
To detect deadlock, the initiator must be in a cycle
Message complexity = O(|E|)
(edge-chasing algorithm)
1
2 3
4
0
E=set of edges
Should the links be FIFO?
49
Communication deadlock
0 1 2 3 4
This has a resource deadlock but no communication deadlock
50
Detection of communication deadlock
A process ignores a probe, if it is not waiting for any process. Otherwise,
first probe mark the sender as
parent;forwards the probe to
successors
Not the first probe Send ack to that sender
ack received from every successor send ack to the parent
Communication deadlock is detected
if the initiator receives ack.
0 1 2 3 4
Has many similarities with Dijkstra-Scholten’s termination detection algorithm
51
Distributed deadlock
May occur due to faulty design or resource sharing problems
[Sometimes prevention is more expensive than detection and
recovery. So certain designs deliberately do not care about deadlocks,
particularly if it is rare.]
Sometimes failures failures or perturbations can modigy the
system state and cause deadlock.
detection prevention recovery
Major issues
52
Wait-for Graph (WFG)
Represents who waits for whom.
No single process can see the WFG.
Review how the WFG is formed.
53
Another classification
Resource deadlock
[R1 AND R2 AND R3 …]
also known as AND deadlock
Communication deadlock
[R1 OR R2 OR R3 …]
also known as OR deadlock
54
Detection of resource deadlock
Notations
w(j) = true (j is waiting)
depend [j,i] = true j succn(i) (n>0)
P(i,s,k) is a probe (i=initiator, s= sender, r=receiver)
3
2
4
1
P(4,4,3)
initiator
55
Detection of resource deadlock
{Program for process k}
do P(i,s,k) received
w[k] (k ≠ i) ¬ depend[k, i]
send P(i,k,j) to each successor
j; depend[k, i]:= true
P(i,s,k) received w[k] (k = i)
process k is deadlocked
od
1
2 3
4
0
Chandy-Misra-Haas algorithm
56
Observations
To detect deadlock, the initiator must be in a cycle
Message complexity = O(|E|)
(edge-chasing algorithm)
1
2 3
4
0
E=set of edges
57
Communication deadlock
0 1 2 3 4
The subgraph of the WFG consisting of black nodes and black edges has a resource deadlock as well as a communication deadlock. However, if we add node 5 and
the red edge (4,5) then the communication deadlock will disappear.
5
58
Detection of communication deadlock
A process ignores a probe, if it is not waiting for any process. Otherwise,
first probe mark the sender as parent;forwards the probe to successors
Not the first probe Send ack to that sender
ack received from every successor send ack to the parent
Communication deadlock is detectedif the initiator receives ack.
0 1 2 3 4
Has many similarities with Dijkstra-Scholten’s termination detection algorithm
59
Acknowledgements
This part of the slides is almost entirely dependent on Dr. Sukumar Ghosh course Iowa University Distributed Systems course 22C:166