![Page 1: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/1.jpg)
Real And Logical ClocksLecture 7
![Page 2: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/2.jpg)
Coordination and Synchronization• How processes can coordinate and synchronize with each other • Process synchronization:
• Wait for another process to complete some action
• Data synchronization• 2 sets of data should be the same
![Page 3: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/3.jpg)
Time in a conventional OS• Processes can get “system time” via systemcalls
• Such as gettimeofday() in UNIX
• Time-stamps can be used for ordering and coordination• Example: make uses file modified time to decide what actions to run • If the OS is the only time-source, then all processes observe the same time• Is it possible to synchronize all clocks in a distributed system?
![Page 4: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/4.jpg)
Physical Clocks and UTC • Local time-keeping: crystal oscillator inside CPUs triggers timer interrupt in the
OS • Multiple CPUs difference in clock values • Temperature (and other factors) also affect frequency clock skew• Standardization: Universal Coordinated Time (UTC)• Based on atomic clocks • UTC “time” broadcast via radios, satellites, and through phone-numbers
• NIST +1-808-335-4363
• Accuracy is around 0.5ms
![Page 5: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/5.jpg)
Clock Synchronization • Precision: Keep deviation between two clocks on any two machines within a
specified bound
• Accuracy: The difference between all clocks and UTC time is less than \alpha
• Internal synchronization: Keep clocks precise• External synchronization: Keep clocks accurate
![Page 6: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/6.jpg)
NTP: Network Time Protocol • Server B is a time-server with an accurate clock (say, atomic)
• Server A wishes to get it’s clock synchronized with B’s clock • By periodically polling B
• A asks B for the time • But messages face network delays!• Collect 8 pairs of (\theta,\delta) and
choose \theta with lowest \delta • Never set a clock backwards!• Adjust rate of clock-ticks to slow-
down or catch-up to A’s time • Hierarchy of NTP servers
![Page 7: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/7.jpg)
Keeping Time Without UTC• NTP allows for external synchronization with UTC time• Sometimes, internal consistency suffices • Time server polls all other machines, computes average time, and broadcasts it
![Page 8: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/8.jpg)
Logical Clocks
![Page 9: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/9.jpg)
![Page 10: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/10.jpg)
Happened Before Relationship• First fundamental result in distributed computing • Partial order among events
• Process-order, send-receive order, transitivity
• Important that all processes agree on the order of events
![Page 11: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/11.jpg)
Ordering Of Events• Happened-before relation:
1. If a and b are two events that occur in the same process, and a comes before b, then a b
2. If a corresponds to sending a msg, and b is the receipt of that message, then ab
3. Transitive: ab and bc, then abc
• This is a partial ordering of events in a system
![Page 12: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/12.jpg)
Ordering Of Events Example
Partial order Some events will be concurrent
![Page 13: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/13.jpg)
Logical Clocks• How to maintain a global view of system behavior that is consistent with
the happened-before relation? • Approach: assign a timestamp C(e) to each event e, such that:
• If ab, then C(a) < C(b)• C must be monotonically increasing
• How to attach a timestamp to an event when there’s no global clock?• Maintain a consistent set of logical clocks, one per process
![Page 14: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/14.jpg)
Lamport Clocks• Each process 𝑃 maintains a local counter 𝐶 and adjusts this counter 1. New local event, increment 𝐶2. Send a timestamp 𝐶 with each message sent by 𝑃 (i.e., ts(m) = 𝐶 )3. Whenever a message is received by 𝑃 , 𝑃 adjusts its local counter
𝐶 = max{𝐶 ts(m)} . Since this is a new event, increment 𝐶
![Page 15: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/15.jpg)
Lamport Clock Example
![Page 16: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/16.jpg)
Total Ordering of Time-stamps
• It’s possible for 𝐶 (𝑒 ) == 𝐶 (𝑒 )
• I.e., Lamport clock timestamps are not totally ordered• We can break ties based on process-id • For process P_i, the clock value becomes 𝐶 . 𝑖
• For example: 3.2 for process-id==2
![Page 17: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/17.jpg)
Example: Total-ordered Multicast• Two replicas of database• P1 adds $100 to an account (initial: $1000)• P2 increments account by 1%• Replica-1: $1111• Replica-2: $1110
![Page 18: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/18.jpg)
Totally Ordered Multicast• Need to ensure that two update operations are performed in the same order
by all nodes• Actual order of operations is immaterial (add first or interest first) • Totally ordered multicast: All messages are delivered in the same order to each
receiver • Assumption: No failures, and FIFO delivery of messages
![Page 19: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/19.jpg)
Totally Ordered Multicast Algorithm• All messages timestamped with sender’s logical time • Sender sends to all recipients, including itself• When a message is received:
• 1. It is put into a local queue • 2. Queue is ordered based on timestamp • 3. The acknowledgement is multicast (with the receiver’s logical time ofcourse)
![Page 20: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/20.jpg)
Totally Ordered Multicast Algorithm, continued • Message is delivered to application only when:
1. It is at the head of the queue2. All the acknowledgement for that message have been received
![Page 21: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/21.jpg)
Example: Totally Ordered Multicast
San Francisco (P1)1.12.1
New York (P2)
1.22.23.2
Issue mSend m
Recv n
Issue nSend nRecv m3.1
![Page 22: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/22.jpg)
Example: Totally Ordered Multicast• The sending of message m consists of sending the update operation and the
time of issue which is 1.1• The sending of message n consists of sending the update operation and the time
of issue which is 1.2 • Messages are multicast to all processes in the group including itself.
• Assume that a message sent by a process to itself is received by the process almost immediately.• For other processes, there may be a delay.
![Page 23: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/23.jpg)
Example: Totally Ordered Multicast• At this point, the queues have the following:
• P1: (m,1.1), (n,1.2)• P2: (m,1.1), (n,1.2)
• P1 will multicast an acknowledgement for (m,1.1) but not (n,1.2). • Why? P1’s identifier is higher then P2’s identifier and P1 has issued a request• 1.1 < 1.2
• P2 will multicast an acknowledgement for (m,1.1) and (n,1.2)• Why? P2’s identifier is not higher then P1’s identifier • 1.1 < 1.2
![Page 24: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/24.jpg)
Example: Totally Ordered Multicast
• P1 does not issue an acknowledgement for (n,1.2) until operation m has been processed.• 1< 2
• Note: The actual receiving by P1 of message (n,1.2) is assigned a timestamp of 3.1.
• Note: The actual receiving by P2 of message (m,1.1) is assigned a timestamp of 3.2
![Page 25: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/25.jpg)
Example: Totally Ordered Multicast• If P2 gets (n,1.2) before (m,1.1) does it still multicast an acknowledgement for
(n,1.2)? • Yes!
• At this point, how does P2 know that there are other updates that should be done ahead of the one it issued?• It doesn’t;• It does not proceed to do the update specified in (n,1.2) until it gets an acknowledgement from all other
processes which in this case means P1.
• Does P2 multicast an acknowledgement for (m,1.1) when it receives it? • Yes, it does since 1 < 2
![Page 26: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/26.jpg)
Example: Totally Ordered Multicast
San Francisco (P1)1.12.1
3.1
5.1
New York (P2)
1.22.23.24.2
Issue mSend m
Recv n
Issue nSend nRecv mSend ack(m)
Recv ack(m)
Note: The figure does not show a process sending a message to itself orthe multicast acks that it sends for the updates it issues.
![Page 27: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/27.jpg)
Example: Totally Ordered Multicast• To summarize, the following messages have been sent:
• P1 and P2 have issued update operations.• P1 has multicasted an acknowledgement message for (m,1.1).• P2 has multicasted acknowledgement messages for (m,1.1), (n,1.2).
• P1 and P2 have received an acknowledgement message from all processes for (m,1.1).
• Hence, the update represented by m can proceed in both P1 and P2.
![Page 28: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/28.jpg)
Example: Totally Ordered Multicast
San Francisco (P1)1.12.1
3.1
5.1
New York (P2)
1.22.23.24.2
Issue mSend m
Recv n
Issue nSend nRecv mSend ack(m)
Recv ack(m)Process m
Note: The figure does not show the sending of messages it oneself
Process m
![Page 29: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/29.jpg)
Example: Totally Ordered Multicast• When P1 has finished with m, it can then proceed to multicast an
acknowledgement for (n,1.2). • When P1 and P2 both have received this acknowledgement, then it is the case
that acknowledgements from all processes have been received for (n,1.2). • At this point, it is known that the update represented by n can proceed in both
P1 and P2.
![Page 30: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/30.jpg)
Example: Totally Ordered Multicast
San Francisco (P1)1.12.1
3.1
5.1
New York (P2)
1.22.23.24.2
Issue mSend m
Recv n
Issue nSend nRecv mSend ack(m)
6.1Send ack(n)
Recv ack(m)
7.2 Recv ack(n)
Process m
Process nProcess n
Process m
![Page 31: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/31.jpg)
Example: Totally Ordered Multicast• What if there was a third process e.g., P3 that issued an update (call it o) at
about the same time as P1 and P2.• The algorithm works as before.
• P1 will not multicast an acknowledgement for o until m has been done.• P2 will not multicast an acknowledgement for o until n has been done.
• Since an operation can’t proceed until acknowledgements for all processes have been received, o will not proceed until n and m have finished.
![Page 32: Real And Logical Clockshomes.sice.indiana.edu/prateeks/ds/Lec7.pdfThe acknowledgement is multicast (with the receiver’s logical time ofcourse) Totally Ordered Multicast Algorithm,](https://reader035.vdocuments.net/reader035/viewer/2022071215/60462100f817f2006f52de93/html5/thumbnails/32.jpg)
State Machine Replication• Totally ordered multicast enables state machine replication• Servers can be thought of deterministic state machines that change state based
on the messages they receive. • Replicating state machines has many benefits: fault-tolerance, performance..
• If a server crashes, then contact other replicas
• With totally ordered multicast, all servers can execute the same operations in the same order
• Totally-ordered multicast is the “holy grail” of distributed computing • Yet we could somehow achieve it? Or did we??