1

Lecture notes: Concurrent And Distributed Programming

© Assaf Schuster

Spring 2004-Itay Maman-

2

Outline (1/3) Virtual Time

Happened Before Relation Cuts Lamport Clock Vector Clock

Shared memory Sequential Consistency vs. Coherence Release Consistency

Release Consistency Implementations: Dash, Munin (Eager) Lazy Release Consistency

Interval Clock Vector Garbage Collection Optimization - Lazy diffing

Home Based Lazy Release Consistency

3

Outline (2/3)

Data Races Djit+

LTF (Local time frame) Lockset

4

Happened Before Relation Partial order over events

An event is an atomic "instruction" in a program

Intuition: β < γ β is bound to precede γ in every execution

Definition: β < γ if either β, γ are on the same thread and β precedes γ β sends a message to γ exists λ such that β < λ, λ < γ

5

Cuts (1/2)

Intuition: In each thread, split the sequence of events into two subsets ("pre", "post"). Then, union all "pre" subsets.

Definition: Cut X over cut events e1,e2, .. en X(e1,e2, .. en) = {e in P1 | e <= e1} U {e in P2 | e <= e2} .. U {e in Pn | e <= en}

6

Cuts (2/2)

Definition: Consistent Cuts A cut X is consistent if for every γ in X: β < γ => β in X

7

Lamport Clock A function L: Event -> Int Definition:

β, γ are two consecutive events on a thread λ is an event on another thread λ sends a message to γ: L(γ) = Max{ L(β), L(λ) } + 1 Otherwise: L(γ) = L(β) + 1

Usage: β < γ => L(β) < L(γ) L(β) < L(γ) => not(γ < β)

However, if β || γ every outcome is possible: L(β) < L(γ), or L(β) = L(γ), or L(β) > L(γ)

8

Vector Clock (1/3)

Intuition Each thread maintains a local vector

of approximated timestamps of all other threads.

When a message is received, the local vector is max-ed with the sender's vector

9

Vector Clock (2/3) Definition: Vector Clock function C: Event ->Intn

β, γ are two consecutive events on a thread j. λ is an event on another thread. Sj is a vector with 1 on the j-th position (rest is zeros). λ sends a message to γ: C(γ)=Max{ C(β), C(λ) } + Sj

Otherwise: C(γ) = C(β) + Sj

CP(m) is the current local vector of thread m. Invariant (for every k, j):

CP(j)[j] >= CP(k)[j], at any point in time.

10

Vector Clocks (3/3)

Theorems A cut X(e1, e2, .., en) is consistent max{ C(e1), C(e2), .., C(en) } = { CP(1)[1], CP(2)[2], .., CP(n)[n] } =>Different consistent cuts have different

timestamps C(β) < C(γ) β < γ C(β) || C(γ) β || γ C(β)[j] <= C(γ)[j] β < γ (where Pj is the thread of β)

11

Memory Models The model: Many threads, A single shared

memory Operations: r(x) => Read x, w(x,n) => Put n in

x All variables are initialized to 0

In reality: Many threads each having a local memory.

The memory model specifications determines how/when one thread "sees" the put operations of other threads

12

Memory Models - definitions:

Program order: The sequence of instructions in each thread.

Serialization: A "merge" of all program orders. Similar to the merging of two decks of

cards Legal Serialization:

r(x) = n Last put to x was w(x,n)

13

Sequential Consistency Vs. Coherence

Sequential Consistency (SC) Execution of a program is a legal serialization

Coherence P is a given execution (serialization) Let sub(p,v) denote the sub execution of P

having only r(v), w(v,n) operations. Coherence ensures that sub(p,v) is a legal

serialization for each variable v in the program p

14

Release Consistency (1/2)

Assumes the existence of Locks in the program acquire(l), release(l) The lock variables are Sequentially

Consistent Intuition: release(l) flushes all put

operations taking place since last acquire(l)

15

Release Consistency (2/2)

In the absence of release/acquire pairs, put operations may never propagate If Data Races exist we may "see" values in

an undefined order Theorem: A data race free program is

sequentially consistent under a Release Consistency Model

16

Release Consistency Implementations - Dash

DASH (original impl.) put(v,n) is immediately sent

(asynchronously) to all threads release(l) blocks until all previous put

requests have completed

17

Release Consistency Implementations - Munin (1/2)

Munin (AKA: Eager Release Consistency) All put operations are saved in a local

buffer release(l)

Eliminates duplications Sends Notification to all other threads of the

local modifications Waits till all threads have acknowledged.

18

Release Consistency Implementations - Munin(2/2)

Twinning & Diffing: The notification creation algorithm A Local "clean" copy (Twin) is created upon first put

to each page release: compare each page with its twin, create a

notification entry for each modified variable Two different notification strategies

Send actual "new" value- or - Send an invalidation message for the modified

variables (receiver will have to refetch on first read)

19

Lazy Release Consistency Yet another memory Model

A More relaxed version of Release Consistency Intuition: A thread needs to be notified only

when it acquires a lock Principles

Notifications are lazily sent: only to threads which actually acquire the released lock (possibly transitively)

Lock variables are sequentially consistent Uses the Invalidation strategy Actually

needed ??

20

Lazy Release Consistency Implementation (1/2)

Each thread maintains a Clock Vector A new interval begins at each synchronization (acq/rel). Thread Pj increments its position j each time it starts a

new interval. Local vector of acquirer is max-ed with vector of

releaser. VP(j,n) denotes the local vector of thread Pj when it

executed its n-th interval Invariant: VP(j,n)[j] = n Interval n of Pj covers interval x of Pk

if VP(j,n)[k] >= x

21

Lazy Release Consistency Implementation (2/2)

Each thread maintains a table of intervals One entry per interval Each entry lists all write notices + all diffs,

"produced" by the interval (including those accepted from other threads)

Acquire protocol:1. Acquirer attaches its local vector to the acquire

request 2. Releaser sends back all notices which belong to

intervals that are not covered by the acquirer.3. Acquirer invalidates all notified variables + Copies

accepted notices to the new interval's entry.

22

Lazy Release Consistency: Garbage Collection

A diff cannot be retained forever if a diff was already sent to all other

threads it can be discarded If a thread is running out of memory =>

"Stop the world" (All threads stop) All write notices are sent to everyone Now, All local diffs/notices can be discarded

23

Lazy Release Consistency: Optimization - Lazy diffing

A diff is not automatically computed The diff is created on the first time a

thread has asked for it Till then the Twin page must be kept

24

Home-based Lazy Release Consistency (HLRC)

Intuition: Each diff is applied only once. Each page has a designated owner (home)

Acquire protocol:1. Acquirer sends an acquire request2. Releaser sends the diff to the page's home 3. Home updates its "master" copy of the page +

sends acknowledgement back4. Releaser sends an invalidation message to the

acquirer

25

HLRC: Guaranteeing update completion (1/2)

The Challenge: the home thread must apply the updated diff before other threads can retrieve it

Solution 1: Flushing- acquirer waits till the home finished "digesting" the diff Problem 1.1: May cause the same page to be

fetched twice from the home (caused by two distinct notifications)

Solution 1.1: Each page has a version number. No fetching is carried out if local version matches the latest version (at the home)

26

HLRC: Guaranteeing update completion (2/2)

Solution 2: Clock Vectors Solves the problem of "waiting" on each acquire Each diff is sent with its clock vector Current vector is attached to each "page-fetch"

request The home waits till it receives a diff whose vector is

at least "as-recent-as" the request vector Summary:

Acquire does not block Fetch may block

27

Djit+ Intuition: Trace execution. Check clock

vector of each access against previously logged accesses

Pros: No false alarms No missed races (in the given scheduling)

Cons: Cannot prove a program is data-race free Requires lots of runs High Sensitivity to scheduling

28

Djit+: Local Time Frame (LTF)

An LTF is a scalar value Each release operation increments LTF

of the current thread (Interval starts) Each thread maintains an LTF vector Acquirer's vector is max-ed with

releaser's

29

Djit+ Algorithm

1st read and 1st write in each interval are recorded (per-thread)

A read operation is checked against all recorded writes A write operation is checked against all recorded reads

+ writes Checking predicate:

CL denotes the current LTF vector held by the "current" thread e denotes a previously recorded entry predicate(e,CL) = e.ltf ≥ CL[e.thread_id] TRUE => A data race

30

Lockset Intuition: Trace execution. Look for

violations of common locking schemes Pros:

Less sensitive to scheduling No missed races (in the given scheduling)

Cons: Cannot prove a program is data-race free Lots of false alarms Still scheduling depended

31

Lockset Algorithm

for each variable v init c(v) with { all-locks } On each access to a variable v

c(v) = c(v) ∩ { currently held locks } if c(v) = { } issue a race warning

Conflicting practices: Initialization: Often not protected Read Only Variable: No protection if no one writes

to it Multiple readers-single writer

32

MPI

Basic functions

MPI_Init(argc, argv);MPI_Finalize();MPI_Comm_rank(MPI_COMM_WORLD, &rank);MPI_Comm_size(MPI_COMM_WORLD, &count);MPI_Send(buf, n, MPI_INT, to, TAG1, MPI_COMM_WORLD);MPI_Recv(buf, n, MPI_INT, from, TAG1, MPI_COMM_WORLD, &st);

33

MPI Blocking Send

MPI_Send() - Wait till last byte sent size < threshold => Transmission starts on spot otherwise => Transmission starts when MPI_Recv() is

placed MPI_SSend() - Wait till MPI_Recv() is placed MPI_RSend() - Failure if no MPI_Recv() is pending

Failure = Program stops MPI_BSend() - Wait till the data is copied to a

client supplied buffer

34

MPI

Non Blocking Send MPI_ISend() - Returns immediately

size > threshold => Wait till MPI_IRecv() is placed

MPI_Request r; MPI_Status s; int f; // Flag

MPI_Isend(buf, n, MPI_INT, to, TAG1, MPI_COMM_WORLD, &r);MPI_Wait(&r, &s); // Wait till a request completesMPI_Test(&r, &f, &s); // Check status of a requestMPI_Probe(from, TAG1, MPI_COMM_WORLD, &s); // Wait for a message

35

Questions

Release Consistency vs. Coherence. Which model is weaker?

Why Lazy Release Consistency was preferred over HLRC (seems simpler, quicker)