linked lists: locking, lock-free, and beyond … the art of multiprocessor programming spring 2007

Linked Lists: Locking, Lock-Free,

and Beyond …The Art of

Multiprocessor Programming

Spring 2007

© Herlihy-Shavit 2007 2

Last Lecture: Spin-Locks

CS

Resets lock upon exit

spin lock

critical section

...

© Herlihy-Shavit 2007 3

Today: Concurrent Objects

• AddingThreads • Should not lower throughput

– Contention effects– Mostly fixed by Queue locks

• Should increase throughput– Not possible if inherently sequential– Surprising things are parallelizable

© Herlihy-Shavit 2007 4

Coarse-Grained Synchronization

• Each method locks the object– Avoid contention using queue locks – Easy to reason about

• In simple cases

– Standard Java model• Synchronized blocks and methods

• So, are we done?

© Herlihy-Shavit 2007 5

Coarse-Grained Synchronization

• Sequential bottleneck– All threads “stand in line”

• Adding more threads– Does not improve throughput– Struggle to keep it from getting worse

• So why even use a multiprocessor?– Well, some apps inherently parallel …

© Herlihy-Shavit 2007 6

This Lecture

• Introduce four “patterns”– Bag of tricks …– Methods that work more than once …

• For highly-concurrent objects• Goal:

– Concurrent access– More threads, more throughput

© Herlihy-Shavit 2007 7

First:Fine-Grained Synchronization• Instead of using a single lock ..• Split object into

– Independently-synchronized components

• Methods conflict when they access– The same component …– At the same time

© Herlihy-Shavit 2007 8

Second:Optimistic Synchronization

• Object = linked set of components• Search without locking …• If you find it, lock and check …

– OK, we are done– Oops, try again

• Evaluation– cheaper than locking– mistakes are expensive

© Herlihy-Shavit 2007 9

Third:Lazy Synchronization

• Postpone hard work• Removing components is tricky

– Logical removal• Mark component to be deleted

– Physical removal• Do what needs to be done

© Herlihy-Shavit 2007 10

Fourth:Lock-Free Synchronization

• Don’t use locks at all– Use compareAndSet() & relatives …

• Advantages– Robust against asynchrony

• Disadvantages– Complex– Sometimes high overhead

© Herlihy-Shavit 2007 11

Linked List

• Illustrate these patterns …• Using a list-based Set

– Common application– Building block for other apps

© Herlihy-Shavit 2007 12

The Evolution of Concurrent List-based Sets

Course Grained

Fine Lock-Coupling

Course Optimistic

FineOptim-istic

Lock-freeLazy

FineOptim-istic

© Herlihy-Shavit 2007 13

Set Interface

• Unordered collection of objects• No duplicates• Methods

– Add() a new object– Remove() an object– Test if set Contains() object

© Herlihy-Shavit 2007 14

List-Based Sets

public interface Set { public boolean add(Object x); public boolean remove(Object x); public boolean contains(Object x);}

© Herlihy-Shavit 2007 15

List-Based Sets

public interface Set { public boolean add(Object x); public boolean remove(Object x); public boolean contains(Object x);}

Add object to set

© Herlihy-Shavit 2007 16

List-Based Sets

public interface Set { public boolean add(Object x); public boolean remove(Object x); public boolean contains(Object x);}

Remove object from set

© Herlihy-Shavit 2007 17

List-Based Sets

public interface Set { public boolean add(Object x); public boolean remove(Object x); public boolean contains(Object x);}

Is object in set?

© Herlihy-Shavit 2007 18

List Node

public class Node { public Object object; public int key; public Node next;}

© Herlihy-Shavit 2007 19

List Node

public class Node { public Object object; public int key; public Node next;}

Object of interest

© Herlihy-Shavit 2007 20

List Node

public class Node { public Object object; public int key; public Node next;}

Usually hash code

© Herlihy-Shavit 2007 21

List Node

public class Node { public Object object; public int key; public Node next;}

Reference to next node

© Herlihy-Shavit 2007 22

The List-Based Set

a b c

Ordered + Sentinel nodes(min & max possible keys)

© Herlihy-Shavit 2007 23

Reasoning about Concurrent Objects

• Invariant– Property that always holds

• Established by– True when object is created– Truth preserved by each method

• Each step of each method

© Herlihy-Shavit 2007 24

Specifically …

• Invariants preserved by– add()– remove()– contains()

• Most steps are trivial– Usually one step tricky– Often linearization point

© Herlihy-Shavit 2007 25

Interference

• Proof that invariants preserved works only if– methods considered– are the only modifiers

• Language encapsulation helps– List nodes not visible outside class

© Herlihy-Shavit 2007 26

Interference

• Freedom from interference neeeded even for removed nodes– Some algorithms traverse removed

nodes– Careful with malloc() & free()!

• Garbage-collection helps here

© Herlihy-Shavit 2007 27

Abstract Data Types

• Concrete representation

• Abstract Type– {a, b}

a b

© Herlihy-Shavit 2007 28

Abstract Data Types

• Meaning of rep given by abstraction map

– S( ) = {a,b}a b

© Herlihy-Shavit 2007 29

Rep Invariant

• Which concrete values are meaningful?– Sorted? Duplicates?

• Rep invariant– Characterizes legal concrete reps– Preserved by methods– Relied on by methods

© Herlihy-Shavit 2007 30

Blame Game

• Rep invariant is a contract• Suppose– add() leaves behind 2 copies of x– remove() removes only 1

• Which one is incorrect?

© Herlihy-Shavit 2007 31

Blame Game

• Suppose– add() leaves behind 2 copies of x– remove() removes only 1

• Which one is incorrect?– If rep invariant says no duplicates• add() is incorrect

– Otherwise• remove() is incorrect

© Herlihy-Shavit 2007 32

Rep Invariant (partly)

• Sentinel nodes– tail reachable from head

• Sorted, no duplicates

© Herlihy-Shavit 2007 33

Abstraction Map

• S(head) =– { x | there exists a such that

•A reachable from head and•a.object = x

– }

© Herlihy-Shavit 2007 34

Sequential List Based Set

a c d

b

a b c

Add()

Remove()

© Herlihy-Shavit 2007 35

Course Grained Locking

a b d

c

Simple but hotspot + bottleneck

© Herlihy-Shavit 2007 36

Coarse-Grained Locking

• Easy, same as synchronized methods

– “One lock to rule them all …”

• Simple, clearly correct– Deserves respect!

• Works poorly with contention– Queue locks help– But bottleneck still an issue

© Herlihy-Shavit 2007 37

Fine-grained Locking

• Requires careful thought– “Do not meddle in the affairs of

wizards, for they are subtle and quick to anger”

• Split object into pieces– Each piece has own lock– Methods that work on disjoint pieces

need not exclude each other

© Herlihy-Shavit 2007 38

Hand-over-Hand locking

a b c

© Herlihy-Shavit 2007 39

Hand-over-Hand locking

a b c

© Herlihy-Shavit 2007 40

Hand-over-Hand locking

a b c

© Herlihy-Shavit 2007 41

Hand-over-Hand locking

a b c

© Herlihy-Shavit 2007 42

Hand-over-Hand locking

a b c

© Herlihy-Shavit 2007 43

Removing a Node

a b c d

remove b

© Herlihy-Shavit 2007 44

Removing a Node

a b c d

remove b

© Herlihy-Shavit 2007 45

Removing a Node

a b c d

remove b

© Herlihy-Shavit 2007 46

Removing a Node

a b c d

remove b

© Herlihy-Shavit 2007 47

Removing a Node

a c d

remove b

© Herlihy-Shavit 2007 48

Removing a Node

a b c d

remove b

remove c

© Herlihy-Shavit 2007 49

Removing a Node

a b c d

remove b

remove c

© Herlihy-Shavit 2007 50

Removing a Node

a b c d

remove b

remove c

© Herlihy-Shavit 2007 51

Removing a Node

a b c d

remove b

remove c

© Herlihy-Shavit 2007 52

Removing a Node

a b c d

remove b

remove c

© Herlihy-Shavit 2007 53

Removing a Node

a b c d

remove b

remove c

© Herlihy-Shavit 2007 54

Removing a Node

a b c d

remove b

remove c

© Herlihy-Shavit 2007 55

Removing a Node

a b c d

remove b

remove c

© Herlihy-Shavit 2007 56

Removing a Node

a b c d

remove b

remove c

© Herlihy-Shavit 2007 57

Uh, Oh

a c d

remove b

remove c

Still in list

© Herlihy-Shavit 2007 58

Problem

• To delete node b– Swing node a’s next field to c

• Problem is,– Someone could delete c concurrently

ba c

ba c

© Herlihy-Shavit 2007 59

Insight

• If a node is locked– No one can delete node’s successor

• If a thread locks– Node to be deleted– And its predecessor– Then it works

© Herlihy-Shavit 2007 60

Hand-Over-Hand Again

a b c d

remove b

© Herlihy-Shavit 2007 61

Hand-Over-Hand Again

a b c d

remove b

© Herlihy-Shavit 2007 62

Hand-Over-Hand Again

a b c d

remove b

© Herlihy-Shavit 2007 63

Hand-Over-Hand Again

a b c d

remove b

Found it!

© Herlihy-Shavit 2007 64

Hand-Over-Hand Again

a b c d

remove b

Found it!

© Herlihy-Shavit 2007 65

Hand-Over-Hand Again

a c d

remove b

Found it!

© Herlihy-Shavit 2007 66

Removing a Node

a b c d

remove b

remove c

© Herlihy-Shavit 2007 67

Removing a Node

a b c d

remove b

remove c

© Herlihy-Shavit 2007 68

Removing a Node

a b c d

remove b

remove c

© Herlihy-Shavit 2007 69

Removing a Node

a b c d

remove b

remove c

© Herlihy-Shavit 2007 70

Removing a Node

a b c d

remove b

remove c

© Herlihy-Shavit 2007 71

Removing a Node

a b c d

remove b

remove c

© Herlihy-Shavit 2007 72

Removing a Node

a b c d

remove b

remove c

© Herlihy-Shavit 2007 73

Removing a Node

a b d

remove b

remove c

© Herlihy-Shavit 2007 74

Removing a Node

a b d

remove b

remove c

© Herlihy-Shavit 2007 75

Removing a Node

a d

remove b

remove c

© Herlihy-Shavit 2007 76

Remove method

public boolean remove(Object object) { int key = object.hashCode(); Node pred, curr; try { … } finally { curr.unlock(); pred.unlock(); }}

© Herlihy-Shavit 2007 77

Remove method

public boolean remove(Object object) { int key = object.hashCode(); Node pred, curr; try { … } finally { curr.unlock(); pred.unlock(); }}

Key used to order node

© Herlihy-Shavit 2007 78

Remove method

public boolean remove(Object object) { int key = object.hashCode(); Node pred, curr; try { … } finally { currNode.unlock(); predNode.unlock(); }}

Predecessor and current nodes

© Herlihy-Shavit 2007 79

Remove method

public boolean remove(Object object) { int key = object.hashCode(); Node pred, curr; try { … } finally { curr.unlock(); pred.unlock(); }}

Make sure locks

released

© Herlihy-Shavit 2007 80

Remove method

public boolean remove(Object object) { int key = object.hashCode(); Node pred, curr; try { … } finally { curr.unlock(); pred.unlock(); }}

Everything else

© Herlihy-Shavit 2007 81

Remove method

try { pred = this.head; pred.lock(); curr = pred.next; curr.lock(); …} finally { … }

© Herlihy-Shavit 2007 82

Remove method

try { pred = this.head; pred.lock(); curr = pred.next; curr.lock(); …} finally { … }

lock previous

© Herlihy-Shavit 2007 83

Remove method

try { pred = this.head; pred.lock(); curr = pred.next; curr.lock(); …} finally { … }

Lock current

© Herlihy-Shavit 2007 84

Remove method

try { pred = this.head; pred.lock(); curr = pred.next; curr.lock(); …} finally { … }

Traversing list

© Herlihy-Shavit 2007 85

Remove: searching

while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

© Herlihy-Shavit 2007 86

Remove: searching

while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

Search key range

© Herlihy-Shavit 2007 87

Remove: searching

while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

At start of each loop: curr and pred locked

© Herlihy-Shavit 2007 88

Remove: searching

while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;If node found, remove

it

© Herlihy-Shavit 2007 89

Remove: searching

while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;If node found, remove

it

© Herlihy-Shavit 2007 90

Remove: searching

while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

Unlock predecessor

© Herlihy-Shavit 2007 91

Remove: searching

while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

Only one node locked!

© Herlihy-Shavit 2007 92

Remove: searching

while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

demote current

© Herlihy-Shavit 2007 93

Remove: searching

while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = currNode; curr = curr.next; curr.lock(); } return false;

Find and lock new current

© Herlihy-Shavit 2007 94

Remove: searching

while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = currNode; curr = curr.next; curr.lock(); } return false;

Lock invariant restored

© Herlihy-Shavit 2007 95

Remove: searching

while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

Otherwise, not present

© Herlihy-Shavit 2007 97

while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

Why remove() is linearizable

head ~> predA --> currA

so the object is in the set

© Herlihy-Shavit 2007 98

while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

Why remove() is linearizable

Node locked, so no other thread can remove it ….

© Herlihy-Shavit 2007 99

while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

Why remove() is linearizable

Linearization point

© Herlihy-Shavit 2007 100

while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

Why remove() is linearizable

Object not present

© Herlihy-Shavit 2007 101

while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

Why remove() is linearizable

predA --> currA

predA.key < keykey < currA.key

© Herlihy-Shavit 2007 102

while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

Why remove() is linearizable

Linearization point: when currA set to node

with higher key

© Herlihy-Shavit 2007 103

Adding Nodes

• To add node e– Must lock predecessor– Must lock successor

• Neither can be deleted– (Is successor lock actually required?)

© Herlihy-Shavit 2007 104

Same Abstraction Map

• S(head) =– { x | there exists a such that

•A reachable from head and•a.object = x

– }

© Herlihy-Shavit 2007 105

Rep Invariant

• Easy to check that– Tail always reachable from head– Nodes sorted, no duplicates

© Herlihy-Shavit 2007 106

Drawbacks

• Better than coarse-grained lock– Threads can traverse in parallel

• Still not ideal– Long chain of acquire/release– Inefficient

© Herlihy-Shavit 2007 107

Optimistic Synchronization

• Find nodes without locking• Lock nodes• Check that everything is OK

© Herlihy-Shavit 2007 108

Optimistic Fine Grained

b d e

To insert C:• Optimistically traverse

list to find b• Lock b.curr then lock

b.succ• Re-Traverse list to find

b and verify predcurr

• Perform insert and release locks

a

13

32 2

c4

© Herlihy-Shavit 2007 109

Same Abstraction Map

• S(head) =– { x | there exists a such that

•A reachable from head and•a.object = x

– }

© Herlihy-Shavit 2007 110

Preserving Invariants

• Not same as before as we traverse deleted elements

• But we establish properties by– Validation– After we lock target nodes

© Herlihy-Shavit 2007 111

Scanning Deleted Items

a

b d

e

remove e

A

Pred and curr could point to removed nodes

© Herlihy-Shavit 2007 112

Removing a Node

a b c d

remove c

return true

© Herlihy-Shavit 2007 113

What Can Go Wrong?

a b c d

remove c

remove b

© Herlihy-Shavit 2007 114

Check that Node is Still Accessible

a b c d

remove c

© Herlihy-Shavit 2007 115

What Can Go Wrong?

a b c d

remove c

Add b’

b’

© Herlihy-Shavit 2007 116

What Can Go Wrong?

a b c d

remove c

b’

© Herlihy-Shavit 2007 117

Check that Nodes Still Adjacent

a b c d

remove c

© Herlihy-Shavit 2007 118

Correctness

• If– Nodes b and c both locked– Node b still accessible– Node c still successor to b

• Then– Neither will be deleted– OK to delete and return true

© Herlihy-Shavit 2007 119

Removing an Absent Node

a b d e

remove c

return false

© Herlihy-Shavit 2007 120

Correctness

• If– Nodes b and d both locked– Node b still accessible– Node d still successor to b

• Then– Neither will be deleted– No thread can add c after b– OK to return false

© Herlihy-Shavit 2007 121

Validationprivate boolean validate(Node pred, Node curry) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false;}

© Herlihy-Shavit 2007 122

private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false;}

Validation

Predecessor & current

nodes

© Herlihy-Shavit 2007 123

private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false;}

Validation

Start at the beginning

© Herlihy-Shavit 2007 124

private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false;}

Validation

Search range of keys

© Herlihy-Shavit 2007 125

private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false;}

Validation

Predecessor reachable

© Herlihy-Shavit 2007 126

private boolean validate(Node pred, Node curry) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false;}

Validation

Is current node next?

© Herlihy-Shavit 2007 127

private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false;}

Validation

Otherwise move on

© Herlihy-Shavit 2007 128

private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false;}

ValidationPredecessor not

reachable

© Herlihy-Shavit 2007 129

Remove: searchingpublic boolean remove(Object object) { int key = object.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (object == curr.object) break; pred = curr; curr = curr.next; } …

© Herlihy-Shavit 2007 130

public boolean remove(Object object) { int key = object.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (object == curr.object) break; pred = curr; curr = curr.next; } …

Remove: searching

Search key

© Herlihy-Shavit 2007 131

public boolean remove(Object object) { int key = object.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (object == curr.object) break; pred = curr; curr = curr.next; } …

Remove: searching

Retry on synchronization conflict

© Herlihy-Shavit 2007 132

public boolean remove(Object object) { int key = object.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (object == curr.object) break; pred = curr; curr = curr.next; } …

Remove: searching

Examine predecessor and current nodes

© Herlihy-Shavit 2007 133

public boolean remove(Object object) { int key = object.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (object == curr.object) break; pred = curr; curr = curr.next; } …

Remove: searching

Search by key

© Herlihy-Shavit 2007 134

public boolean remove(Object object) { int key = object.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (object == curr.object) break; pred = curr; curr = curr.next; } …

Remove: searching

Stop if we find object

© Herlihy-Shavit 2007 135

public boolean remove(Object object) { int key = object.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (object == curr.object) break; pred = curr; curr = curr.next; } …

Remove: searching

Move along

© Herlihy-Shavit 2007 136

On Exit from Loop

• If object is present– curr holds object– pred just before curr

• If object is absent– curr has first higher key– pred just before curr

• Assuming no synchronization problems

© Herlihy-Shavit 2007 137

Remove Methodtry { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.object == object) { pred.next = curr.next; return true; } else { return false; }}} finally {

pred.unlock();curr.unlock();

}}}

© Herlihy-Shavit 2007 138

try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.object == object) { pred.next = curr.next; return true; } else { return false; }}} finally {

pred.unlock();curr.unlock();

}}}

Remove Method

Always unlock

© Herlihy-Shavit 2007 139

try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.object == object) { pred.next = curr.next; return true; } else { return false; }}} finally {

pred.unlock();curr.unlock();

}}}

Remove Method

Lock both nodes

© Herlihy-Shavit 2007 140

try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.object == object) { pred.next = curr.next; return true; } else { return false; }}} finally {

pred.unlock();curr.unlock();

}}}

Remove Method

Check for synchronization

conflicts

© Herlihy-Shavit 2007 141

try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.object == object) { pred.next = curr.next; return true; } else { return false; }}} finally {

pred.unlock();curr.unlock();

}}}

Remove Method

Object found, remove node

© Herlihy-Shavit 2007 142

try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.object == object) { pred.next = curr.next; return true; } else { return false; }}} finally {

pred.unlock();curr.unlock();

}}}

Remove Method

Object not found

© Herlihy-Shavit 2007 143

Summary: Wait-free Traversals

b c da

1. Traverse removed nodes2. Must have non-interference3. Natural in GC languages like

Java™

Reachable =in the set

© Herlihy-Shavit 2007 144

Summary Optimistic List

b c ea

1. Limited Hotspots (Only at locked Add(), Remove(), Find() destination locations, not traversals)

2. But two traversals3. Yet traversals are wait-free!

© Herlihy-Shavit 2007 145

So Far, So Good

• Much less lock acquisition/release– Performance– Concurrency

• Problems– Need to traverse list twice– contains() method acquires locks

• Most common method call

© Herlihy-Shavit 2007 146

Evaluation

• Optimistic works if– cost of scanning twice without locks

<– cost of Scanning once with locks

• Drawback– Contains() acquires locks– 90% of calls in many apps

© Herlihy-Shavit 2007 147

Lazy List

• Like optimistic, except– Scan once– Contains() never locks …

• Key insight– Removing nodes causes trouble– Do it “lazily”

© Herlihy-Shavit 2007 148

Lazy List

• Remove Method– Scans list (as before)– Locks predecessor & current (as before)

• Logical delete– Marks current node as removed (new!)

• Physical delete– Redirects predecessor’s next (as before)

© Herlihy-Shavit 2007 149

Lazy Removal

a 0 0 0a b c 0d1c

Logical Removal =Set Mark Bit

PhysicalRemova

l

Add a separate mark bit field to each node: 1. mark node as deleted 2. redirect pointer

© Herlihy-Shavit 2007 150

Lazy Removal

a 0 0 0a b c 0d

1 32 2

1c

4

LogicalRemova

l

PhysicalRemova

l

ReleaseLocks

1. Optimistically traverse list to find b

2. Lock b.curr then lock b.succ

3. verify predcurr4. Perform logical then

physical removal5. Release locks

© Herlihy-Shavit 2007 151

Lazy List

• All Methods– Scan through locked and marked

nodes– Removing a node doesn’t slow down

other method calls …

• Must still lock pred and curr nodes.

© Herlihy-Shavit 2007 152

Validation

• No need to rescan list!• Check that pred is not marked• Check that curr is not marked• Check that pred points to curr

© Herlihy-Shavit 2007 153

Business as Usual

a b c d

© Herlihy-Shavit 2007 154

Business as Usual

a b c d

© Herlihy-Shavit 2007 155

Business as Usual

a b c d

© Herlihy-Shavit 2007 156

Business as Usual

a b c d

© Herlihy-Shavit 2007 157

Business as Usual

a b c d

© Herlihy-Shavit 2007 158

Interference

a b c d

© Herlihy-Shavit 2007 159

Interference

a b c d

© Herlihy-Shavit 2007 160

Interference

a b c d

© Herlihy-Shavit 2007 161

Interference

a b c d

Remove c

© Herlihy-Shavit 2007 162

Validation

a b c d

b not marke

d

© Herlihy-Shavit 2007 163

Interference

a b c d

c not marke

d

© Herlihy-Shavit 2007 164

Interference

a b c d

b still points

to c

© Herlihy-Shavit 2007 165

Logical Delete

a b c d

mark

© Herlihy-Shavit 2007 166

Scan Through

a b c d

So what

?

Remove c

© Herlihy-Shavit 2007 167

Physical Deletion

a b c d

So what

?

Remove c

© Herlihy-Shavit 2007 168

New Abstraction Map

• S(head) =– { x | there exists node a such

that•A reachable from head and•a.object = x and•a is unmarked

– }

© Herlihy-Shavit 2007 169

Invariant

• If not marked then item in the set• and reachable from head • and if not yet traversed it is

reachable from pred

© Herlihy-Shavit 2007 170

Validationprivate boolean validate(Node pred, Node curr) { return !pred.next.marked && !curr.next.marked && pred.next == curr); }

© Herlihy-Shavit 2007 171

private boolean validate(Node pred, Node curr) { return !pred.marked && !curr.marked && pred.next == curr); }

List Validate Method

Predecessor not Logically removed

© Herlihy-Shavit 2007 172

private boolean validate(Node pred, Node curr) { return !pred.next.marked && !curr.next.marked && pred.next == curr); }

List Validate Method

Current not Logically removed

© Herlihy-Shavit 2007 173

private boolean validate(Node pred, Node curr) { return !pred.next.marked && !curr.next.marked && pred.next == curr); }

List Validate Method

Predecessor stillPoints to current

© Herlihy-Shavit 2007 174

Removetry { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally {

pred.unlock();curr.unlock();

}}}

© Herlihy-Shavit 2007 175

Removetry { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally {

pred.unlock();curr.unlock();

}}}

Validate as before

© Herlihy-Shavit 2007 176

Removetry { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally {

pred.unlock();curr.unlock();

}}}

Key found

© Herlihy-Shavit 2007 177

Removetry { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally {

pred.unlock();curr.unlock();

}}}

Logical remove

© Herlihy-Shavit 2007 178

Removetry { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally {

pred.unlock();curr.unlock();

}}}

physical remove

© Herlihy-Shavit 2007 179

Containspublic boolean contains(Object object) { int key = object.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked;}

© Herlihy-Shavit 2007 180

Containspublic boolean contains(Object object) { int key = object.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked;}

Start at the head

© Herlihy-Shavit 2007 181

Containspublic boolean contains(Object object) { int key = object.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked;}

Search key range

© Herlihy-Shavit 2007 182

Containspublic boolean contains(Object object) { int key = object.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked;}

Traverse without locking(nodes may have been

removed)

© Herlihy-Shavit 2007 183

Containspublic boolean contains(Object object) { int key = object.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked;}

Present and undeleted?

© Herlihy-Shavit 2007 184

Summary: Wait-free Contains

a 0 0 0a b c 0e1d

Use Mark bit + Fact that List is ordered 1. Not marked in the set2. Marked or missing not in the set

© Herlihy-Shavit 2007 185

Lazy List

a 0 0 0a b c 0e1d

Lazy Add() and Remove() + Wait-free Contains()

© Herlihy-Shavit 2007 186

Evaluation

• Good:– Contains method doesn’t need to lock– In fact, its wait-free! – Good because typically high %

contains()– Uncontended calls don’t re-traverse

• Bad– Contended calls do re-traverse– Traffic jam if one thread delays

© Herlihy-Shavit 2007 187

Traffic Jam

• Any concurrent data structure based on mutual exclusion has a weakness

• If one thread– Enters critical section– And “eats the big muffin” (stops running)

• Cache miss, page fault, descheduled …

– Everyone else using that lock is stuck!

© Herlihy-Shavit 2007 188

Reminder: Lock-Free Data Structures

• No matter what …– Some thread will complete method

call– Even if others halt at malicious times– Weaker than wait-free, yet

• Implies that– You can’t use locks– Um, that’s why they call it lock-free

© Herlihy-Shavit 2007 189

Lock-free Lists

• Next logical step• Eliminate locking entirely• Contains() wait-free and Add() and

Remove() lock-free• Use only compareAndSet()• What could go wrong?

© Herlihy-Shavit 2007 190

Adding a Node

a b c

© Herlihy-Shavit 2007 191

Adding a Node

a b c

b

© Herlihy-Shavit 2007 192

Adding a Node

a b c

b

CAS

© Herlihy-Shavit 2007 193

Adding a Node

a b c

b

© Herlihy-Shavit 2007 194

Adding a Node

a b c

b

© Herlihy-Shavit 2007 195

Removing a Node

a b c d

remove b

remove c

CAS CAS

© Herlihy-Shavit 2007 196

Look Familiar?

a b c d

remove b

remove c

Still in list

© Herlihy-Shavit 2007 197

Problem

• Method updates node’s next field• After node has been removed

© Herlihy-Shavit 2007 198

Solution

• Use AtomicMarkableReference• Atomically

– Swing reference and– Update flag

• Remove in two steps– Set mark bit in next field– Redirect predecessor’s pointer

© Herlihy-Shavit 2007 199

Marking a Node

• AtomicMarkableReference class– Java.util.concurrent.atomic package

address F

mark bit

Reference

© Herlihy-Shavit 2007 200

Extracting Reference & Mark

Public T get(boolean[] marked);

© Herlihy-Shavit 2007 201

Extracting Reference & Mark

Public T get(boolean[] marked);

Returns reference to object of type T

Returns mark at array index

0!

© Herlihy-Shavit 2007 202

Changing State

Public boolean compareAndSet( T expectedRef, T updateRef, boolean expectedMark, boolean updateMark);

© Herlihy-Shavit 2007 203

Changing State

Public boolean compareAndSet( T expectedRef, T updateRef, boolean expectedMark, boolean updateMark);

If this is the current reference

…

And this is the current mark …

© Herlihy-Shavit 2007 204

Changing State

Public boolean compareAndSet( T expectedRef, T updateRef, boolean expectedMark, boolean updateMark);

…then change to this new reference …

… and this new mark

© Herlihy-Shavit 2007 205

Changing State

public boolean attemptMark( T expectedRef, boolean updateMark);

© Herlihy-Shavit 2007 206

Changing State

public boolean attemptMark( T expectedRef, boolean updateMark);

If this is the current reference …

© Herlihy-Shavit 2007 207

Changing State

public boolean attemptMark( T expectedRef, boolean updateMark);

.. then change to this new mark.

© Herlihy-Shavit 2007 208

Removing a Node

a b c d

remove c

CAS

© Herlihy-Shavit 2007 209

Removing a Node

a b d

remove b

remove c

cCASCAS

failed

© Herlihy-Shavit 2007 210

Removing a Node

a b d

remove b

remove c

c

© Herlihy-Shavit 2007 211

Removing a Node

a d

remove b

remove c

© Herlihy-Shavit 2007 212

Traversing the List

• Q: what do you do when you find a “logically” deleted node in your path?

• A: finish the job.– CAS the predecessor’s next field– Proceed (repeat as needed)

© Herlihy-Shavit 2007 213

Lock-Free Traversal

a b c dCAS

Uh-oh

© Herlihy-Shavit 2007 214

The Window Class

class Window { public Node pred; public Node curr; Window(Node pred, Node curr) { this.pred = pred; this.curr = curr; }}

© Herlihy-Shavit 2007 215

The Window Class

class Window { public Node pred; public Node curr; Window(Node pred, Node curr) { this.pred = pred; this.curr = curr; }}

A container for pred and current

values

© Herlihy-Shavit 2007 216

Using the Find Method

Window window = find(head, key); Node pred = window.pred; curr = window.curr;

© Herlihy-Shavit 2007 217

Using the Find Method

Window window = find(head, key); Node pred = window.pred; curr = window.curr;

Find returns window

© Herlihy-Shavit 2007 218

Using the Find Method

Window window = find(head, key); Node pred = window.pred; curr = window.curr;

Extract pred and curr

© Herlihy-Shavit 2007 219

The Find Method

Window window = find(head, key);

At some instant,

pred curr succ

objector …

© Herlihy-Shavit 2007 220

The Find Method

Window window = find(head,key);

At some instant,

predcurr= succ

succ

object not in list

© Herlihy-Shavit 2007 221

Removepublic boolean remove(T object) {Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true;}}}

© Herlihy-Shavit 2007 222

Removepublic boolean remove(T object) {Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true;}}}

Keep trying

© Herlihy-Shavit 2007 223

Removepublic boolean remove(T object) {Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true;}}} Find neighbors

© Herlihy-Shavit 2007 224

Removepublic boolean remove(T object) {Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true;}}} She’s not there …

© Herlihy-Shavit 2007 225

Removepublic boolean remove(T object) {Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true;}}}

Try to mark node as deleted

© Herlihy-Shavit 2007 226

Removepublic boolean remove(T object) {Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true;}}}

If it doesn’t work, just retry, if it does, job

essentially done

© Herlihy-Shavit 2007 227

Removepublic boolean remove(T object) {Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true;}}}

Try to advance reference(if we don’t succeed, someone else did or will).

a

© Herlihy-Shavit 2007 228

Addpublic boolean add(T object) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(object); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true;}}}}

© Herlihy-Shavit 2007 229

Addpublic boolean add(T object) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(object); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true;}}}} Object already there.

© Herlihy-Shavit 2007 230

Addpublic boolean add(T object) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(object); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true;}}}}

create new node

© Herlihy-Shavit 2007 231

Addpublic boolean add(T object) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(object); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true}}}}

Install new node, else retry loop

© Herlihy-Shavit 2007 232

Wait-free Contains

public boolean contains(T object) { boolean marked; int key = object.hashCode(); Node curr = this.head; while (curr.key < key) curr = curr.next; Node succ = curr.next.get(marked); return (curr.key == key && !marked[0]) }

© Herlihy-Shavit 2007 233

Wait-free Contains

public boolean contains(T object) { boolean marked; int key = object.hashCode(); Node curr = this.head; while (curr.key < key) curr = curr.next; Node succ = curr.next.get(marked); return (curr.key == key && !marked[0]) }

Only diff from lazy list is that

we get and check marked

© Herlihy-Shavit 2007 234

Lock-free Findpublic Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; }}}

© Herlihy-Shavit 2007 235

Lock-free Findpublic Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; }}}

If list changes

while traversed, start overLock-Free

because we start over

only if someone

else makes progress

© Herlihy-Shavit 2007 236

public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; }}}

Lock-free Find

Start looking from head

© Herlihy-Shavit 2007 237

public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; }}}

Lock-free Find

Move down the list

© Herlihy-Shavit 2007 238

public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; }}}

Lock-free Find

Get ref to successor and current deleted bit

© Herlihy-Shavit 2007 239

public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; }}}

Lock-free Find

Try to remove deleted nodes in path…code details

soon

© Herlihy-Shavit 2007 240

public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; }}}

Lock-free Find

If found curr key that is greater or equal, return

pred and curr

© Herlihy-Shavit 2007 241

public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; }}}

Lock-free Find

Otherwise advance “window” and loop again

© Herlihy-Shavit 2007 242

Lock-free Find

retry: while (true) { … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); }…

© Herlihy-Shavit 2007 243

Lock-free Find

retry: while (true) { … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); }…

Try to snip out node

© Herlihy-Shavit 2007 244

Lock-free Find

retry: while (true) { … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); }…

if predecessor’s next field changed must

retry whole traversal

© Herlihy-Shavit 2007 245

Lock-free Find

retry: while (true) { … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); }…

Otherwise move on to check if next node

deleted

© Herlihy-Shavit 2007 246

Summary: Summary: Lock-free Removal

a 0 0 0a b c 0e1c

Logical Removal =Set Mark Bit

PhysicalRemovalCAS pointer

Use CAS to verify pointer is correct

Not enough!

© Herlihy-Shavit 2007 247

Summary: Summary: Lock-free Removal

a 0 0 0a b c 0e1c

Logical Removal =Set Mark Bit

PhysicalRemovalCAS

0dProblem: d not added to list…Must Prevent manipulation of removed node’s pointer

Node added BeforePhysical Removal CAS

© Herlihy-Shavit 2007 248

Our Solution: Combine Bit and Pointer

a 0 0 0a b c 0e1c

Logical Removal =Set Mark Bit

PhysicalRemovalCAS

0d

Mark-Bit and Pointerare CASed together

Fail CAS: Node not added after logical Removal

© Herlihy-Shavit 2007 249

A Lock-free Algorithm

a 0 0 0a b c 0e1c

1. Add() and Remove() physically clean up (remove) marked nodes

2. Wait-free find() traverses both marked and removed nodes

© Herlihy-Shavit 2007 250

Performance

On 16 node shared memory machineBenchmark throughput of Java List-based Setalgs. Vary % of Contains() method Calls.

© Herlihy-Shavit 2007 251

High Contains Ratio

Lock-free Lazy list

Course GrainedFine Lock-coupling

© Herlihy-Shavit 2007 252

Low Contains Ratio

Lock-free

Lazy list

Course GrainedFine Lock-coupling

© Herlihy-Shavit 2007 253

As Contains Ratio Increases

Lock-free

Lazy list

Course GrainedFine Lock-coupling

% Contains()

© Herlihy-Shavit 2007 254

Summary

• Coarse-grained locking• Fine-grained locking• Optimistic synchronization• Lock-free synchronization

© Herlihy-Shavit 2007 255

“To Lock or Not to Lock”

• Locking vs. Non-blocking: Extremist views on both sides

• The answer: nobler to compromise, combine locking and non-blocking– Example: Lazy list combines blocking

add() and remove() and a wait-free contains()

– Blocking/non-blocking is a property of a method