1

PASTRYPASTRY

Partially borrowed from Gabi Kliot’s presentation

2

“Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems” Antony Rowstron (Microsoft Research) Peter Druschel (Rice University)

Sources

3

PastryGeneric p2p location and routing substrate

(DHT)

Self-organizing overlay network (join, departures, locality repair)

Consistent hashing Lookup/insert object in < log2

b N routing steps

(expected) O(log N) per-node state Network locality heuristics

Scalable, fault resilient, self-organizing, locality aware, secure

4

Pastry: Object distribution

objId/key

Consistent hashing

128 bit circular id space

nodeIds (uniform random)

objIds/keys (uniform random)

Invariant: node with numerically closest nodeId maintains object

nodeIds

O2128 - 1

5

Pastry: Object insertion/lookup

X

Route(X)

Msg with key X is routed to live node with nodeId closest to X

Problem:

complete routing table not feasible

O2128 - 1

6

Pastry: Routing

Tradeoff

O(log N) routing table size 2b * log2

bN + 2l

O(log N) message forwarding steps

7

Pastry: Routing table (# 10233102)

L nodes in leaf set

log2b N Rows

(actuallylog2b 2128=

128/b)

2b columns

L neighbors

Pastry: Leaf sets

Each node maintains IP addresses of the nodes with the L numerically closest larger and smaller nodeIds, respectively. routing efficiency/robustness fault detection (keep-alive) application-specific local coordination

9

Pastry: Routing procedure

If (destination is within range of our leaf set) forward to numerically closest member

elselet l = length of shared prefix let d = value of l-th digit in D’s addressif (Rl

d exists) forward to Rl

d

else forward to a known node (from ) that (a) shares at least as long a prefix(b) is numerically closer than this node

MRL

10

Pastry: Routing

Properties• log2

b N steps • O(log N) state

d46a1c

Look for (d46a1c)

d462ba

d4213f

d13da3

65a1fc

d467c4d471f1

11

Pastry: Locality propertiesAssumption: scalar proximity metric e.g. ping/RTT delay, # IP hops traceroute, subnet masks a node can probe distance to any other node

Proximity invariant: Each routing table entry refers to a node closeto the local node (in the proximity space), amongall nodes with the appropriate nodeId prefix.

12

Pastry: Geometric Routing in proximity space

d46a1c

Route(d46a1c)

d462ba

d4213f

d13da3

65a1fc

d467c4d471f1 d467c4

65a1fcd13da3

d4213f

d462ba

Proximity space

The proximity distance traveled by message in each routing step is exponentially increasing (entry in row l is chosen from a set of nodes of size N/2bl)The distance traveled by message from its source increases monotonically at each step (message takes larger and larger strides)

NodeId space

13

Pastry: Locality properties

Each routing step is local, but there is no guarantee of globally shortest path

Nevertheless, simulations show: Expected distance traveled by a message

in the proximity space is within a small constant of the minimum

Among k nodes with nodeIds closest to the key, message likely to reach the node closest to the source node first

14

Pastry: Self-organization

Initializing and maintaining routing tables and leaf sets

Node addition Node departure (failure)

The goal is to maintain all routing table entries

to refer to a near node, among all live nodes with appropriate prefix

15

New node X contacts nearby node A A routes “join” message to X, which arrives

to Z, closest to X X obtains leaf set from Z, i’th row for

routing table from i’th node from A to Z X informs any nodes that need to be aware

of its arrival X also improves its table locality by requesting

neighborhood sets from all nodes X knows In practice: optimistic approach

Pastry: Node addition

16

Pastry: Node addition

X=d46a1c

Route(d46a1c)

d462ba

d4213f

d13da3

A = 65a1fc

Z=d467c4d471f1

New node: X=d46a1c

A is X’s neighbor

17

d467c4

65a1fcd13da3

d4213f

d462ba

Proximity space

Pastry: Node addition

New node: d46a1c

d46a1c

Route(d46a1c)

d462bad4213f

d13da3

65a1fc

d467c4d471f1

NodeId spaceX is close to A, B is close to B1. Why X is close to B1?The expected distance from B to its row one entries (B1) is much largerthan the expected distance from A to B (chosen from exponentially decreasing set size)

X

B1 is first row of B

18

Node departure (failure)

Leaf set repair (eager – all the time): Leaf set members exchange keep-alive

messages request set from furthest live node in set

Routing table repair (lazy – upon failure): get table from peers in the same row, if not

found – from higher rows Neighborhood set repair (eager)

19

Pastry: Summary

Generic p2p overlay network Scalable, fault resilient, self-

organizing, secure O(log N) routing steps (expected) O(log N) routing table size Network locality properties