The Impact of DHT Routing Geometry on Resilience and

Proximity

K. Gummadi , R. Gummadi..,S.Gribble, S. Ratnasamy, S. Shenker, I. Stoica

Introduction• how routing geometries affect the resilience and

proximity properties of DHTs– coping with node failures: static resilience– adapting to Internet topology, proximity issues: path latency

and local convergence– flexibility in selection of neighbors and routes an essential

factor

• evaluating various geometries for gaining insight for better designs

• admitted flaws– not evaluating all routing algorithms– not considering management overhead– focusing on only two performance issues

Terminology• Algorithm: the exact details of selecting neighbors and next-hops• Geometry: a geometric interpretation of how the selections are

made– geometry constrains the choices, but small changes in algorithm do not

change the geometry

• Flexibility: the algorithmic freedom left after the geometry is chosen– neighbor selection

• does the geometry allow selecting neighbors based on proximity• does the geometry support sequential neighbors

– route selection• how many options for next-hop in case of failure• does the geometry allow selecting next-hops based on proximity

• Sequential neighbors– can route and progress towards all destinations– global ordering on distances required (naturally in Ring, add-on for others)

Basic Routing Geometries (1/2)Tree-e.g. PRR

-node ids are leaf nodes

-distance = smallest common subtree

-routing MSB first

Hypercube-e.g. CAN

-node id represents position

-neighbors differ by 1 bit only

-distance is # of differing bits

-routing in any order, previous corrections preserved

Butterfly-e.g. Viceroy

-nodes in log n stages

-nodes at stage i can correct ith bit

-imposed global ordering: global and stage successors/predecessors required to be held as neighbors

-routing in log n hops with constant state at each node: first through stages, then through successors/predecessors

Basic Routing Geometries (2/2)Ring-e.g. Chord

-nodes in one-dimensional cyclic id space

-distance is the clockwise numeric distance between nodes

-routing in log n hops, because each hop cuts the distance in half

XOR-e.g. Kademlia

-distance is XOR of ids

-routing MSB first, but in case of failures can correct the next bit

-these corrections are not necessarily preserved

=> multiple non-optimal paths

Hybrid-e.g. Pastry

-dual mode, e.g. tree + ring

-nodes both leafs and on circle

-distance both tree and ring distance

-fallback to ring when tree fails

-can progress on the tree while not on progressing on the ring

Static Resilience

• DHTs resilient to node failures

• Three aspects of resilience– data replication– routing recovery– static resilience

• routing before recovery algorithms kick in

• the only aspect considered in this paper

Static Resilience• Performance of DHTs with routing tables of

equal sizes

• What happens when failures are present?– failed paths?– increase in path length?

Static Resilience of Various Geometries

• consistent with flexibility in route selection

• Ring and Hypercube have twice the flexibility of Hybrid and XOR

• Tree and Butterfly have no flexibility

• Hypercube has equal length alternative paths

• Hybrid, Tree, and XOR have only longer alternative paths

• Ring has some longer paths

• Butterfly not applicable

Static Resilience: Sequential Neighbors

• 16 sequential neighbors added

• XOR not applicable, Tree included in Hybrid

• Ring has natural support for sequential neighbors

• greatly increased resilience

• increase in resilience comes at the cost of path stretch

• note the scale!

• path increase in Butterfly is way higher

Static Resilience: Sequential vs. Regular Neighbors

• at high failure rates, sequentials are better

• sequentials can lead to longer paths

• a large number of regulars is a good compromise

Proximity• DHTs designed to route effectively in terms of hopcounts

• end-to-end latency issues approached through proximity methods

PNS- neighbor selection

- identifying closest nodes is hard => heuristics needed

- PNS(K), K=# of samples

- a node’s latency distribution helps in choosing K

PRS- next-hop selection

-complicated tradeoff between # of hops and latency

- different heuristics for different geometries

PIS- ids based on location

- load balancing hard

- not discussed here

Proximity: Performance Results• Basic assumptions in this evaluation

– evaluation based on recursive routing

– geometries have little effect, support for PNS/PRS essential

– performance of proximity methods depends on topology and its latency characteristics

• good approximation by looking at the latency distribution for a typical node

– Hybrid left out, Butterfly not applicable

– PNS/PRS not expensive in terms of hopcounts

Proximity: PNS vs. PRS

• XOR and Ring support both

• PRS better than plain

• PNS far better (PNS 2i neighbor options, PRS i options)

• PRS+PNS only a small improvement over PNS

• PNS(16) gives similar results

• 16 sequential neighbors improve PRS and Plain a little, but not significantly

=> PNS support most important

Proximity: More Results

• Does geometry matter?– Tree: PRS n/a

– Hypercube: PNS n/a

– performance of pairs very close

=> PNS/PRS support matters, not geometry

• Absolute performance– depends on latency distribution

=> overlay latencies can be reduced to small multiples of underlying Internet path latencies

Local Convergence (1/2)• messages sent from two nearby nodes converge at a node

near the sources

=> low latencies or bandwidth savings in e.g.

- overlay multicast

- caching

- server selection

• measured by number of exit points

PNS vs. PRS: PNS nearly optimal, PRS not effective

Local Convergence (2/2)

• PNS(16) and PRS equally ineffective

• PNS(16)+PRS better

=> PRS more relevant with limited samples and domain sizes

=> geometries don’t matter, PNS/PRS support is the key

Summary

• geometry constrains other design choices• flexibility important (neighbor & next-hop

selection)• Ring & XOR are flexible: support for both PNS

and PRS• Why not Ring?

– flexibility– natural support for sequential neighbors– tested well