Databases and Distributed Systems

Maintaining Replicas in Unstructured P2P SystemsCoNEXT, Madrid, 12/11/2008

Christof Leng, TU Darmstadt

Wesley W. Terpstra, TU Darmstadt

Bettina Kemme, McGill University

Wilhelm Stannat, TU Darmstadt

Alejandro P. Buchmann, TU Darmstadt

http://www.bubblestorm.net

http://www.dvs.tu-darmstadt.de

2 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

Replicate, what?

3 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

Replicas far and small

Our research focuses on peer-to-peer search The data we replicate is usually small (< 10kB) Modern unstructured overlays (Sarshar’04, Ferreira’05,

BubbleStorm) can have several hundred copies of an object

Why replicate so far? The more copies, the easier it is to find one The more providers, the harder to overload them When a node leaves, its copy is gone with the wind

To be clear: this talk is not about replicating files

4 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

Two kinds of replicated data

Maintained

“I’m online @132.160.222.1” “Tell me when a paper is

published with ‘P2P’&‘search’” “I can provide these files” “I am waiting for event X”

Service Lists Subscriptions

Collective

Wiki articles Information about physical

objects outside the network Distributed file systems System backups

“Persistent” information

5 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

Who maintains replicated data?

A Maintainer

Ideal for data that should never outlive its owner

The owner can manage it

If the owner crashes, any remaining replicas are junk

The Collective

Ideal for data that should live until explicitly deleted

No clear managing authority

Replicas of undeleted objects should remain in the system

6 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

Our Paper: Maintainer-based

“Let there be replicas!”

7 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

Maintaining Replicas

We want to be able to Ensure the system has no junk replicas (of objects whose

maintainer has left) – they cause bad search results! Ensure that there are exactly the number of replicas requested

We need to be able to Keep junk contained so the system remains useful Increase/decrease the density of replicas in the system Hold the density of replicas steady against network churn Update or destroy all the replicas of an object

...you can’t always get what you want…

8 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

An INformal model

9 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

The Good (aka assumptions)

Nodes all run our software – so they mostly cooperate

Computing a sum over participants in the network is easy

…and since we do unstructured peer-to-peer:

We don’t care too much about who has which particular replica

10 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

The Bad (aka Reality)

Churn is out of our control We can’t stop it Its rate changes over time We cannot influence participant lifetimes

Nodes sometimes crash They don’t say good-bye; they just leave This happens a lot

11 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

The Ugly

Storage Providing Peers crash silently fixed replication is impossible

Guarantee probability distribution? sufficient to prove the correctness of

Stochastic algorithms

Maintainers crashes are silent replicas should have been deleted Zero junk is impossible

Perhaps guarantee below a threshold? sufficient to prove the performance of

Stochastic algorithms

12 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

Don’t try this at home

13 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

A common maintenance strategy

Maintainer:1. Push desired replicas into system2. Wait for X minutes3. Goto 1

Storage Providing Peer: After Y minutes, replica is deleted

14 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

Why this is bad

What should the parameters X and Y be?

If X is too slow:

If Y is too slow:

If X or Y are too fast, more traffic is expended than necessary

Problem: Churn rate is out of our control (unboundable) Correctness requires setting X for the worst possible situation (costly)

repl

icas

time

RequirementReplicas

repl

icas

time

TolerableJunk

15 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

Pull on Join

16 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

Observation: Density

When a storage providing peer leaves the system Replica count might be reduced Expected replica density remains the same Holding replica density fixed dodges the problem of crashes

Not just a semantic difference! Must compensate by replicating whenever peers join

We can adjust the density as the population changes It is still possible to hold replicas at a fixed value (or √n)

17 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

Our replication algorithm

Maintainer: Push out initial replicas Record who has received replicas (superset) Push extra replicas to increase density Probabilistically delete existing replicas to decrease density

Storage Providing Peer: On join, ask randomly selected maintainers for replicas Accept replicas pushed out from maintainers

18 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

Visual Example

“Give me more replicas!”“Hello!”

“Bye Bye!”

“I don’t need so many.”

…

“Chow!”

19 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

Convergence to the Binomial

20 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

For the whole network

21 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

Flushing Junk

22 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

Observation: Pulls have no Junk

Recall: Junk is bad Unnecessarily consumes storage Results in spurious query results

When a node joins, all the replicas it receives are valid

So, to control junk… Blow everything away Reload fresh replicas

23 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

But the cost?!

Sounds expensive

Not all storage providing peers (c)rash, only c % We require only g % of replicas to be (g)ood

If c=10% and g=80%, the overhead of flushing is ½ c/(1/g – 1) = 20% extra replica transfers only applies to especially long-lived storage peers

It might be possible to optimize this cost away. We don’t. Be careful! Most of the obvious optimizations are wrong It’s easy to introduce statistical defects that accumulate over time

24 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

When to flush?

Expected number of replicas stored is easy to compute (a sum)

Flush when: stored replicas > expected replicas + tolerable junk

Only one problem:

Peers that happen to store more than average flush earlier those peers are preferentially destroyed there are less replicas than there should be

repl

icas

time

Desired ReplicasAverage Replicas

25 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

Independence is needed

P(v stores o | v flushes) = P(v stores o) This equality fails if a node flushes because it is full

Solution:

Use the flow of replicas through one bucket to flush the other The buckets’ replica flows are statistically independent: done!

26 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

Distribution of Junk

27 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

Things you’ll find in the paper

How pull on join handles different replication densities The details of the flush threshold equation Formal Stochastic proof of correctness How to support peers with different storage capacities How to support maintainers behind NAT/firewalls Cost (in operations) or the proposed algorithms Simulations of what-if scenarios

28 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

Conclusion

Providing replication guarantees in peer-to-peer is feasible Strong guarantees are impossible Probabilistic guarantees are tricky Seemingly innocent choices result in statistically bad behaviour

Maintainer-based replication is a relevant sub-problem Required for service lists and query subscriptions

Compared to collective replication, maintainer-based requires junk control has an obvious node (maintainer/owner) to manage replication

29 Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk

Thanks for listening!

?Questions

http://www.bubblestorm.net

http://www.dvs.tu-darmstadt.de