implementing declarative overlays from two talks by: boon thau loo 1 tyson condie 1, joseph m....

Implementing Declarative Overlays

From two talks by:Boon Thau Loo1

Tyson Condie1, Joseph M. Hellerstein1,2, Petros Maniatis2, Timothy Roscoe2, Ion Stoica1

1University of California at Berkeley, 2Intel Research Berkeley

P2

Overlays Everywhere…

Overlay networks are widely used today: Routing and forwarding component of large-

scale distributed systems Provide new functionality over existing

infrastructure

Many examples, variety of requirements: Packet delivery: Multicast, RON Content delivery: CDNs, P2P file sharing, DHTs Enterprise systems: MS Exchange

Overlay networks are an integral part of many large-scale distributed systems.

Overlay networks are an integral part of many large-scale distributed systems.

Problem

Non-trivial to design, build and deploy an overlay correctly: Iterative design process:

Desired properties Distributed algorithms and protocols Simulation Implementation Deployment Repeat…

Each iteration takes significant time and utilizes a variety of expertise

The Goal of P2

Make overlay development more accessible: Focus on algorithms and protocol designs, not

the implementation

Tool for rapid prototyping of new overlays: Specify overlay network at a high level Automatically translate specification to protocol Provide execution engine for protocol

Aim for “good enough” performance Focus on accelerating the iterative design

process Can always hand-tune implementation later

Outline

Overview of P2Architecture By Example Data Model Dataflow framework Query Language

ChordAdditional Benefits Overlay Introspection Automatic Optimizations

Conclusion

6

All-Pairs Reachability

R2: reachable(S,D) link(S,Z), reachable(Z,D)

R1: reachable(S,D) link(S,D)

Input: link(source, destination)Output: reachable(source, destination)

“For all nodes S,D, If there is a link from S to D, then S can reach D”.

link(a,b) – “there is a link from node a to node b”

reachable(a,b) – “node a can reach node b”

7





“For all nodes S,D and Z, If there is a link from S to Z, AND Z can reach D, then S can reach D”.

8








9






10

Towards Network Datalog

Specify tuple placement Value-based partitioning of tables

Tuples to be combined are co-located Rule rewrite ensures body is always single-site

All communication is among neighbors No multihop routing during basic rule

execution Link-restricted rules: Enforced via simple

syntactic restrictions

11


R1: reachable(@S,D) link(@S,D)

R2: reachable(@S,D) link(@S,Z), reachable(@Z,D)

Network Datalog

Query: reachable(@M,N)

@S

D

@a

b

@a

c

@a

d

reachableOutput

table:

Input table:

Query: reachable(@a,N)

@S

D

@c b

@c d

link@

SD

@b

c

@b

a

link@

SD

@a

b

link @

SD

@d

c

link

b dca

@S

D

@b

a

@b

c

@b

d

reachable @

SD

@c a

@c b

@c d

reachable @

SD

@d

a

@d

b

@d

c

reachable

Location Specifier “@S”

Query: reachable(@a,N)

12








13






14

R1: path(@S,D,P) link(@S,D), P=(S,D).

R2: path(@S,D,P) link(@S,Z), path(@Z,D,P2), P=SP2.

@S

D P @S

D P

@c d [c,d]

Query Execution

@S

D P @S

D P

Query: path(@a,d,P)

Neighbor table:

@S

D

@c b

@c d

link@S D

@b c

@b a

link@

SD

@a

b

link @S D

@d c

link

b dca

path path path

Forwarding table:

Chord Model

Relational data: relational tables and tuples Two kinds of tables: Stored, soft state:

E.g. neighbor(Src,Dst), forward(Src,Dst,NxtHop) Transient streams:

Network messages: message (Rcvr, Dst) Local timer-based events: periodic (NodeID,10)

...

Netw

ork In Dataflow

...

...

Netw

ork Out D

ataflow

node

Local Tables

route ...

Example: Ring Routing

3

28

15

1840

60

58 13

37

0

56

42

222433

Each node has an address and

an identifier

Each object has an

identifier.

Every node

knows its successor

Objects “served” by successor

3

28

15

1840

60

58 13

37

Ring State

node(IP40,40) succ(IP40,58,IP58)


Stored tables: node(NAddr, N) succ(NAddr, Succ, SAddr)

Example: Ring lookup

3

28

15

1840

60

58 13

37

0

56

42

222433

Find the responsible node for a given key k?

n.lookup(k)if k in (n, n.successor)

return n.successor.addr

elsereturn n.successor.

lookup(k)

lookup(IP40,IP37,59)

Ring Lookup Events

3

28

15

1840

60

58 13

37


Event streams: lookup(Addr, Req, K) response(Addr, K, Owner)


response(IP37,59,IP60)



n.lookup(k)

if k in (n, n.successor) return n.successor.addrelse return n.successor. lookup(k)

Query Language: Overlog

Datalog rule syntax: <head> <condition1>, <condition2>, … , <conditionN>.

Overlog rule syntax: <Action> <event>, <condition1>, … , <conditionN>.

Query Language: Overlog

Event: RECEIVE lookup(NAddr, Req, K)

Condition: lookup(NAddr, Req, K) & node(NAddr, N) & succ(NAddr, Succ, SAddr) & K in (N, Succ]

Action: SEND response(Req, K, SAddr) to Req

response@Req(Req, K, SAddr)

lookup@NAddr(Naddr, Req, K),node@NAddr(NAddr, N),

succ@NAddr(NAddr, Succ, SAddr), K in (N,Succ].

Overlog rule syntax: <Action> <event>, <condition1>, … , <conditionN>.

P2-Chord

Chord Routing, including: Multiple successors Stabilization Optimized finger

maintenance Failure recovery

47 OverLog rules13 table definitionsOther examples:

Narada, flooding, routing protocols

10 pt font

Introspection with Queries

Unifying framework for debugging and implementation Same query language, same platform

Execution tracing/logging Rule and dataflow level Log entries stored as tuples and queried

Correctness invariants, regression tests as queries: “Is the Chord ring well formed?” (3 rules) “What is the network diameter?” (5 rules) “Is Chord routing consistent?” (11 rules)

With Atul Singh (Rice) and Peter Druschel (MPI)

Automatic OptimizationsApplication of traditional Datalog optimizations to network routing protocols (SIGCOMM 2005)Multi-query sharing:

Common “subexpression” elimination Caching and reuse of previously computed results Opportunistically share message propagation across

rules Join

lookup.Addr =

node.Addr

Joinlookup.Ad

dr = succ.Addr

lookup SelectK not in (N,Succ)

Projectlookup(SAddr

,Req,K)

lookup

ProjectResponse(Req,K,SAddr)

SelectK in

(N,Succ)

responseJoinlookup.Add

r = node.Addr

lookupJoin

lookup.Addr =

succ.Addr

Automatic Optimizations

Cost-based optimizations Join ordering affects performance

Joinlookup.Add

r = node.Addr

Joinlookup.Add

r = succ.Addr

SelectK not in (N,Succ)

lookup Projectlookup(SAddr

,Req,K)

lookup

ProjectResponse(Req

,K,SAddr)

SelectK in

(N,Succ)

response

Joinlookup.Add

r = node.Addr

Joinlookup.Add

r = succ.Addr

Future Work

“Right” languageFormal data and query semantics Static analysis Optimizations Termination Correctness

Conclusion

P2: Declarative Overlays Tool for rapid prototyping new overlay

networks

Declarative Networks Research agenda: Specify and

construct networks declaratively Declarative Routing : Extensible

Routing with Declarative Queries (SIGCOMM 2005)

implementing declarative overlays from two talks by: boon thau loo 1 tyson condie 1, joseph m....

Documents

d links

d query

nodes s

b node

node breachablea

node ballpairs reachabilityr2

iterative design processcan

n reachableoutput table