opportunistic multipath forwarding in publish/subscribe systems reza sherafat kazemzadeh and...

Opportunistic Multipath Forwarding in Publish/Subscribe Systems

Reza Sherafat Kazemzadeh AND Hans-Arno Jacobsen

Middleware Systems Research GroupUniversity of Toronto

Opportunistic Multipath Forwarding 2

Agenda

• The content-based publish/subscribe model– Characteristics and challenges

• Our approach– Overlay neighborhoods– Adaptive multipath forwarding

• Experimental evaluation results

Middleware 2012


Content-Based Pub/Sub Model

Pub/Sub

S

SS SS

S

S

PPublish PP

P

SS

Subscribers

PPP

Publishers

Middleware 2012


Content-Based Pub/Sub Model

Pub/Sub

Publish

SS

Subscribers

PPP

Publishers

Middleware 2012

Many to many communication between a large number of publishers/subscribers

Selective delivery based on subscription matching• Unicast (one)• Multicast (some)• Broadcast (all)

Variations in traffic patterns makes it difficult to design an optimal overlay network


Forwarding paths in the overlay are constructed in fixed end-to-end manner (no/little path diversity)

This results in a high number of “pure forwarding” brokers

Current Pub/Sub Overlays

D CE B A PS✗ ✗ ✓✗✓

PP

#msgs delivered#msgs sentSystem Yield = = 1/5 = 20%

Middleware 2012


Summary of Our Goals

• Construct a highly connected overlay mesh that provides high path diversity between publishers and subscribers

• Avoid pure forwarders by allowing brokers to make fine-grained forwarding decisions based on individual publications and their matching sets

• Improve system yield, efficiency, scalability and delivery delay

• Support dynamic adaptive routing based on live traffic patters while avoiding high costs of full overlay reconfiguration

Middleware 2012


• Fixed end-to-end (baseline)Total msgs: 6

• Forwarding strategy 1Total msgs: 5

• Forwarding strategy 2Total msgs: 3

A B C

*

D

*

*

Forwarding Strategies

A B C

*

D

*

*

p * *

*

p * *

*

A B C

*

D

*

*

p * *

*

Middleware 2012


Routing Tables(Δ-neighborhoods

knowledge)

Links Management(best links via a gain

function)

Pub. Forwarding(Path Computations

for strategies)

Our Approach in a Nutshell

A

Δ=1

Δ=2

Δ=3

S

Middleware 2012


A C

* *

*

B DA C

* *

*

B D

Path Computation forForwarding Strategies

Strategy 1

Middleware 2012

A B C

*

D

*

*

p * *

*

A B C

*

D

*

*

p * *

*

Strategy 2


Experimental Evaluations• We have implemented the algorithms and performed large-scale

experimental evaluations with up to 500 brokers

• Datasets– Synthesized based on Zipf distribution– Social networking traces from Facebook

• We measured performance of the system in terms of:– Overlay mesh connectivity– Delivery delay– Maximum system throughput– System yield– Publication propagation path length– Memory and CPU overhead

Middleware 2012


Num

ber o

f Ava

ilabl

e pa

ths

Path Diversity in Overlay Mesh

Network size:120Delta=3

Network size:250Delta=3

Path diversity:20% of brokers w/ 100 paths

Path diversity:10% of brokers w/ 1000 paths

Middleware 2012

Graph is based on a snapshot of the state of links in a running system


Publication Hop CountExperiment setup:• 120 Brokers• Publish rate is 1,800 msgs/sec and number of deliveries: 73,000 (in 5 min)

Middleware 2012

Fixed-end-to-end

Strategy 2

115% throughput enhancement


Conclusions

• Brokers build a highly connected overlay mesh and make fine-grained forwarding decisions for each publication in order to avoid pure forwarding neighbors

• We used the notion of overlay neighborhoods to enable local traffic profiling and avoid high costs of overlay reconfiguration

• Our approach enhances system’s efficiency and yield, and ultimately improves its scalability and performance

Middleware 2012


Thanks

Questions!

Middleware 2012


Client

Broker

Pub/Sub Characteristics

• Many to many communication between a large number of publishers/subscribers

• Selective delivery based on subscription matching– Unicast (one)– multicast (some)– broadcast (all)

• Traffic patterns depends on workload application and may change over time

Middleware 2012


Related Work: Overlay Reconfiguration

Broker overlay is “reconfigured” by addition and removal of links between brokers

Advantages– Forwarding path may be improved bringing some

publishers and subscribers closer together

Disadvantages– Some forwarding paths between publishers and

subscribers may indeed suffer– Resulting overlay still relies on fixed end-to-end

paths– Reconfiguration is costly and requires full or partial

re-propagation of subscriptions

[1] Virgillito, A., Beraldi, R., Baldoni, R.: On event routing in content-based publish/subscribe through dynamic networks, FTDCS ‘03[2] Virgillito, A., Beraldi, R., Baldoni, R.: On event routing in content-based publish/subscribe through dynamic networks. In: FTDCS. (2003)

Re-configur

e

Middleware 2012


• Links types– Primary links– Soft links – shortcut links– Candidate links – expeditionary links

• Traffic profiling– In intervals of T sec, brokers count the

number of pubs sent over each links– Gain function

gain(A,B,T) = pub_traffic_during_T * dist(A,B)

• Soft link selection– Profiling input– Link state measured– Broker load information exchanged

Links ranking

A B C D E F G E F …

high → low

Links ManagementOverlay Network

A B

C

DB

Neighbor

loadHig

h ra

nkLink

quality

Primary linkCandidate link Soft link

Middleware 2012


Master v. Working Routing Data Structures

• Overlay map captured by brokers’ Δ-neighborhoods are relatively static Master overlay Map (MOM)

• Brokers link connectivity change dynamically, brokers need an efficient way to compute forwarding paths over the changing set of links Working Overlay Map (WOM)

• MOM is a concise representation of the primary overlay that only contains brokers with a direct link

Master Routing Tables

Working Routing Tables construct

Middleware 2012


Master v. Working Subscription Tables

• Similar to WOM, brokers adapt their subscription tables based on the current set of available links: Working Subscription Table (WST)

Between Brokers

Sets

Beyond Brokers

Sets

Behind Brokers

Sets

Master Subscription

Table

Working Subscription

Table

Working Overlay Map

Middleware 2012


Delivered publications

Strategy Number of pure Pure Forwarders

System Yield

284,000(Dense Workload)

Baseline 195,000 59%

Strategy 1 104,000 73%

Strategy 2 69,000 80%

Delivered publications Strategy Number of pure

Pure ForwardersSystem Yield

73,000(Sparse Workload)

Fixed end-to-end 91,000 44%

Strategy 1 42,000 63%

Strategy 2 29,000 71%

System Yield (measure of efficiency)

Middleware 2012


Publication Hop CountExperiment setup:• 120 Brokers• Sparse and dense workloads• Publish rate of 1,800 msgs/sec Deliveries: 73,000 in 5 min

Middleware 2012


Publication Hop CountExperiment setup:• 120 Brokers• Sparse publication/subscription workload• Publish rate of 1,800 msgs/sec Deliveries: 73,000 in 5 min

Middleware 2012


Publication Hop CountSparse Matching Workload Dense Matching Workload

Multi-path forwarding is more effective in sparse workloads

Middleware 2012


Impact of Broker Fanout on Subscription CoveringExperiment setup:• 500 brokers• Fanout of 5-25

Middleware 2012

opportunistic multipath forwarding in publish/subscribe systems reza sherafat kazemzadeh and...

Documents

forwarding paths

forwarding strategy

optimal overlay network

forwarding path computations

running system slide

resulting overlay

forwarding strategies

connected overlay mesh