shrideep pallickara and geoffrey fox

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,[email protected]

NaradaBrokering: A Middleware Framework and Architecture for Enabling P2P Grids

Middleware 2003Rio de Janeiro, Brazil16-20 June 2003

Shrideep Pallickara and Geoffrey Foxspallick, [email protected]

Community Grid Computing Laboratory,Pervasive Technology Labs

Indiana University.http://www.naradabrokering.org



Talk Outline• P2P Grids

• Messaging Infrastructure requirements

• NaradaBrokering Overview

• P2P support

• Extensible transport framework

• Security and Monitoring Infrastructure

• Conclusions

• Future Work



P2P Grids• Grids - Typified by infrastructure for seamless access to

high-end computing resources.– Job submissions, data management services.

• P2P systems – Sophisticated resource sharing environments. – Search, discovery & sharing of resources (CPU/files).

• P2P Grid comprises services of both Grids and P2P systems.

• Integrates ideas of computational grids, web services, P2P systems and message-oriented-middleware.



Messaging Infrastructure for P2P Grids• Scaling – Support large number of devices/users.

• Efficient disseminations of interactions

• Guaranteed Delivery Mechanisms

• Location Independence

• Support for P2P interactions – JXTA, Gnutella

• Interoperate with Messaging clients

• Performance Monitoring and Security services.



NaradaBrokering: Features• Based on a network of cooperating broker nodes

– Cluster based architecture allows system to scale to arbitrary size

• Originally to provide uniform software multicast to support real-time collaboration linked to publish-subscribe for asynchronous systems.

• Now has four major core functions– Message transport (based on performance) in multi-link

fashion– General publish-subscribe including JMS, JXTA and Gnutella

(started) – Support for RTP-based audio/video conferencing.– Federation of multiple instances (just starting) of Grid services



NB: Engineering Issues Addressed• Tunnel through firewalls/proxies

– Microsoft’s ISA, Checkpoint, Apache

• Support for multiple network protocols such as TCP, UDP, Multicast, SSL, RTP and HTTP.

• Support for both blocking and non-blocking IO.• Scaling of software multicast

– Efficient calculation of destinations and routes..

• Supports local broker accesses• Transparently replace single server JMS systems

with a distributed solution.



NaradaBrokering: Organization

SSC-A SC-1

SC-2

SC-3

l13 14

15

n20

21

i4 5

6

j7 8

9

m16 17

18

k10 11

12

h1 2

3

19

Broker Node

Service Provider

End Client

1, 10 Super-super-cluster controller

5, 9, 10, 16 Super-cluster controller 2,4, 6,8, 12,14,18,20 Cluster controller

k

10 11

12

SP

SPSP

SP

SP

11a10a

12a

EC EC



Organization of Profiles and Routing• Client subscriptions are stored hierarchically within the

system.– A broker maintains client subscriptions, cluster-controller

maintains broker subscriptions and so on.

• When an event is received, the event is matched against stored profiles and destinations are computed– A cluster-controller computes broker destinations. A broker

computes client destinations.

• Every broker node, when supplied with a set of destinations, computes the best broker-hops to take to reach these destinations.



NaradaBrokering: Distributed Results

Transit Delays under different matching rates:22 Brokers 102 Clients

Match Rate=100% Match Rate=50% Match Rate=33% Match Rate=4%

0100

200300

400500

600700

Publish Rate (Events/sec) 0 50100150200250300350400450500

Event Size (Bytes)

050

100150200250300350400450

Transit Delay (MilliSeconds)

i4 5

6 l13 14

15

j7 8

9

h1 2

3

k10 11

12

m16 17

18

n20

21

19

22

MeasuringSubscriber

Publisher

Every broker – SPARC Ultra-5 (128 MB, 333 MHz)105 Clients – SPARC-Ultra-60 (512MB, 360 MHz)JRE-1.2.1, 100 Mbps Network



NaradaBrokering: Matching Engines• Matching engines are responsible for matching

events against stored profiles.

• NaradaBrokering supports a variety of matching engines supporting– “/” separated String based topics– Equality based <tag,value> topics– Integer based topic– SQL (from JMS) – XPath based queries



Matching Engine Performance

0

2

4

6

8

10

20 30 40 50 60 70 80 90 100

Dela

y (

Mic

rose

conds)

Number of subscriptions (in thousands) being matched

Average delay to match event with String-based Matching for subscriptions with different sizes

String size=16String size=24String size=32

0

10

20

30

40

50

60

70

20 30 40 50 60 70 80 90 100

Dela

y (

Seco

nds)

Number of subscriptions (in thousands) being matched

Average delay to match event to Xpath and SQL-based subscriptions

SQL Delay XPath Delay

<?xml version="1.0" encoding="ISO-8859-1"?><menu> <softdrinks> <brand>Minute Maid</brand> <fruit>Apple</fruit> <source>Brazil</source> <company>Coca Cola</company> <price>2.90</price> <year>2003</year> </softdrinks></menu>

XPath Query type: /menu/softdrinks[price>1.80]

Stand alone processPentium-3 1 GHZ 256MB RAMJRE 1.4



Why Support P2P?• Core features – Resource Sharing & Discovery

• CPU cycles: SETI@home, Folding@HOME

• File Sharing: Napster, Gnutella

• Deployments user driven – No dedicated management• Management of resources

– Expose resources & specify security strategy

– Replicate resources based on demand

• Dynamic peer groups, fluid group memberships• Sophisticated search mechanisms

– Peers respond to queries based on their interpretations

– Responses do not conform to traditional templates.



P2P Systems: The Downsides• Routing not very sophisticated

– Inefficient network utilization • Usually relying on simple forwarding of interactions

– Peer Traces (to eliminate echoing)– Attenuations (to suppress propagation)

• TTL’s associated with interactions.

• Interactions are attenuated– Resulting in localized interactions and a fragmented

world of multiple P2P subsystems



NaradaBrokering & JXTA Interaction Model

• Based on proxy model– Acts as both Rendezvous peer

(JXTA routers) and NaradaBrokering client.

• No changes to JXTA core or straitjacketing of interactions– Change made to Rendezvous

layer

• Peers are not aware that they interact with a Narada-JXTA proxy or Rendezvous peer.

• Narada-JXTA provides JXTA guaranteed long distance delivery

NARADA-JXTA proxy

NARADAbroker cloud

Peers

JXTARendezvousPEER

Dynamic/fluidpeer groups

High end "long lived"/persistent resources



NaradaBrokering-JXTA Proxy• Glean relevant information from JXTA interactions.

– Peer group advertisements (XML Doc describing resource)

– Requests/Responses to be part of peer group.

– Messages sent to a peer group.

– Queries and responses to these queries.

• Subscribe to relevant topics to ensure delivery

• Construct corresponding Narada-JXTA event from interactions.– These events lend themselves to efficient routing.



NaradaBrokering-JXTA Apps and Setups• Applications

– Integrated NaradaBrokering-JXTA environment tested under JXTA shell and myJxta (InstantP2P)

– Plan to integrate myJxta into Anabas (distance education software) with NaradaBrokering managing P2P and middle-tier (JMS) style interactions.

• Experimental Setup– Sender/receiver - (Pentium-3, 1 GHz, 256 MB RAM). – Every node (broker/router) hosted on a different machine

(Pentium-3, 1 GHz, 256 MB RAM). – Machines reside on a 100 Mbps LAN– Run-time environment for all the processes is

JDK-1.3 build Blackdown-1.3.1, Red Hat Linux 7.3



0

50

100

150

200

250

300

500 1000 1500 2000 2500 3000 3500 4000 4500 5000 5500

Tra

nsit

De

lay

(Mill

ise

co

nd

s)

Message Payload Size (Bytes)

Transit delay for message samples in Narada, JXTA and Narada-JXTA Topology - III (8 routers) Internal Machines

NaradaBrPure JXTA

Narada-JXTAR R

R

R

R R

R

R

(a)

R R

R

R

R R

R

R

N

(b)

N N

N

N

N N

N

N

(c)



NaradaBrokering Communications• Applications interface to NaradaBrokering through

UserChannels.• UserChannels have publish/subscribe semantics• Links implement a single conventional “data” protocol.

– Different links can have different underlying transport implementations

– Addition of new transport protocols within the Framework is easy to achieve.

– Administrative channel negotiates the best available communication protocol

• Link implementations can incorporate their own handshaking protocols to facilitate communications and data exchange.



NaradaBrokering Communications - II

Transport InterfacesLink

PerformanceData

TransportHandler Link

Factory

LinkFactory

LinksSpecific to a transport

Link Monitors

Data accumulated byMonitoring Service

Brokernode Administrative Link (HTTP)

Optimal Transport

Alternate Link

TransportInterfaces

(Application andContent Dependent)

Negotiated with infoexchanged over

Administrative Link

Brokernode

MonitoringService



Performance Monitoring• Every broker incorporates a Monitoring service

that monitors links originating from the node.

• Every link measures and exposes a set of metrics– Average delays, jitters, loss rates, throughput.

• Individual links can disable measurements for individual or the entire set of metrics.

• Measurement intervals can also be varied

• Monitoring Service, returns measured metrics to Performance Aggregator.



Performance Aggregation• Aggregated information will be used to

– Circumvent bottlenecks

– Aid routing algorithms

– Facilitate Dynamic Load-balancing

BrokerNode

LinkData

BrokerNode

LinkData

Performance AggregationService

Control MessageExchange

Aggregates infofrom nodes in acertain domain

MonitoringService



NaradaBrokering: Security Framework• Based on Message Level Security• Authentication – Confirm whether a user is really who he says he

is.• Authorization – Identify if the user is authorized to receive

certain events• Key distribution – Based on authentication & authorization

distribute keys, which ensure that only the valid clients are able to decrypt encrypted data.

• Digital Signing – Have the ability to verify the source of the event and whether the source is authorized to publish events conforming to the specified template.

• Communication Protocol Independent • Detection and Response to Security Compromise

– Clients required to respond to queries (stored during initializations) issued at random intervals.



2Respond back with topickey if authorized to publish

NaradaBrokering Broker Cloud

KeyManagementCenter (KMC)

1 2 3

4

5

6

78

Broker Node

Entity (Publisher or Subscriber)

SSL encryptedcommunications

6Respond back with topic key ifauthorized to subscribe

7

Create subscription requestCompute Message DigestSign MD and message IDIssue Subscription request Message

4Verify Signature & PermissionsCheck integrity by verifying MDCheck ID for replay attacks

3

Encrypt message with topic keyCompute Message Digest(MD)Sign MD and message IDPublish Message

1 Request permission to publish

5 Request permission to subscribe

8

Verify SignatureVerify Permissions for SubscribingCheck integrity by verifying MDCheck ID for replay attacks



Security Results• Experiments performed on a Pentium-3, 1.5GHz,

512 MB RAM.

• JRE 1.4.1, Cryptographic provider is IAIK

• Points in the graphs represent the average value of the operation being performed 1000 times.

• Results indicate that the security scheme does not introduce unacceptable delays.– Since messages are encrypted only once, costs are

amortized during traversal over multiple broker hops.



0

2

4

6

8

10

12

14

16

18

500 1000 1500 2000 2500 3000 3500 4000 4500 5000

Tim

e in

Mill

ise

con

ds

Data Size in Bytes

Encryption/decryption times for symmetric key algorithms

192-bit 3DES Decryption 128-bit AES Decryption

192-bit 3DES Encryption 128-bit AES Encryption



1

10

100

1000

10000

500 1000 1500 2000 2500 3000 3500 4000 4500 5000

Tim

e in

Mic

rose

con

ds

Data Size in Bytes

Time to compute Message Digests and Signing of messages using SHA-1 and RSA algorithm

Compute Digest RSA Sign



Conclusions• NaradaBrokering is a messaging infrastructure

that is appropriate for P2P Grids.

• Results demonstrate that it can be used for synchronous and asynchronous applications.

• Availability of multiple matching engines provides for sophisticated interactions between entities within the system.



Future Work• Federation of P2P systems and accompanying

search/query/response mechanisms. – Ongoing work with Limewire (Gnutella)

• Distributed A/V conferencing management.

• Dynamic Resource Management

• Management of lightweight XML databases– Ongoing investigations with Apache Xindice and

Source Forge eXist.

shrideep pallickara and geoffrey fox

Documents

broker subscriptions

p2p systems

best brokerhops

set of destinations

p2p gridsmiddleware

asynchronous systems

stored profiles

p2p interactions jxta