presented by: prof mark baker

8th December 2009

mark.baker@computer.org

Presented by: Prof Mark Baker

SSE, University of Reading Tel: +44 118 378 8615

E-mail: Mark.Baker@computer.org Web: http://acet.rdg.ac.uk/~mab

Using a RESTful messaging and registry system to support a

range a distributed applications

http://acet.rdg.ac.uk/projects/tycho

Outline• General Introduction.• Representational State Transfer (REST).• Tycho’s motivation, architecture and deployment.• Some Tycho applications:

– GridRM,– Necho,– SORMA,– Map Service.– LinkSphere.

• Utilities.• Summary and future work.

8th December 2009

mark.baker@computer.org

General Introduction• Various technologies seem to appear in

waves, some are taken up and are successful, and others die out quickly.

• I have been working in the parallel, distributed computing and HPC arena for 20+ years:– Seen lots of interesting technologies come and go!

• CORBA, Jini… etc…– Spent a lot of time working on grid technologies

(1995 onwards) and e-Science… now interested in e-Research!

• However, the Web 2.0 area seems to have been one of those domains of interest that has taken off like a rocket!

mark.baker@computer.org8th December 2009

Representational State Transfer (REST)

• The representational state transfer (REST), was originally described by Roy Fielding in his doctoral dissertation.

• REST is a collection of principals that are technology independent, except for the requirement that it be based on HTTP:– All components of the system communicate through

interfaces with clearly defined methods and dynamic code.– Each component is uniquely identified through a link (URL).– A client/server architecture is followed (web browser and

server).– All communication are stateless.– The architecture is tiered, and data can be cached at any

layer.

8th December 2009

mark.baker@computer.org

RESTful• These principals map directly to those used in the

development of the Web, and account for much of its success.

• The HTTP protocol, its interface of methods (GET, POST, HEAD, and so on), the use of URLs, HTML, and JavaScript, all map directly to these principals.

• Overall, REST can be described as a technology and platform-independent architecture where loosely coupled components communicate via interfaces over standard Web protocols.

• The software, hardware, and data-centric designs maximise system efficiency, scalability, and network throughput.

8th December 2009

mark.baker@computer.org

Tycho - motivation• Tycho was conceived in 2003 in response to a need

by the resource-monitoring project for a “light-weight”, scalable and easy to use wide-area distributed registry and messaging system.

• Tycho’s was released in 2006; a number of modifications have been made to the system to make it easier to use and more flexible.

• Tycho has been utilised across a number of application domains including – a listing is shown at the end of the talk.

• Tycho can be found here… – http://acet.rdg.ac.uk/projects/tycho/

8th December 2009

mark.baker@computer.org

8th December 2009

mark.baker@computer.org

Tycho• We decided that we wanted system that did

communication and a had a registry…• Tycho is a based a Service Oriented Architecture – it

uses the publish, subscribe and bind paradigm. • Design Philosophy:

– The system has an architecture similar to the Internet, where every node provides reliable core services, and the complexity is kept, as far as possible, to the edges.

– Tycho’s core is small, simple and efficient, so that it has a minimal memory foot-print, is easy to install, and is capable of providing robust and reliable services.

– More sophisticated services can then be built on this core and are provided via libraries and tools to applications.

• This allows Tycho to be flexible and extensible so that it will be possible to incorporate additional features and functionality.

Tycho’s Architecture

• Producers, Consumers and Mediators.• Mediators create virtual registry.

8th December 2009

mark.baker@computer.org

Tycho Deployment• Mediators allow producers and consumers to

discover each other and establish remote communications:– URL published in registry,– P2P virtual registry (VR).– Consumers can search for producers in VR.– Adding functionality to registry – metadata and Semantic

Web technologies.

• Consumers typically subscribe to receive information or events from producers.

• Producers gather and publish information for consumers:– Creating producer/mediator pairs,– Gather metadata from source and publishing it in VR.

8th December 2009

mark.baker@computer.org

8th December 2009

mark.baker@computer.org

Tycho’s Deployment• Tycho consists of the

following components:– Mediators that allow

producers and consumers to discover each other and establish remote communications,

– Consumers that typically subscribe to receive information or events from producers,

– Producers that gather and publish information for consumers.

• There is an asynchronous messaging API.

• In Tycho, producers and/or consumers (clients) can publish their existence in a Virtual Registry (VR).

Registry

Consumer

Registry

Core

Mediator

Producer

Core

Consumer Producer

WAN (HTTP)

LAN (Socket)

WAN (P2P)

Tycho Applications

8th December 2009

mark.baker@computer.org

GridRM• GridRM is an extensible, wide-area, monitoring

system that specialises in combining data from existing agents and monitoring systems so that a consistent view of the underlying resources and services can be achieved, regardless of heterogeneity.

• Gateways provide access to local resource information at each site.

• Clients connect to gateways to perform resource queries and to subscribe for events.

• GridRM uses Tycho in a number of ways to bind together clients and Gateways for wide-area communications, and to provide the basis of an event mechanism (both wide-area events to clients, and events from local monitoring).8th December 2009

mark.baker@computer.org

8th December 2009

mark.baker@computer.org

GridRM Structure• Global layer of peer-related gateways:

– Which in turn have a local layer that interacts with the local data sources, and/or a hierarchy child gateways.

GatewayGateway

Gateway

Gateway

XDB• Integrated search facilities

across multiple archival databases.

• The test system searches for records from the Silchester IADB, held in an ordinary relational database, and a classics-oriented collection of Roman-era inscriptions found in Vindolanda database, which is based on a RDF store.

• The cross-database search engine is now a key component in http://LinkSphere.org

8th December 2009

mark.baker@computer.org

Necho• The Necho project is creating a multi-tiered peer-to-

peer system, which is akin to BitTorrent, for distributing multi-terabyte scientific datasets across the Internet.

• The concepts for this project first appeared when working with the Sloan Digital Sky Survey (multi-terabyte), where it was necessary to split the original dataset up and use a modified version of WGET to download and update the database.

• It consists of a hierarchical Peer-to-Peer (P2P) system that is based around shared-portal services and unique peers donated by participating individuals and organisations.

8th December 2009

mark.baker@computer.org

Necho• The goal of the project is to combine P2P, volunteer

computing and social networks to provide a way to distribute, contribute to, and manage very large datasets.

• Necho uses Tycho to distribute, index and retrieve chunks of data that comprise each overall dataset.

• We are currently testing Necho against Azureus, our single tier version of Necho is proving to be much faster.

8th December 2009

mark.baker@computer.org

SORMA (Self-Organizing ICT Resource Management)

8th December 2009

mark.baker@computer.org

SORMA (Self-Organizing ICT Resource Management)

• The Economically Enhanced Resource Manager (EERM) exists at each resource provider’s site, and acts as a centralised resource allocator (e.g. business goals and resource requirements) in order to achieve maximum economic profit and resource utilisation.

• The EERM utilises GridRM to obtain resource information for system and per-process monitoring in order to determine if Service Level Agreements (SLAs) have been violated.

• The EERM is composed as a confederation of loosely bound components for scalability and availability reasons.

• Tycho provides messaging and event mechanisms.

8th December 2009

mark.baker@computer.org

SORMA (Self-Organizing ICT Resource Management)

8th December 2009

mark.baker@computer.org

Map Service• Geographical maps are

used to describe the Earth’s surface and its contours: – These maps are hosted on

servers called “map servers” around the globe and can be used by scientists in many ways.

• The environmental science community lacks a searchable registry of available Map Services.

• Created Services to find these service via Tycho.

8th December 2009

mark.baker@computer.org

LinkSphere.org

8th December 2009

mark.baker@computer.org

LinkSphere.org

• LinkSphere is a JISC funded project that includes – A Social Web site that helps researchers and

academics get to know each other.• The site has fine-grained security.

– We are also joining together databases around the University.

8th December 2009

mark.baker@computer.org

LinkSphere.org• We are joining the various databases is based on

the XBD project.• The idea is instead of the databases being

remote, we can search across them, and create a list, that has a ranking.

• This means that researchers will be able to do useful searches, plus they will get links to ePrints, and books in the Library.

• The idea is to make it easier to R&D within the University.

• The underlying infrastructure is based on Tycho, which includes metadata and URIs too.

8th December 2009

mark.baker@computer.org

Tycho Utilities• BonJour - Locates devices, these devices offer a

local network using multicast DNS records.• Synchronous API - Many application need blocking

communications, so we have add this to Tycho.• HTTP pipelining - To optimise the communication

performance, it requires us to open multiple parallel pipes, with buffering, when sending data to a remote destination.

• Lightweight Transactions - There is a transaction manager, which uses main memory that decouples the performance of transactions from the disk.

• Additional Caching - to improve performance by altering caching in the mediator to include local data-store queries in addition to remote responses.

8th December 2009

mark.baker@computer.org

Summary• Tycho is a RESTful asynchronous messaging system

with an integrated peer-to-peer virtual registry. • Since, Tycho was released in 2006, we have

increasingly used the system to support a range of distributed applications.

• We have used Tycho as RESTful services are easy to install and use.

• Tycho uses HTTP (HTTPS) and Sockets (SSL) for communications.

• Internally, it uses SQL as the query language and uses LDAP LDIF to mark up responses from the VR.

• None of these standards have changed, and it ensures that applications, based on Tycho, will continue to work for the foreseeable future.

8th December 2009

mark.baker@computer.org

List of Applications– GridRM,– VOTEchBroker,– Slogger,– XDB,– DNWAy– Necho,– SORMA,– Map Service,– LinkSphere.org,– Sun SPOT Wireless Sensor Networks.

8th December 2009

mark.baker@computer.org