presented by: prof mark baker
DESCRIPTION
Using a RESTful messaging and registry system to support a range a distributed applications http://acet.rdg.ac.uk/projects/tycho. Presented by: Prof Mark Baker SSE, University of Reading Tel: +44 118 378 8615 E-mail: [email protected] Web: http://acet.rdg.ac.uk/~mab. Outline. - PowerPoint PPT PresentationTRANSCRIPT
8th December 2009
Presented by: Prof Mark Baker
SSE, University of Reading Tel: +44 118 378 8615
E-mail: [email protected] Web: http://acet.rdg.ac.uk/~mab
Using a RESTful messaging and registry system to support a
range a distributed applications
http://acet.rdg.ac.uk/projects/tycho
Outline• General Introduction.• Representational State Transfer (REST).• Tycho’s motivation, architecture and deployment.• Some Tycho applications:
– GridRM,– Necho,– SORMA,– Map Service.– LinkSphere.
• Utilities.• Summary and future work.
8th December 2009
General Introduction• Various technologies seem to appear in
waves, some are taken up and are successful, and others die out quickly.
• I have been working in the parallel, distributed computing and HPC arena for 20+ years:– Seen lots of interesting technologies come and go!
• CORBA, Jini… etc…– Spent a lot of time working on grid technologies
(1995 onwards) and e-Science… now interested in e-Research!
• However, the Web 2.0 area seems to have been one of those domains of interest that has taken off like a rocket!
[email protected] December 2009
Representational State Transfer (REST)
• The representational state transfer (REST), was originally described by Roy Fielding in his doctoral dissertation.
• REST is a collection of principals that are technology independent, except for the requirement that it be based on HTTP:– All components of the system communicate through
interfaces with clearly defined methods and dynamic code.– Each component is uniquely identified through a link (URL).– A client/server architecture is followed (web browser and
server).– All communication are stateless.– The architecture is tiered, and data can be cached at any
layer.
8th December 2009
RESTful• These principals map directly to those used in the
development of the Web, and account for much of its success.
• The HTTP protocol, its interface of methods (GET, POST, HEAD, and so on), the use of URLs, HTML, and JavaScript, all map directly to these principals.
• Overall, REST can be described as a technology and platform-independent architecture where loosely coupled components communicate via interfaces over standard Web protocols.
• The software, hardware, and data-centric designs maximise system efficiency, scalability, and network throughput.
8th December 2009
Tycho - motivation• Tycho was conceived in 2003 in response to a need
by the resource-monitoring project for a “light-weight”, scalable and easy to use wide-area distributed registry and messaging system.
• Tycho’s was released in 2006; a number of modifications have been made to the system to make it easier to use and more flexible.
• Tycho has been utilised across a number of application domains including – a listing is shown at the end of the talk.
• Tycho can be found here… – http://acet.rdg.ac.uk/projects/tycho/
8th December 2009
8th December 2009
Tycho• We decided that we wanted system that did
communication and a had a registry…• Tycho is a based a Service Oriented Architecture – it
uses the publish, subscribe and bind paradigm. • Design Philosophy:
– The system has an architecture similar to the Internet, where every node provides reliable core services, and the complexity is kept, as far as possible, to the edges.
– Tycho’s core is small, simple and efficient, so that it has a minimal memory foot-print, is easy to install, and is capable of providing robust and reliable services.
– More sophisticated services can then be built on this core and are provided via libraries and tools to applications.
• This allows Tycho to be flexible and extensible so that it will be possible to incorporate additional features and functionality.
Tycho’s Architecture
• Producers, Consumers and Mediators.• Mediators create virtual registry.
8th December 2009
Tycho Deployment• Mediators allow producers and consumers to
discover each other and establish remote communications:– URL published in registry,– P2P virtual registry (VR).– Consumers can search for producers in VR.– Adding functionality to registry – metadata and Semantic
Web technologies.
• Consumers typically subscribe to receive information or events from producers.
• Producers gather and publish information for consumers:– Creating producer/mediator pairs,– Gather metadata from source and publishing it in VR.
8th December 2009
8th December 2009
Tycho’s Deployment• Tycho consists of the
following components:– Mediators that allow
producers and consumers to discover each other and establish remote communications,
– Consumers that typically subscribe to receive information or events from producers,
– Producers that gather and publish information for consumers.
• There is an asynchronous messaging API.
• In Tycho, producers and/or consumers (clients) can publish their existence in a Virtual Registry (VR).
Registry
Consumer
Registry
Core
Mediator
Producer
Core
Consumer Producer
WAN (HTTP)
LAN (Socket)
WAN (P2P)
GridRM• GridRM is an extensible, wide-area, monitoring
system that specialises in combining data from existing agents and monitoring systems so that a consistent view of the underlying resources and services can be achieved, regardless of heterogeneity.
• Gateways provide access to local resource information at each site.
• Clients connect to gateways to perform resource queries and to subscribe for events.
• GridRM uses Tycho in a number of ways to bind together clients and Gateways for wide-area communications, and to provide the basis of an event mechanism (both wide-area events to clients, and events from local monitoring).8th December 2009
8th December 2009
GridRM Structure• Global layer of peer-related gateways:
– Which in turn have a local layer that interacts with the local data sources, and/or a hierarchy child gateways.
GatewayGateway
Gateway
Gateway
XDB• Integrated search facilities
across multiple archival databases.
• The test system searches for records from the Silchester IADB, held in an ordinary relational database, and a classics-oriented collection of Roman-era inscriptions found in Vindolanda database, which is based on a RDF store.
• The cross-database search engine is now a key component in http://LinkSphere.org
8th December 2009
Necho• The Necho project is creating a multi-tiered peer-to-
peer system, which is akin to BitTorrent, for distributing multi-terabyte scientific datasets across the Internet.
• The concepts for this project first appeared when working with the Sloan Digital Sky Survey (multi-terabyte), where it was necessary to split the original dataset up and use a modified version of WGET to download and update the database.
• It consists of a hierarchical Peer-to-Peer (P2P) system that is based around shared-portal services and unique peers donated by participating individuals and organisations.
8th December 2009
Necho• The goal of the project is to combine P2P, volunteer
computing and social networks to provide a way to distribute, contribute to, and manage very large datasets.
• Necho uses Tycho to distribute, index and retrieve chunks of data that comprise each overall dataset.
• We are currently testing Necho against Azureus, our single tier version of Necho is proving to be much faster.
8th December 2009
SORMA (Self-Organizing ICT Resource Management)
• The Economically Enhanced Resource Manager (EERM) exists at each resource provider’s site, and acts as a centralised resource allocator (e.g. business goals and resource requirements) in order to achieve maximum economic profit and resource utilisation.
• The EERM utilises GridRM to obtain resource information for system and per-process monitoring in order to determine if Service Level Agreements (SLAs) have been violated.
• The EERM is composed as a confederation of loosely bound components for scalability and availability reasons.
• Tycho provides messaging and event mechanisms.
8th December 2009
Map Service• Geographical maps are
used to describe the Earth’s surface and its contours: – These maps are hosted on
servers called “map servers” around the globe and can be used by scientists in many ways.
• The environmental science community lacks a searchable registry of available Map Services.
• Created Services to find these service via Tycho.
8th December 2009
LinkSphere.org
• LinkSphere is a JISC funded project that includes – A Social Web site that helps researchers and
academics get to know each other.• The site has fine-grained security.
– We are also joining together databases around the University.
8th December 2009
LinkSphere.org• We are joining the various databases is based on
the XBD project.• The idea is instead of the databases being
remote, we can search across them, and create a list, that has a ranking.
• This means that researchers will be able to do useful searches, plus they will get links to ePrints, and books in the Library.
• The idea is to make it easier to R&D within the University.
• The underlying infrastructure is based on Tycho, which includes metadata and URIs too.
8th December 2009
Tycho Utilities• BonJour - Locates devices, these devices offer a
local network using multicast DNS records.• Synchronous API - Many application need blocking
communications, so we have add this to Tycho.• HTTP pipelining - To optimise the communication
performance, it requires us to open multiple parallel pipes, with buffering, when sending data to a remote destination.
• Lightweight Transactions - There is a transaction manager, which uses main memory that decouples the performance of transactions from the disk.
• Additional Caching - to improve performance by altering caching in the mediator to include local data-store queries in addition to remote responses.
8th December 2009
Summary• Tycho is a RESTful asynchronous messaging system
with an integrated peer-to-peer virtual registry. • Since, Tycho was released in 2006, we have
increasingly used the system to support a range of distributed applications.
• We have used Tycho as RESTful services are easy to install and use.
• Tycho uses HTTP (HTTPS) and Sockets (SSL) for communications.
• Internally, it uses SQL as the query language and uses LDAP LDIF to mark up responses from the VR.
• None of these standards have changed, and it ensures that applications, based on Tycho, will continue to work for the foreseeable future.
8th December 2009
List of Applications– GridRM,– VOTEchBroker,– Slogger,– XDB,– DNWAy– Necho,– SORMA,– Map Service,– LinkSphere.org,– Sun SPOT Wireless Sensor Networks.
8th December 2009