v r e a virtual research environment (extending the grid to the desktop) rob crouchley (university...
TRANSCRIPT
V R
E
A Virtual Research Environment
(Extending the Grid to the Desktop)
Rob Crouchley (University of Lancaster)Rob Allan (CCLRC Daresbury Laboratory)
VRE
A vision by Daresbury and Lancaster
V R
EWhy have a VRE ?
“to make the use of e-Science technologies, methodologies and resources easier and more transparent for users than simply developing bespoke applications on a generic infrastructure toolkit (such as Globus GT2 or OGSI/WSRF).” We need to:
• Bridge the gap between different types of technology (database management, computational methods, sensor Grids, networks, Condor resources, visualisation systems, collaborative working, Access Grid, etc.);
• Provide an environment to enhance the programmability and usability of such a Grid by integrating work from a number of ongoing research projects;
• Add value to the Grid by implementing a VRE on the JCSR clusters and resources at other e-Science Centres;
• Include Grid-based tools for research collaboration.
V R
EPerceived problems with using the GRID
•Currently requires heroic effort to use it;•GT2 is very complicated and difficult to install,
requires root privileges to install it and various firewall ports to be open, duplicates some system libraries;
•Functionality nevertheless limited;•Can make other University services vulnerable if not
properly managed;•User requirements not fully articulated;•Human factors not addressed, needs familiar GUI,
pull down menus, etc.
Requirements gathering now high on the agenda (JISC, ETF, UTF, …)
V R
EThe Grid “Client Problem”
Grid Core
Consumer clients: PC, TV, video, AG
Workplace: desktop clients
Portable clients: phones, laptop, pda, data entry…
Middleware
e.g. Globus
Grid Core
Many clients want to access a few Grid-enabled resources
V R
EInstitutions need Autonomy and Security
Host – client relationship
Example solution suggested by
Web server - browser
Communication must be initiated by client because of firewall around client’s institution. Can use a proxy or gateway.
V R
ESo a Virtual Research Environment?
Requirements:
Easy to install
Familiar interface
Personalisable
Work through firewalls
Extensible functionality
Persistent
Pervasive
Secure
V R
EOptions
• Provide heavyweight functionality (Globus?), but only on Grid-enabled hosts;
• Implied need for client-server software architecture, e.g.:– Web-based portal with familiar browser– Client programming library, API in C, C++ Java,
Perl, Python, R etc. – Ability to link to existing applications/ GUIs– Command-based shell interface– Drag and Drop interface (a la Mac)
• Need a published set of services on Grid hosts – OGSA model, registry, semantics;
• Need easy development and deployment framework for applications and client tools, e.g. using Web services - encourage community contribution via an open process.
V R
EHuman Factors
• Growing recognition of the need to design a behaviourally appropriate interface to the Grid;
• e.g. Rick Stevens’ Access Grid and work on human factors issues;
• Lot of industrial knowledge here, ergonomics etc. needs to be built on;
• Usability Task Force will take a lead;• Job of scientists already hard – need tools that do not
make it harder!
V R
E
Digital libraries & documents
groups-to-information
groups-to-facilities
people-to-people
Communication,Collaboration
Services
Distributed,media-richinformationtechnology
Remote instruments
http://www.scienceofcollaboratories.org/
Science of Collaboratories
NSF Funded ITR
V R
ESome Tools to aid Collaborations
• Phone, e-mail…• WIKI – Web based on-the-fly multi-editorial document
management using hypertext• Forum – Web based threaded mail discussion and archives• Chandler – personal information management system: mail,
calendar, contact list, tasks, repositories, shared docs, etc.• Microsoft Outlook – ditto, but Windows based• BSCW –Web based shared workspace system, document
upload, event notification, group management and much more• GridSite – Web based secure document sharing and multi-
editorial• PHPNuke – news, messages, topics, resources, mail, links, etc.
– used for HPCPortal and InfoPortal v2• Access Grid – on-line multicast meetings and shared
presentations• NetMeeting – Microsoft tool for on-line meetings• CHEF – workgroup based system with chat, resources, news
etc. – used in ReDReSS
V R
ESome basic VRE Functions
VRE must take care of many things behind the scenes:
• Authentication and authorisation (Shibboleth and Permis in line with JISC proposals…);
• Shared development of content by staff using content management and editing tools:– Access to middleware resources and
documentation,– Access to training materials and resources,– Collaboration services,– Access to support, consultancy and other services
• Access to Grid Services - user access via pre-defined tools and applications to the UK e-Science Grid;
• Data access – e.g. using Storage Resource Broker;• Access to broadcasts – e.g. on the Access Grid
network;• Management functions - for experts to maintain the
system and deploy new applications.
V R
EThe VRE needs to be more than a Web
page
Why should it be different?
• Like the Web, persistent and pervasive, but:• It provides a managed environment, giving secure
access to autonomous Grid services, providing resources, based on user requirements;
• It uses diagnostic/ background data to orchestrate the material for each individual (via session management/ profiling services);
• It will be specific to the needs of groups of scientists (virtual organisations), providing new routes to e-Science;
• The technology will be easily extendable to include all new tools;
• It could be an early adopter of new WSRF/ GT4 and portlet standards.
V R
EExamples of Activites: 1
IeSE
IeSE: Integrated e-Science Environment for CCLRC.
•Web service interfaces + presentation APIs + Grid (via Globus GT2). Hosted on IBM BladeCenter at Daresbury.
•HPCPortal – services for Grid resources and applications
•DataPortal – search and access remote data repositories
•GapTk – scientific visualisation services•InfoPortal – Grid information services•GROWL – lightweight C and R library interfaces
Customised for e-Science Pilot projects: e-Minerals, e-Materials, e-HTPX, SABRE-R, ETF, NGS (soon?), CCLRC Facilities, etc.
V R
EExamples of IeSE Portal Interfaces
HPCPortal
DataPortal
V R
EExamples of Activites: 2
SakaiCollaborative framework for higher education institutions
to develop and share open source software.
•Principally aimed at educational portal development, course management, workgroup management, etc. Adopted by U. Michigan, Indiana U., MIT, Stanford etc.
•Easily customised for e-Science projects, e.g. NEESGrid•Open Knowledge Initiative OSID (Open Services
Interface Definitions)•Research support collaborative system•Workflow engine•Tool portability profile•Funding
– $4.4M in institutional staff (27 FTE)– $2.4M Mellon Foundation– Additional investment through partners
Built on Java portlet standard JSR-168 plus CHEF v2/ uPortal v3 framework
V R
ESakai Timelines
19981991 - 1997 1999 2000 2001 2002 2003 2004 2005
SPARC
Science of Collaboratories
Sakai
Worktools (Notes Based) WTNG
Coursetools (Notes Based) CTNG
CHEF 1 CHEF 2
NMI Grid Portal
NEESGrid
Portal TechnologyJetspeed 2.0uPortal 3.0
Websphere É
Channels, Teamlets
JSR-168 Portlets
CHEF Services
JSR-168 Technology
OKI Services
Legacy
SakaiTeamlet
OtherServices
Sakai GUI
SakaiTeamlet
Sakai GUI
Java Swing
V R
EExamples of Activities: 3
ARDA
ARDA: Architectural Roadmap towards Distributed Analysis.
•LCH Computing Grid Report – CERN-LCG-2003-033;•Builds on existing software - e.g. AliEn portal;•Assesses future user requirements for LCG
application area;•Build and extend Grid/ database services;•Provide application frameworks, shells, APIs,
interactive GUIS, portals etc.
Proposed as an example component of the EGEE work programme for the EU Grid.
V R
EARDA: Key Services for Distributed Analysis
Information Service
Authentication
Authorisation
Audi ting
Grid Monitoring
Workload Management
Metadata Catalogue
File Catalogue
Data Management
Computing Element
Storage Element
Job Monitor
Job Provenance
Package Manager
DB Proxy
User Interface
API
Accounting
7:
12:
5:
13:
8:
15: 11:
9: 10:
1:
4:
2:
3:
6:
14:
V R
EARDA: API and User Interface
API
+ Authentication+ Data Management
+ Grid Service Management+ Job Control
+ Metadata Management+ NewInterface
+ Posix I/O
SOAP
(from API)
Grid File Access
(from API)
Experiment Frameworks
POOL/ROOT/...
(from Experiment Frameworks)...)
API (OGSI User Interface Factory)
Storage Element (POSIX I/O service)
Portals
Grid Shells
Grid File System
V R
EObjectives of the VRE
• To deploy a framework and extensible set of services developed by the e-Science and Information communities;
• To provide customisable portals for projects using these and other bespoke services;
• To develop a Grid client toolkit for application developers based on current on Grid services;
• To link together applications from many research domains;
• To put semantic, discovery and compositional interfaces in place as tools to create such a rich environment;
• To link active services and sources of information to generate and exploit new knowledge. Project plan
V R
EScoping a VRE Project:
Phase I (year 1)
•£10k to perform a comprehensive review of research requirements and initial tests of alternative software;
•£100k to do some critical ground work and provide some support for others who want to use portals e.g. Sakai.
Critical Ground work:Sakai may already be the front runner, UK could
become founder member ($10k p.a. for 3 years); Extend existing portals for NGS (JCSR clusters);Integrate AGN and video delivery portlets (based on
current work with Geoffrey Fox) , enable joint working on documents presentations etc.
Extend CHEF functionality to create testbed, with HPCPortal services, etc.
V R
EEnhancing the CHEF Collaborative
Environment
• We have already added video conferencing for joint working on the same document;
• Will link into AGN and other portlets (collaboration with Geoffrey Fox, Indiana).
V R
EScoping a VRE Project:
Phase II (year 2)
•Builds on phase I;•£300k would get more ground work and establish
links to other resources, e.g. RDN via SPP portal etc…•£600k would get a foot in the door with international
partnerships and technical contributions to activities such as Sakai;
•Framework support for VRE and VLE;•Additional customised support for users of NGS.
Use open source software and standards such as JSR-168 and WSRP. Different groups may go different ways but software can still be shared. JISC may choose to extend scoping study and small-scale demonstrator based on requirements collection and prototyping such as currently being carried out at Daresbury and Lancaster. It is already becoming clear that the Sakai framework may be the front runner.
V R
EScoping a VRE Project:
Phase III (year 3)•Builds on phase II;•£X gets a functional e-Science VRE - scoping study
will determine X;•New services and clients customised and re-usable
for UK projects;•Integrate JISC IE and e-Science/ GridPP Grids;•Includes some VLE functionality;•Plug and play for diverse community requirements;•Extendable, depending on user feedback, e.g. text
mining;•Ensure contribution to standards and components to
worldwide fora, e.g. GGF, ???.
This work would establish an operational framework and suite of tools with a UK “stamp”. It would consolidate e-Science and Information Env. work and allow contribution of new tools.
V R
E
Middleware/Software Library
Access GRID
Security Authorisation Authentication
Text Mining/ Data services
UK GRID Services
D
JJISC PortalJISC Portal
Portal Management
Semantic GRID Services
VLE Portal VRE
Portal
Awareness Raising Resources
Workshops
Possible functionality/ content of a VRE
V R
E
Lancaster•CHEF framework•ReDReSS
Daresbury•IeSE framework•HPCPortal, etc.•DataPortal•GROWL
Bristol, Oxford, Bath•SPP Portal
•Access Grid
•E-Science Grid•Tools and resources•National Grid Service•Links to EGEE
•GridPP•Tools and resources•Links to EGEE
Phase IIIVRE 2.0 Release•Tool Portability Profile•Framework•Services-based Portal
VRE Tools•Complete CMS•Workflow•Research Tools•Authoring Tools•Advanced search tools•Advanced Grid tools
Primary VRE ActivityRefining VRE Framework,
Tuning and conforming additional tools
Intensive community building/training
Activity: Ongoing implementation work at local institutions…
Jan 04 Jan 05 Jan 06 Dec 06
Activity: Maintenance &
Transition from aproject to
a community
Phase IIVRE 1.0 Release•Tool Portability Profile•Framework•Services-based Portal•Refined OSIDs & implementations
VRE Tools•Complete CMS•Collaborations•Basic Grid tools•Cross sesource searches•Support for VLE
Link to SAKAI ActivityArchitecting for JSR-168 Portlets,
Refactoring “best of” features for toolsConforming tools to Tool Portability Profile
Develop a VRE framework
Possible VRE Timelines
Sakai source availableConverge with EGEE