cern the status of eu datagrid project presented by bob jones cern technical coordinator...
Post on 27-Mar-2015
220 Views
Preview:
TRANSCRIPT
CERNThe Status of EU DataGrid projectThe Status of EU DataGrid project
Presented by Bob Jones CERN
Technical Coordinator
bob.jones@cern.ch
EU DataGrid 2
CERN
November 2001
Main project goals and characteristicsMain project goals and characteristics
To build a significant prototype of the LHC computing model To collaborate with and complement other European and US
projects To develop a sustainable computing model applicable to other
sciences and industry: biology, earth observation etc. Specific project objectives
Middleware for fabric & Grid management (mostly funded by the EU) evaluation, test, and integration of existing M/W S/W and research and development of new S/W as appropriate
Large scale testbed (mostly funded by the partners)
Production quality demonstrations (partially funded by the EU)
Open source and technology transfer Global GRID Forum
Industry and Research Forum
EU DataGrid 3
CERN
November 2001
Main PartnersMain Partners
CERN – International (Switzerland/France)
CNRS - France
ESA/ESRIN – International (Italy)
INFN - Italy
NIKHEF – The Netherlands
PPARC - UK
EU DataGrid 4
CERN
November 2001
ParticipantsParticipants
Main partners: CERN, INFN(I), CNRS(F), PPARC(UK), NIKHEF(NL), ESA-Earth Observation
Other sciences: KNMI(NL), Biology, Medicine
Industrial participation: CS SI/F, DataMat/I, IBM/UK
Associated partners: Czech Republic, Finland, Germany, Hungary, Spain, Sweden (mostly computer scientists)
Formal collaboration with USA established
Industry and Research Project Forum with representatives from
Denmark, Greece, Israel, Japan, Norway, Poland, Portugal, Russia, Switzerland
EU DataGrid 5
CERN
November 2001
Project ScopeProject Scope
9.8 M Euros EU funding over 3 years
90% for middleware and applications (HEP, EO and biology)
Three year phased developments & demos (2001-2003)
Possible extensions (time and funds) on the basis of first successful results:
DataTAG (2002-2003)
CrossGrid (2002-2004)
GridStart (2002-2004)
…
EU DataGrid 6
CERN
November 2001
Programme of workProgramme of work
MiddlewareWP1 Grid Workload Management F. Prelz/INFNWP2 Grid Data Management P. Kunszt/CERNWP3 Grid Monitoring services S. Fisher/RALWP4 Fabric Management O. Barring/CERNWP5 Mass Storage Management J. Gordon/RAL
TestbedWP6 Testbed Integration F. Etienne/CNRSWP7 Network Services C. Michau/CNRS
Scientific ApplicationsWP8 HEP Applications F. Carminati/CERNWP9 Earth Observation Applications L. Fusco/ESA-ESRINWP10 Biology Applications C. Michau/CNRS
Dissemination WP11 M. Lancia/CNR
Project Management WP12 F. Gagliardi/CERN
EU DataGrid 15
CERN
November 2001
Status Status
EU contract signed on December 29th, 2000
Project started on 1/1/2001
Work ramping up at CERN and the collaborating institutes (Globus
initial installation, tests and prototype production)
International test bed infrastructure being deployed
Architecture Task Force produced 2nd version of Architecture doc.
First (internal milestone) at PM9 - test-bed 1
EU DataGrid 16
CERN
November 2001
Test Bed ScheduleTest Bed Schedule
TestBed 0 (early 2001)
International test bed 0 infrastructure deployed
Globus 1 only - no EDG middleware
TestBed 1 ( now )First release of EU DataGrid software to defined users within the project:
HEP experiments (WP 9)
Biology applications (WP 10)
Earth Observation (WP 11)
TestBed 2 (Sept. 2002)
Builds on TestBed 1 to extend facilities of DataGrid
TestBed 3 (March 2003) & 4 (Sept 2003)
EU DataGrid 17
CERN
November 2001
DataGrid statusDataGrid status
Preliminary architecture definedEnough to deploy testbed 1
First M/W delivery(GDMP, first workload management system, fabric
management tools, Globus installation, including certification and authorization, Condor tools)
First application test cases ready, long term cases defined
Integration team actively building Testbed 1 WP8
WP8
WP9
WP9
WP10
PierGiorgio Cerello
Eric Van Herwijnen
Julian Lindford
Andrea Parrini
Yannick Legre
WP6
WP6
WP6
WP6
WP6
WP6
WP6
WP6
WP6
WP6
WP6
Brian Coghlan
Flavia Donno
Eric Fede
Fabio Hernandez
Nadia Lajili
Charles Loomis
Pietro Paolo Martucci
Andrew McNab
Sophie Nicoud
Yannik Patois
Anders Waananen
WP1
WP2
WP3
WP4
WP5
WP7
Elisabetta Ronchieri
Shahzad Muzaffar
Alex Martin
Maite Barroso Lopez
Jean Philippe Baud
Frank Bonnassieux
EU DataGrid 18
CERN
November 2001
Collective ServicesCollective Services
Information &
Monitoring
Information &
Monitoring
Replica ManagerReplica
ManagerGrid
SchedulerGrid
Scheduler
Local ApplicationLocal Application Local DatabaseLocal Database
Underlying Grid ServicesUnderlying Grid Services
Computing Element Services
Computing Element Services
Authorization Authentication and Accounting
Authorization Authentication and Accounting
Replica CatalogReplica Catalog
Storage Element Services
Storage Element Services
SQL Database Services
SQL Database Services
Fabric servicesFabric services
ConfigurationManagement
ConfigurationManagement
Node Installation &Management
Node Installation &Management
Monitoringand
Fault Tolerance
Monitoringand
Fault Tolerance
Resource Management
Resource Management
Fabric StorageManagement
Fabric StorageManagement
Grid
Fabric
Local Computing
Grid Grid Application LayerGrid Application Layer
Data Management
Data Management
Job Management
Job Management
Metadata Management
Metadata Management
Object to File Mapping
Object to File Mapping
Service Index
Service Index
EU-DataGrid Architecture
EU DataGrid 19
CERN
November 2001
Test bed 1 ApproachTest bed 1 Approach
Software integration
combines software from each middle-ware work package and underlying external tool kits (e.g. Globus)
performed by integration team at CERN on a cluster of 10 Linux PCs
Basic integration tests
performed by integration team to verify basic functionality
Validation tests
application groups use testbed 1 to exercise their application software
e.g. LHC experiments run jobs using their offline software suites on test-bed 1 sites
EU DataGrid 20
CERN
November 2001
Detailed TestBed 1 ScheduleDetailed TestBed 1 Schedule October 1:
Intensive integration starts
Based on Globus 2 November 1:
First beta release of DataGrid (CERN & Lyon)
(depends on changes needed Globus 1->2) November 15:
Initial limited application testing finished
DataGrid ready for deployment on partner sites (~5 sites) November 30:
Widespread deployment
Code machines split for development
Testbed 1 open to all applications (~40 sites) December 29: WE ARE DONE!
EU DataGrid 21
CERN
November 2001
TestBed 1 SitesTestBed 1 Sites
First round (15 Nov.)
CERN, Lyon, RAL, Bologna
Second Round (30 Nov.)
Netherlands: NIKHEF
UK: See John Gordon’s talk
Italy: 6-7 sites: Catania, Legnaro/Padova, Milan, Pisa, Rome, Turin, Cagliari?
France: Ecole-Polytechnique
Russia: Moscow
Spain: Barcelona?
Scandinavia: Lund?
WP9 (GOME): ESA, KNMI, IPSL, ENEA
EU DataGrid 22
CERN
November 2001
Licenses & CopyrightsLicenses & Copyrights
Package Repository and web siteProvides access to the packaged Globus, DataGrid and required external software
All software is packaged as source and binary RPMs
Copyright StatementCopyright (c) 2001 EU DataGrid – see http://www.edg.org/license.html
LicenseWill be the same (or very similar) to Globus license
A BSD-style license which puts few restrictions on use
Condor-G (used by WP1)Not open source or redistributable
Through special agreement, can redistribute within DataGrid
LCFG (used by WP4)Uses GPL
EU DataGrid 23
CERN
November 2001
SecuritySecurity
The EDG software supports many Certification Authorities from the various partners involved in the project
http://marianne.in2p3.fr/datagrid/ca/ca-table-ca.html
but not Globus CA
For a machine to participate as a Testbed 1 resource all the CAs must be enabled.
all CA certificates can be installed without compromising local site security
Each host running a Grid service needs to be able to authenticate users and other hosts
site manager has full control over security for local nodes
Virtual Organisation represents a community of users6 VOs for testbed 1: 4 HEP (ALICE, ATLAS, CMS, LHCb), 1 EO, 1 Biology
EU DataGrid 24
CERN
November 2001
ldxprofldxprof
GenericComponent
GenericComponent
rdxprofrdxprof
LCFG Components
DBM File
LCFG configuration files
mkxprofWeb ServerXML Profile
(one per client node)
Server node
HTTP
Client nodes
Node configuration and installation toolsNode configuration and installation toolsNode configuration toolsNode configuration tools
For reference platform (Linux RedHat 6.2)
Initial installation tool using system image cloning
LCFG (Edinburgh University) for software updates and maintenance
EU DataGrid 25
CERN
November 2001
Middleware componentsMiddleware components
Job Description Language (JDL)
script to describe the job parameters
User Interface (UI)
sends the job to the RB and receives
the results
Resource Broker (RB)
locates and selects the target
Computing Element (CE)
Job Submission Service (JSS)
submits the job to the target CE
Logging and Book-keeping (L&B )
records job status information
Grid Information Service (GIS)
Information Index about state of
Grid fabric
Replica Catalog
list of data sets and their duplicates
held on Storage Elements (SE)
EU DataGrid 26
CERN
November 2001
A Job Submission ExampleA Job Submission Example
UIJDL
Logging &Logging &Book-keepingBook-keeping
Job SubmitEvent
ResourceResourceBrokerBroker
Output Sandbox
Input Sandbox
Job SubmissionJob SubmissionServiceService
StorageStorageElementElement
ComputeComputeElementElement
Brokerinfo
Output Sandbox
Input Sandbox
Information Information ServiceService
Job Status
ReplicaReplicaCatalogueCatalogue
EU DataGrid 27
CERN
November 2001
Iterative ReleasesIterative Releases
Planned intermediate release schedule
TestBed1: October 2001
Release 1.1: January 2002
Release 1.2: March 2002
Release 1.3: May 2002
Release 1.4: July 2002
TestBed 2: September 2002
Similar schedule will be organised for 2003 Each release includes
feedback from use of previous release by application groups
planned improvements/extension by middle-ware WPs
use of software infrastructure
feeds into architecture group
EU DataGrid 28
CERN
November 2001
Software InfrastructureSoftware Infrastructure
Toolset for aiding the development & integration of middle-warecode repositories (CVS)
browsing tools (CVSweb)
build tools (autoconf, make etc.)
document builders (doxygen)
coding standards and check tools (e.g. CodeChecker)
nightly builds
Guidelines, examples and documentationshow the software developers how to use the toolset
Development facilitytest environment for software (small set of PCs in a few partner sites)
Provided and managed by WP6setting-up toolset and organising development facility
EU DataGrid 29
CERN
November 2001
Future PlansFuture Plans
Tighter connection to applications principal architects
Closer integration of the software components
Improve software infrastructure toolset and test suites
Evolve architecture on the basis of TestBed results
Enhance synergy with US via DataTAG-iVDGL and InterGrid
Promote early standards adoption with participation to GGF WGs
First project EU review end of February 2002
Final software release by end of 2003
top related