What is a Grid
2
In 1998, I. Foster and C. Kesselman wrote in The Grid:
Blueprint for a New Computing Infrastructure:
“A computational grid is a hardware and software infrastructure that provides dependable, consistent, pervasive access to highend computational capabilities.”
In 2002, I. Foster wrote in What is the Grid? A Three
Point Checklist:
“... The essence of the [definition] can be captured in a simple checklist,... a Grid is a system that:
1. coordinates resources that are not subject to centralized control…
2. … using standard, open, general purpose protocols and interfaces...
3. … to deliver nontrivial qualities of service.
4
Architettura e servizi di Grid e confronto con l’architettura di Internet
ip
linkfabric
application
collective
Resource
Connectivity transport
application
Inte
rnet
pro
toco
l arc
hit
ect
ure
Gri
d p
roto
col
arc
hit
ect
ure
GRAM
GSI
neck
Resource BrokerGIS, RLS..
CE, SE,..
Computer, storage, rete, apparati sperimentali…
5
OGSA capability gLite UNICORE ARC
Security.AttributeAuthority VOMS (AC+SAML) WS-UUDB, SAML-VOMS
VOMS+SAM
Security.Accounting DGAS, APEL RUS SGAS, APEL
Data.Management.Storage StoRM, DPM SMS Smart-SE, dCache
ARC Gridftp
Data.Management.Transfer FTS JMS FTS, gridftp2
Data.Access.Relational
Data.Access.FlatFiles GFAL TSI ARC Caching
Information.Model GLUE GLUE (future) GLUE, arcschema
Information.Discovery OpenLDAP OpenLDAP
Information.Monitoring GridICE, LLview, CIS, RSS NG-Monitor
(R-GMA)
ExecMan.ExecService GT2 (4) Gram, TSS, OGSA-BES Grid-Manager+AREX (BES), gridftp interface
CREAM+BES
ExecMan.JobManager WMS XNJS ARC Client
ExecMan.CandidateSetGen
ExecMan.ExecPlannService
EGI staks
6
UserInterface
VOMSatlas
Grid services
CE SE
Computing Element
Storage Element
GIIS GRIS1
GRIS
Information SystemResource Broker
UserInterface
CREAM
VOMS cms
WorkerNodeWorkerNode
...WorkerNode
CE SE
Computing Element
Storage Element
GIIS GRIS1
GRIS
GRAM
WorkerNodeWorkerNode
...WorkerNode
Grid Monitoring(GridICE)
LFCLogical File Catalogue
VO GPboxcms
Site GPboxcms
VO GPboxatlas
account system
AuthorisationPolicies
AuthenticationCredential manager
Computers and storage
Job Submission / WMS
Workload Management System (WMS)• Essential functionality:
– manage grid jobs from submission to completion– match and rank suitable resources– deal with failures (resubmit jobs transparently)
• Highlights:– job rate up to 100Kjob/day (single job)
– job collections, bulk matchmaking– parallel matchmaking– prioritized queue
Job Management / LB
Logging and Bookkeeping (L&B)
• Essential functionality:– gather information on job execution on grid
components– provide users with digested consistent view on job
state
• Highlights:– knowledge on job processing concentrated in state
machine– reliable information delivery
• Short term plans:– integration with monitoring tools (RTM, Dashboard,
GridView)– advanced authorization
Job Submission/cream CE• The CREAM (Computing Resource Execution And Management) Service is a
simple, lightweight service that implements all the operations at the Computing Element (CE) level; its well-defined WebService-based interface and its implementation as an extension of the Java-Axis servlet (running inside the Apache Tomcat container) provide interoperability with clients written in any programming language and running on any computer platform.
• The CREAM interface is well-defined using the Web Service Description Language (WSDL); anyone can generate his/her CREAM client by simply filling in the stub code generated by WSDL parser (gSOAP for C/C++, Axis for Java, Perl module for perl).
• CREAM Main functionality: – Job Submission
• Possibility of direct staging of input sandbox files GLITE WMS JDL compliance (with CREAM-specific extensions)
• Support for batch and MPI jobs • Support for bulk jobs being integrated
– Manual and automatic proxy delegation – Job Cancellation – Job Info with configurable level of verbosity and filtering based on submission time and/or
job status – Job List – Job Suspension and Resume – GSI based authentication – VOMS based authorization – Job Purge for terminated jobs – Possibility (for admin) to disable new submissions
CREAM can be used
• by the Workload Management System (WMS), via the ICE (Interface to CREAM Environment) service
• by a generic client, e.g. an end-user willing to directly submit jobs to a CREAM CE. A C++ command line interface and Java clients are available.
Thanks to ICE (a gSOAP/C++ intermediate layer) CREAM can receive job operations
directly from a Grid WMS (Resource Broker). The ICE layer subscribes to the CEMon
service in order to asynchronously receive notifications about job status changes. In
case some notifications are lost, ICE performs synchronous status polling for jobs for
which it hasn't received status for some time. To maintain its subscriptions ICE
periodically checks them and renews the expiring ones. CEMon gets informations about
job status from the CREAM data persistency backend through JNDI APIs. ICE basically
has the role that the Job Controller (LC) and Log Monitor (LM) play when dealing with
non-CREAM CEs
Job Submission / cream
Supported Job Types
• Batch-like• Dag workflow
• Collection• Parametric
• MPI
• Interactive
11
Compound
K
Storage Element for files
• SE definition– The Storage Element (SE) is responsible for saving/retrieving files to/from the
local storage that can be a disk or a mass storage system. It manages disk space for files and maintains the cache for temporary files.
• SE Interfaces: there are two interfaces into the SE– The first interface is the Storage Resource Management (SRM) interface. This
interface allows the client to manage the storage space - to allocate space for jobs, to prepare data to be retrieved through a certain protocol, etc.
– posix-like File I/O.
• Files and Replica catalogs– In the Grid the user identifies files by logical file names (LFNs). The LFN is the
key by which the users locate the actual locations of their files. We refer to file replicas if the file has several managed copies, being tracked by the Replica Catalogue (Globus-RLS (Replica Location Service), LFC (LCG File Catalogue)).
12
13
SE – SRM implentations
An SRM standard interface hides storage characteristics.
From client point of view no difference appears!
SRM Interface
SRM-dCache
Access
StoRMSRM-Castor
Storage System
GPFS
Storage System
DCACHE
Storage System
CASTOR
Access Protocol
rfio
Access Protocol
dcap
Access Protocol
posix
Data Access
ManagementAccess
Access Protocol
posix
rfioand
da R.Zappi
StoRM overview
• StoRM is a storage resource manager for disk based
storage systems, implementing the SRM interface v2.2.• It is designed to take advantage from high performing
cluster file system, as GPFS or Lustre, but it supports also every standard POSIX FS (anche Amazon S2).
• It allows direct access (through the protocol le:// ) to the storage resource, as well as other standard grid protocol as gsiftp and rfio.
• Authentication and authorization are based on the VOMS• credential.• Permission enforcing are based on setting physical ACLs
on files and directories.
14
Data Management
Grid file access library (GFAL)– POSIX-like access to SE– SRM-compliant back-ends– unified access for LFN, GUID, SURL, and TURL
LCG file catalog (LFC)– map LFN to GUID and SURL– authorization on logical namespace
File transfer service (FTS)– manage transfers between storage elements– asynchronous bulk requests– parform transfers reliably, monitor progress
18
EGI: The future evolution
Testbeds Utility ServiceRoutine Usage
National
Global
European e-Infrastructure
19
EGI_DS Schedule
Duration 27 months:
5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10
Develop EGI Proposal NGIs signing Proposal
Start of EGEE-III
Final Draft of EGI Blueprint Proposal
EGI Blueprint Proposal
EGEE-III transition to EGI-like structure
EGI Entity in place
EU Call Deadline for EGI Proposal
Submission of EGEE-III
Start of EGI Design Study
2008 2009 2010
EGEE-II (2YEARS) EGEE-III (2YEARS) EGI operational