workshop della commissione calcolo e reti castiadas, 25-27 maggio 2004 il progetto egee luciano...
TRANSCRIPT
Workshop della Commissione Calcolo e RetiCastiadas, 25-27 maggio 2004
Il progetto EGEELuciano Gaido
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
Il contesto
01/01 – 03/04
12/01 – 03/04
2002-2005 + 2006-2008
06/00 - ......
Grid.it (FIRB) 01/03 – 12/05
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
Risultati dei progetti precedenti
• Release di middleware• EDG 2.x
• LCG 2.0 (basata su EDG 2.0)
• INFN-GRID 2.0 (basata su LCG 2.0)
• EGEE-0 (aka LCG 2.0)
• Infrastrutture• Testbed EDG (development, application)
• Testbed INFN-GRID
• Testbed DataTAG
• Grid di produzione italiana (INFN-GRID, GRID-IT)
• Infrastruttura di produzione LCG
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
Le infrastrutture Le infrastrutture
TB0 TB1 TB2 TB3
Mar 2004
2001 2002 2003
Globus 1 EDG 1 EDG 1.4 EDG 2
2004
AssessmentR&D
Jan 2001
LCG-1 LCG-2
INFNGRID-1 INFNGRID-2Figura presa dalla pres. di M. Parsons a Cork
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
Il progetto EGEE
• Obiettivo:
“EGEE aims to integrate current national, regional and thematic Grid efforts in order to create a seamless European Grid Infrastructure. This infrastructure will build on the underlying EU Research Network GEANT and exploit Grid expertise that has been generated by projects such as the European Union DataGrid project, other EU supported Grid projects and the national Grid initiatives such as UK e-Science, InfnGrid and NorduGrid.”
• Iniziato il 1 aprile 2004, con durata biennale
• Previsto un rinnovo per un secondo biennio
• 70 partner, 32 MEuro
• Kick-off meeting a Cork (19-22 aprile 2004)
Applications
Geant network
Grid infrastr.
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
Organizzazione di EGEE
I 70 partner sono stati divisi in 12 federazioni:
1. CERN2. Central Europe including Austria, Czech Republic, Hungary, Poland, Slovakia and Slovenia3. France4. Germany and Switzerland5. Ireland and the United Kingdom6. Italy7. Northern Europe including Belgium, Denmark, Finland, the Netherlands, Norway and Sweden8. Russia9. South-East Europe including Bulgaria, Cyprus, Greece, Israel and Romania10. South-West Europe including Portugal and Spain11. NRENS (National Research and Education Networks)12. United States
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
Struttura di EGEE
Tre filoni di attività:
• Joint Research Activities:• JRA1 – Middleware Engineering and IntegrationJRA1 – Middleware Engineering and Integration• JRA2 – Quality Assurance • JRA3 – Security• JRA4 - Network Services Development
• Service Activities:• SA1 – Grid Operation and SupportSA1 – Grid Operation and Support• SA2 – Network Resource Provision
• Networking Activities:• NA1 – Management of I3 (Integrated Infrastructure Initiative)• NA2 – Dissemination and Outreach• NA3 – User Training and Induction• NA4 – Application Identification and SupportNA4 – Application Identification and Support• NA5 – Internal Cooperation
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
EGEE Implementation
• From day 1 (1st April 2004)Production grid service based on the LCG infrastructure running LCG-2 grid middleware (SA)
LCG-2 will be maintained until the new generation has proven itself (fallback solution)
• In parallel develop a “next generation” grid facility (JRA)Produce a new set of grid services according to evolving standards (Web Services)
Run a development service providing early access for evaluation purposes
Will replace LCG-2 on production facility in 2005
Globus 2 based Web services based
EGEE-2EGEE-1LCG-2LCG-1
EDGVDT . . .
LCG
EGEE
. . .AliEn
Dalla pres. di Bob Jones a Cork
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
EGEE and LCG
EGEE builds on the work of LCG to establish a grid operations service• LCG: a worldwide collaboration of
• The LHC experiments• The Regional Computing Centres• Physics institutes
• Mission:• Prepare and deploy the computing environment that will be used by the
experiments to analyse the LHC data
• Strategy:• Integrate thousands of computers at dozens of participating institutes
worldwide into a global computing resource• Rely on software being developed in advanced grid technology projects, both
in Europe and in the USA
• Status:• LCG service up and running with LCG-2 mware – successfully being used for
LHC data challenges
Dalla pres. di Bob Jones a Cork
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
JRA1: Middleware Engineering and Integration
L’attività di sviluppo è distribuita su vari cluster:
• UK
• IT/CZ (F. Prelz, M. Sgaravatto)
• CERN
• Nordic
Ogni cluster ha un suo development testbed distribuito
L’integrazione e testing del m/w e lo sviluppo di tools è responsabilità del CERN.
Del cluster italo/ceco fa parte anche Datamat
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
JRA1: Middleware Engineering and Integration
Punto di partenza per la reingegnerizzazione del m/w: • Design e componenti grid basati su quanto deciso dall’RTAG* ARDA (A
Realization of Distributed Analysis)
• Design Team costituito da S. Fisher (UK), F. Prelz (IT/CZ), D. Groep (Nordic), M. Livny (VDT), P. Buncic (CERN), P. Kunszt (CERN), F. Hemmer (CERN) e E. Laure (CERN)
• Responsabilità del CE (IT/CZ)
• Esperienza e componenti da AliEn, VDT e EDG
• Web Services * Requirements and Technical Assessment Group (LCG)
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
JRA1: Middleware Engineering and Integration
• Key deliverables/milestones
• Jun’04 Tools for mware engineering and integration deployed (MJRA1.1)
• Jun’04 Architecture and Planning Document - Rel 1 (DJRA1.1)
• Jun’04 SW cluster develop. and testing infrastructure available (MJRA1.2)
• Aug’04 Integration and testing infrastructure in place with test plans – Rel 1 (MJRA1.3)
• Aug’04 Design Document for grid services - Rel 1 (DJRA1.2)
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
NA4 Work Breakdown Structure
EGEE
NA4NA3
NA3 liaison
Test team
HEPapplications
Biomedicalapplications
Genericapplications
Industryforum
Application specificsoftwares
Gridinterfaces
Test suitesMeetings,
reportsWeb site,
deliverables, internal notes
http://alipc1.ct.infn.it/grid/egee/na4Dalla pres. di R. Barbera a Cork
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
NA4 Generic Applications goals
Activity NA4 focuses on the identification and support of early-user and established applications for use on the EGEE infrastructure. It has the following objectives:
To identify through the dissemination partners and a well defined integration process a portfolio of early user applications from a broad range of application sectors from academia, industry and commerce.
To support development and production use of all of these applications on the EGEE infrastructure and thereby establish a strong user base on which to build a broad EGEE user community.
To initially focus on two well-defined application areas – Particle Physics and Life sciences.
The expected outcome of the activity will be the establishment of a broad portfolio of applications across a wide range of sectors suited to execution on the EGEE infrastructure meeting the needs of a broad collection of user groups from many sectors across Europe as illustrated in Figure (note the timeline for the introduction of each application domain is purely illustrative).
Dalla pres. di R. Barbera a Cork
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
Conclusions and outlook for GILDA
• GILDA is a complete suite of grid elements (test-bed, CA, VO, monitoring system, web portal) and applications fully dedicated to dissemination purposes.
• GILDA runs and will run the last production (stable) version of the grid middleware (currently Grid.it 2.0.0 based on LCG 2.0.0).
• GILDA is the dissemination tool which will be used by NA3 during induction courses and tutorials (first in Edinburgh next week!).
• Next future (2-3 weeks):• adding new sites: NeSC, InterQuanta (private company in Spain), any other
interested;• finalize and advertise the “24x7” public “grid demonstrator”: a customized
version of the full GENIUS web portal from where all users belonging to the GILDA VO can submit a pre-defined set of applications to the GILDA Testbed;
• publish all the documentation on the GILDA web site.
Dalla pres. di R. Barbera a Cork
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
The GILDA Tutorials (1/2)
• Edinburgh, 7 April 2004
• Tunis, 22-23 April 2004
• Edinburgh, 26-28 April 2004, registration and program
• CERN, 17-19 May 2004
Dalla pres. di R. Barbera a Cork
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
SA1: obiettivi
• Core Infrastructure services: • Operate essential grid services
• Grid monitoring and control: • Proactively monitor the operational state and performance, • Initiate corrective action
• Middleware deployment and resource induction: • Validate and deploy middleware releases• Set up operational procedures for new resources
• Resource provider and user support: • Coordinate the resolution of problems from both Resource Centres and users• Filter and aggregate problems, providing or obtaining solutions
• Grid management: • Coordinate Regional Operations Centres (ROC) and Core Infrastructure Centres (CIC) • Manage the relationships with resource providers via service-level agreements.
• International collaboration: • Drive collaboration with peer organisations in the U.S. and in Asia-Pacific • Ensure interoperability of grid infrastructures and services for cross-domain VO’s• Participate in liaison and standards bodies in wider grid community
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
SA1: strutture (1)
RC (Resource Centres):
• Provide computing and storage resources for the grid
OMC (Operation Management Center):
• Manage the operations of the entire EGEE grid from a single centralised location (CERN)
CIC (Core Infrastructure Center):
• Provide the basic service infrastructure of the Grid
• Operate the key services which connect users with resources
• Support the Regional Operation Centers
Numeri: RC (ad libitum), OMC (1), ROC (9), CIC (4)
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
SA1: strutture (2)
ROC (Regional Operation Center):• Customize and certificate the middleware releases• Provide the middleware release and up-to-date documentation to a set of
Resource Centers (RC)• Validate the set of grid services and components to form a functioning
Grid to serve user communities• Validate installation/upgrade procedures and documentation• Develop/Run test suites for components and Grid services to certify a
site installation• Refer and escalate middleware problems to developers• Collaborate with CIC to have monitoring systems and performance
evaluator running • Define the tools to measure the service level provided by resource
centers• Manage the access policies and sharing of the resources • Provide 24x7 support service for users and local administrators
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
SA1: distribuzione geografica delle strutture
10 federazioni:• CERN OMC + CIC
• Francia ROC + CIC
• Italia ROC + CIC
• Germania e Svizzera ROC
• UK e Irlanda ROC + CIC
• Northern Europe ROC
• South West Europe ROC
• South East Europe ROC
• Central Europe ROC
• Russia ROC
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
SA1 in Italia: da dove partiamo
La grid in italia (InfnGrid) esiste dal 2001:
• E’ diventata una infrastruttura di produzione (Grid-It) nel 2003
• 22 siti operativi (alcuni fanno parte anche dell’infrastruttura LCG2)
• Usata per le attività degli esperimenti (Data Challenge e produzioni):
• CMS preproduction (INFN-GRID_1.0 aka LCG_1.0, winter 2003)• ALICE DC prima fase (febbraio-maggio 2004)• ATLAS DC e produzioni di VIRGO e BABAR in partenza• Prove di produzione di LHCb in corso
• Release: si usa INFN-GRID 2.0.0 (basata su LCG_2.0.0, 100% compatibile)
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
http://grid-it.cnaf.infn.it
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
SA1 in Italia
Chi siamo:
• INFN-GRID• Firb (Grid.it) tramite INFN• ENEA• UniLE*• UniNA*• UniCal*
* SPACI (Southern Partnership for Advanced Computational Infrastructures)
Responsabili ROC italiano: C. Vistoli, L. Gaido
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
SA1-IT: da dove partiamo
The existing italian Grid Operation Center is the first step towards the EGEE ROC + CIC. It will evolve according to the EGEE plans.
• Existing teams:
• Release and Documentation Team - “builds” the INFN-GRID release, starting from the LCG release and (up to now) - provides the LCFGng profiles - provides documentation and a sw repository
• Central Management Team - Takes care of the deployment of the release, certificates the sites and manages the grid services
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
SA1-IT: da dove partiamo (cont.)
• Existing teams (cont.): • Monitoring Team
GridICE development and support
Provides a customized view for – VO, – Sites, – Grid Operation Center (+ services),– Users
• Support Team
Provides support through a web portal (interim solution: OneorZero
with a ticketing system and a knowledge base system)
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
SA1: dove andiamo
Per costruire una grid europea (EGEE), risultato dell’interconnessione delle
grid nazionali, le strutture di operation, management e supporto esistenti nelle
differenti regioni/federazioni devono armonizzarsi ed integrarsi evolvendo in
una struttura comune.
Percorso: • definire un modello generale per ROC e CIC• identificare le differenti esigenze dei ROC (e quindi dei Rersouce Center)• Trovare il minimo comune denominatore (dei requirement) per definire
il contenuto della release EGEE (in collaborazione con JRA1)• Organizzarsi perche’ il resto (quello che non farà parte delle release
ufficiale) venga prodotto e supportato da SA1 e/o da uno o più ROC
fondamentale evitare divergenze
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
SA1: cosa stiamo facendo?
I primi passi:
• Continuiamo a gestire le infrastrutture nazionali esistenti
• Creato gruppo ROC managers:• Riunione telefonica settimanale• Primo meeting ‘de visu’: milano 10-11 maggio• Questionario sulle piattaforme hw e sistemi operativi• Molte altre cose da definire.....
• In via di definizione i ruoli di CIC e ROC
• Definizione Execution Plan
• Definizione dei diversi testbed necessari (vedi figura)
EGEE Certification, Testing and Release Cycle
CERTIFICATIONTESTING SERVICES
Integrate
BasicFunctionality
Tests
Run testsC&T suitesSite suites
RunCertification
Matrix
Releasecandidate
tag
PR
E-P
RO
DU
CT
ION
PR
OD
UC
TIO
N
APPINTEGR
Certifiedrelease
tag
DE
VE
LO
PM
EN
T &
IN
TE
GR
AT
ION
UN
IT &
FU
NC
TIO
NA
L T
ES
TIN
G
DevTag
JRA1
HEPEXPTS
BIO-MED
OTHERTBD
APPSSW
Installation
DE
PL
OY
ME
NT
PR
EP
AR
AT
ION
Deploymentrelease
tag
DEPLOY
SA1
Productiontag
Workshop della Commissione Calcolo e Reti - 25-27 maggio 2004
Domande?