the worldgrid transatlantic testbed a successful example of grid interoperability

DataTag is a project funded by the European UnionDataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 -CHEP 2003 – 24-28 March 2003 -

The WorldGRID transatlantic testbedA successful example of Grid interoperabilityacross EU and US domains

Flavia Donno (Formerly of DataTAG WP4, LCG)

[email protected]

http://chep03.ucsd.edu/files/249.ppt

CHEP 2003 – 24-28 March – nCHEP 2003 – 24-28 March – noo. (1). (1)

DataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – nDataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – noo 22

Talk Outline

• Motivation

• Participants

• Interoperability issues

• Solutions

• Architecture

• Monitoring/Support

• Spin off

F. DonnoCERN/IT and INFN

R. GardnerUniversity of Chicago

Applications

CMS

ATLAS

Monitoring with Nagios

Monitoring with Ganglia

Conclusions

Next Steps


ParticipantsParticipants

DataTAG:F. Donno, CERN - INFNS. Andreozzi, R. Barbera, V. Ciaschini, S. Fantinel, A. Ghiselli, M. Mazzucato, D. Rebatto, G. Tortone, L. Vaccarossa, M. Verlato, C. Vistoli – INFNM. Draoli, CNR-Rome

CrossGrid:A. Garcia, M. Hardt, FZK - GermanyJ. Marco, UC - Spain M.David, J. Gomes, LIP - PortugalO. Maroney, U.Bristol, UK

Trillium/iVDGL:P. Avery, J. Rodriguez - U. FloridaE. Deelman, N. Olomu - USC/ISIJ. Gieraltowski, S. Gose, E. May, J. Schopf – ArgonneAfaq, J. Annis, R. Glossum, R. Pordes, V. Sekrhi – FermilabW. Deng, J. Smith, D. Yu - BNL A. DeSmit, A. Roy - Wisconsin C. Dumitrescu, I. Foster, R. Gardner, U. ChicagoL. Grundhoefer ,J. Hicks, F. Luehring, L. Meehan - U. IndianaS. Youssef, Boston UniversityB. Moe - MilwaukeeD. Olson – LBNLS. Singh - Caltech


MotivationsMotivations

Basic collaboration between European and US Grid projects

Interoperability between Grid domains for applications submitted by users from different virtual organizations Controlled use of shared resources subject to agreed policy Integrated use of heterogeneous resources from iVDGL and DataGrid/CrossGrid testbed domains

Goal:Goal:

Build a “transatlantic grid” based on the existent European Build a “transatlantic grid” based on the existent European and American Grids with the goal of offering transparent and American Grids with the goal of offering transparent access to the distributed computing infrastructure necessary access to the distributed computing infrastructure necessary to the “data-intensive” modern applicationsto the “data-intensive” modern applications


Interoperability IssuesInteroperability Issues Many grids with several OSseveral OS (RH 6.2, RH 7.x, Fermi Linux, CERN Linux,…), several compilers and software components.Different Grid ArchitecturesGrid Architectures (VDT server/client vs. Computing Elements, Storage Elements, User Interfaces, …)Need to identify minimum set of core servicesminimum set of core services and define collective/optional servicesoptional services Common protocols/Same or compatible versions of the softwareAuthentication and AuthorizationAuthentication and Authorization mechanism: authority trusting, user authentication/authorization via LDAP VO Servers.Grid resource descriptionGrid resource description/status: Globus schema vs. EDG schema vs. GLUE schema Several Grid Data managementGrid Data management ToolsSoftware distribution and configurationSoftware distribution and configuration : rpm based vs. PACMAN


SolutionsSolutions

Many grids with several OSseveral OS (RH 6.2, RH 7.x, Fermi Linux, CERN Linux,…), several compilers and software components.

Partition WorldGrid in subdomains with uniform or compatible set of basic services. Such resources will advertise themselves with specific targets to the applications (such as RH6.2).Try to keep the subdomains as large as possible.


SolutionsSolutions

Different Grid ArchitecturesGrid Architectures (VDT server/client vs. Computing Elements, Storage Elements, User Interfaces, …)

CECE

UIUI

SESERBRB

VDT Client

VDT Server

RCRC

ISIS

RCRC

ISIS


SolutionsSolutions

Need to identify minimum set of core servicesminimum set of core services and define collective/optional servicesoptional services Common protocols/Same or compatible versions of the software

Globus and Condor core services (GRAM, GSI, MDS, GridFTP, …)

Resource Broker, User Interface and JDL, Data Management high level tools (edg-replica-manager, MAGDA, Globus Replica Catalog, …) collective optional services not installed universally

User Grid Portals (Genius, GRAPPA, …): a variety available not to change the User Interface to the GRID


SolutionsSolutions

Authentication and AuthorizationAuthentication and Authorization mechanism: authority trusting, user authentication/authorization via LDAP VO Servers.

DOE and EDG certificates universally accepted DataTAG and iVDGL VO LDAP servers trusted mkgridmap tool universally installed Local security policy sites agreed to allow

access to grid demonstration users (kerberos, …)


SolutionsSolutions

Grid resource descriptionGrid resource description/status: Globus schema vs. EDG schema vs. GLUE schema

three coexistent schemas in place (Globus, EDG, GLUE) installed on all resources

Some tool (monitoring) working with all of them

EDG middleware using both EDG and GLUE US tools using none or Globus


SolutionsSolutions

Software distribution and configurationSoftware distribution and configuration : rpm based vs. PACMAN

Created WorldGrid distribution (rpm/LCFGng and PACMAN)

Effort to ensure coherency and automatic configuration


Final ArchitectureFinal Architecture

CECE

UIUI

SESERBRB

VDT Client

VDT Server

RCRC

ISIS


Monitoring and SupportMonitoring and Support

Two monitoring tools VO based in place: Two monitoring tools VO based in place: edt-monitor based edt-monitor based on Nagios and iVDGL based on Ganglia (see talk from R. on Nagios and iVDGL based on Ganglia (see talk from R. Gardner)Gardner)

Support infrastructure: Support infrastructure: to support to support

site site administrators administrators during the during the installation and installation and configuration configuration procedure. Also procedure. Also for problem for problem fixing fixing during during normal normal operationoperation


Spin-offSpin-off GLUE schema:GLUE schema: WorldGrid has allowed to prove the validity of WorldGrid has allowed to prove the validity of the GLUE schema and encouraged EDG to deploy itthe GLUE schema and encouraged EDG to deploy it VOMS:VOMS: The authentication/authorization problems were The authentication/authorization problems were identified and parallel research activities started, like the one on identified and parallel research activities started, like the one on Virtual Organization Manager ServiceVirtual Organization Manager Service GLUE Packaging:GLUE Packaging: A working group is trying to find a solution A working group is trying to find a solution for a standardization of the packaging, distribution and for a standardization of the packaging, distribution and configuration problem for a software releaseconfiguration problem for a software release GLUE Testing:GLUE Testing: The problem of verifying an installation and The problem of verifying an installation and validate a site for joining the Grid has been addressed and a validate a site for joining the Grid has been addressed and a working group has startedworking group has started Support:Support: A first operation/monitoring center has started A first operation/monitoring center has started in US taking advantage of the in US taking advantage of the monitoring toolsmonitoring tools. Other . Other centers in EUcenters in EU LCG-0:LCG-0: After the demonstration at IST2002 and SC2002, After the demonstration at IST2002 and SC2002, LCG has based his LCG has based his first middleware distribution onfirst middleware distribution on the the WorldGridWorldGrid experience experience

DataTag is a project funded by the European UnionDataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 -CHEP 2003 – 24-28 March 2003 -

Rob Gardner University of Chicago

on behalf of the WG group

The WorldGRID transatlantic testbed, Part 2A successful example of Grid interoperabilityacross EU and US domains

16 R. Gardner CHEP 2003 – 24-28 March 2003 – nR. Gardner CHEP 2003 – 24-28 March 2003 – noo 1616

Talk Outline

Motivation

Participants

Interoperability issues

Solutions

Architecture

Monitoring/Support

Spin off

F. DonnoCERN/IT and INFN

Part 2

R. GardnerUniversity of Chicago

Applications

CMS

ATLAS

Monitoring with Nagios

Monitoring with Ganglia

Conclusions

Next Steps


Installing Apps on 2 Grids We needed a way to get applications from three

experiments (VO’s) setup on the execution sites On DataTAG resources, selected CE’s were loaded

with CMS or ATLAS rpms On iVDGL resources, we Pacmanized binaries

(rpms and tarballs) of bundled applications– %pacman –get iVDGL:ScienceGrid

> Atlas-kit, Atlas-ATLFAST

> CMS-MOP, EDG-CMS

> SDSS Astrotools

– binaries, and run time environments

3 experiments


ATLSIMJob

Write to Grid

Storage Element

Read from Grid

Storage Element Input Data

Output ZEBRA

Grid Storage

Apache

EnginFrame

GENIUS

https+java/xml+rfb

WEB Browser

EDGUI

Local

WS

the GridEDG+GSI

ATLAS and CMS with GENIUS

see R. Barbera’s Genius talk this conference


RB/JSS II

SE

input data location

Replica Catalog TOP

GIIS

. . .CE

Executable = "/usr/bin/env";Arguments = "zsh prod.dc1_wrc 00001";

VirtualOrganization="datatag";Requirements=Member(other.GlueHostApplicationSoftwareRunTimeEnvironment,"ATLAS-3.2.1" );Rank = other.GlueCEStateFreeCPUs;InputSandbox={"prod.dc1_wrc",“rc.conf","plot.kumac"};OutputSandbox={"dc1.002000.test.00001.hlt.pythia_jet_17.log","dc1.002000.test.00001.hlt.pythia_jet_17.his","dc1.002000.test.00001.hlt.pythia_jet_17.err","plot.kumac"};ReplicaCatalog="ldap://dell04.cnaf.infn.it:9211/lc=ATLAS,rc=GLUE,dc=dell04,dc=cnaf,dc=infn,dc=it";InputData = {"LF:dc1.002000.evgen.0001.hlt.pythia_jet_17.root"};StdOutput = " dc1.002000.test.00001.hlt.pythia_jet_17.log";StdError = "dc1.002000.test.00001.hlt.pythia_jet_17.err";DataAccessProtocol = "file";

JDL GLUE-aware files

WNATLAS sw

data

registration

GLUE-Schema basedInformation System

WorldGridTestbed

JDL

Job

GENIUS

UI

see WorldGrid Poster this conf.


CMS Applications Monte Carlo Production chain on Grid

– CMKIN: generation physics events with PYTHIA

– CMSIM: simulation of the detector with GEANT3 CMS production software installed in the WN’s Job workflow and data management

– CMKIN jobs sent by the RB to WN with CMS software, store the output at nearby SE

> register LFN to the RC

– CMSIM jobs sent by the RB to WN nearby SE> Register LFN to the RC


ATLAS Applications Grappa and Genius

submissions ATLAS Detector

Simulations– Simulation of the detector

response using ATLSIM (GEANT3)

– Based on DC1 Grid script

ATLAS production software installed in the WN’s


Grappa Portal Engine

Web browser interface

https

Cactus framework

Scriptinterfac

e

Resource A Resource Z. . . MAGDA: replica and metadataCompute Elements

Java CoGsubmission,monitoring

Storage Elements: - Disk/HPSS

inputfiles

Grappa and ATLASsee D. Enghthis conf.


Job Submission Animation


VO Monitoring Initial Requirements:

– Grid-level resource activity, utilization, and performance monitoring;

– VO-level resource activity and resource utilization monitoring;

– Customized views: > Hardware resources (clusters, sites, grids);

> VO usages, jobs, work-types;

Design Goals: – Scalability over large number of resources and networks;

– Simplicity and distributed architecture; Two approaches

– iVDGL: built on popular Ganglia resource monitoring package from UC Berkeley

– DataTAG: built on popular Nagios package http://www.nagios.org/


Web php client

Grid Aggregation

Round Robin DB Tool

Logging &

Bookeeping

UI

RB JSS CE

RRDB Tool

gmond gmond gmond gmond

Site a

RRDB Tool

gmond gmond gmond gmond

Site b

DataTAG

VO Ganglia

iVDGL


Site Level VO Usage and Policy


VO Nagios Monitoring based on Nagios (a host and service monitoring engine)

[detailed information on: http://www.nagios.org]

host local plug-ins – collect info from OS- CPU load- RAM- disk- jobs

MDS plug-ins - collect aggregate info from GRIS- number of running/waiting jobs- number of total/free CPUs

history graphs for all monitoring metrics

aggregate info/graphs per Site and Virtual Organization


Status and Summary Map

grid-aggregate monitors

3-levelstatus map


VO Usage Graphs

MDS collected

site andaggregated montiors

see G. Tortone et. al., this conference


WorldGrid Next Steps New developments in DataTAG:

– Test/experiment with SRM solutionSRM solution for Storage Element access (multiple implementations of the protocol)

– Test/experiment with advanced Data Management toolsData Management tools such as Globus-EDG/RLS

– Propose alternative Grid Resource DiscoveryGrid Resource Discovery mechanisms based on WEB services

– Improve the monitoringmonitoring tools taking advantage of OGSAOGSA

– Develop a WorldGrid GOCWorldGrid GOC, coordinated operations centers

Continue themes in iVDGL:– site-friendly installations, untouched by humans

– multi-VO (controlled use of shared resources)

– pursue concept of ‘projects’


A project consists of

• A (typically small) list of distinguished names or VO(s).

• Email and phone contact.

• A software environment expressed as a Pacman package.

• Local disk space requirements.

• A url describing the project.

Basic site management operations:

• Join a project

• Leave a project

• Pause a project

Site manager commands

Projects as unit of access


% worldgrid –info

-join <project>

-leave <project>

-pause <project>

-kill <project>

-update <project>

-getCA <CA>

-setForum <URL>

Example Site Manager Commands


iVDGL WorldGrid FAQ Forum Help

Joined projects

Demo

ATLASDC2-higgs

ChimeraTest8

Projects Certified

Demo

CMS-DC2-SUSY

ChimeraTest8

ChimeraTest9

ATLASDC2-higgs

SDSC-scan45

Installed Software

WorldGrid

ScienceGrid

ProjectAccess

CAs

Batch jobs

Performance

10/150 G used in WorkSpace

History


Conclusions Lessons from WorldGrid 2002

– Grid building> Packaging and configuration key

> GLUE meta-packaging study launched, report available

> Testing and site validation

– Interoperability> Configuration of common MDS schema allowed joint use of VDT

and EDG middleware installations

> good experience for LCG

– Integrate two very different grids> “Top down” EDG-style of Grids with high level services

> “Bottoms up” VDT-style grids providing core services with

– Transatlantic cooperation can be fun!

the worldgrid transatlantic testbed a successful example of grid interoperability

Documents

european union chep

grid domains

glue schema datatag

globus schema

edg schema

datatag wp4

solutions authentication

computing elements