towards seamless grid computing the egee experience on interoperable grid infrastructures

EGEE-II INFSO-RI-031688

Enabling Grids for E-sciencE

www.eu-egee.org

EGEE and gLite are registered trademarks

Towards Seamless Grid ComputingThe EGEE Experience on Interoperable Grid Infrastructures

Erwin Laure

EGEE-II Technical Director

[email protected]


EGEE-II INFSO-RI-031688 EuroVO Workshop - April 2008 2

eScience

• Science is becoming increasingly digital, needs to deal with increasing amounts of data and computational needs

• Simulations get ever more detailed– Nanotechnology – design of new materials from

the molecular scale– Modelling and predicting complex systems

(weather forecasting, river floods, earthquake)– Decoding the human genome

• Experimental Science uses ever moresophisticated sensors to make precisemeasurements Need high statistics Huge amounts of data Serves user communities around the world



Scientific trends

Scientific advances are more and more based on simulations using “virtual laboratories”

Ulf Dahlsten, Former Director Emerging Technologies and Infrastructures, EU, predicts that “in five years 80 percent of all scientific papers in all areas will be made in virtual laboratories. Fifty percent of social science documents will go the same way in five to ten years.”

The size of data an organization owns, manages, and depends on is dramatically increasing:–Ownership cost of storage capacity goes down–Data generated and consumed goes up–Network capacity goes up–Distributed computing technology matures and is more widely adopted



EGEE

Main Objectives• Operate a large-scale,

production quality grid infrastructure for e-Science

• Attract new resources and users from industry as wellas sciences

– Flagship grid infrastructure project co-funded by the European Commission

– Now in 2nd phase with 91 partners in 32 countries



EGEE – What do we deliver?• Infrastructure operation

– Sites distributed across many countries Large quantity of CPUs and storage Continuous monitoring of grid services & automated site

configuration/management Support multiple Virtual Organisations from diverse

research disciplines

• Middleware– Production quality middleware distributed under

business friendly open source licence Implements a service-oriented architecture that virtualises

resources

Adheres to recommendations on web service inter-operability and evolving towards emerging standards

• User Support - Managed process from first contact through to production usage– Training– Expertise in grid-enabling applications– Online helpdesk– Networking events (User Forum, Conferences etc.)



250 sites48 countries50,000 CPUs13 PetaBytes>5000 users>200 VOs>140,000 jobs/day

ArcheologyAstronomyAstrophysicsCivil ProtectionComp. ChemistryEarth SciencesFinanceFusionGeophysicsHigh Energy PhysicsLife SciencesMultimediaMaterial Sciences…

32%


EGEE-II INFSO-RI-031688 7

Users and resources distribution

EuroVO Workshop - April 2008



EGEE Grid Management Structure

• Operations Coordination Centre (OCC)

– management, oversight of all operational and support activities

• Regional Operations Centres (ROC)

– providing the core of the support infrastructure, each supporting a number of resource centres within its region

– Grid Operator on Duty

• Resource centres – providing resources (computing,

storage, network, etc.);

• Grid User Support (GGUS)

– At FZK, coordination and management of user support, single point of contact for users



Example: GridMap Monitoring Visualization

9



Registered Collaborating Projects

Applicationsimproved services for academia,

industry and the public

Support Actionskey complementary functions

Infrastructuresgeographical or thematic coverage

25 projects have registered as of September 2007: web page

http://egee-technical.web.cern.ch/egee-technical/related-projects/list-projects.htm



EGEE working with related infrastructure projects

GIN


EGEE-II INFSO-RI-031688 EuroVO Workshop - April 2008

LHC Use of Multiple Grid Infrastructures

12


EGEE-II INFSO-RI-031688 EuroVO Workshop - April 2008

Why is Interoperability difficult?

• Grid infrastructures use different technologies– And even if same technologies are used they are usually heavily

customized

• Only a few widely adopted standards– gridFTP, X.509 (but used differently!)– Prototypes: BES, JSDL, …

Production Grids are difficult to change – adopting standards takes time Standards need to be stable before adoption

• Apart from technological differences, access policies also differ– Dialog among major Grid infrastructure providers started at last OGF22.

• Strong interactions between infrastructures and application community needed– HEP was driving interop efforts for LHC– Other applications can build on these experiences

13



Access to Compute Resources

PBS/Torque

LSF

Condor

Load Leveler

Sun Grid Engine

ARCCREAM

NAREGI

Unicore

OSG

Nordugrid

Naregi

DEISA

EGEE

Teragrid

GRAM

v2/v4

GRAM

v4WS




How to Start

• Understanding the differences – Compatibility matrix

• Domains that have to be linked for interoperability– Security– Information Services – Job Management– Data Management

• For interoperation you have to add– Monitoring– Accounting – Operational links and joint policies– Trouble ticket systems – Operational security




Interoperability Matrix

1. Understand both middleware stacks

2. Identify the “common” interfaces

3. Create an interoperability matrix

SRMSRMSRMStorage Control Protocol

GSI/VOMS

GridFTP

GLUE v1

LDAP/GIIS

GRAM

OSG

GSI/VOMSGSI/VOMSSecurity

GridFTPGridFTPStorage Transfer Protocol

GLUE v1.2ARCSchema

LDAP/BDIILDAP/GIISService Discovery

GRAMGridFTPJob Submission

EGEEARC




Different Strategy

• Long term solution– Common interfaces– Standards

• Medium term solutions– Gateways – Adaptors and Translators

• Short term solutions– Parallel Infrastructures

User driven Site driven




Parallel Infrastructures

• User Driven– The user joins both grids

Uses different clients• Depending on which interface

– More work for the User Required for each infrastructure

– Keyhole approach Restricts functionality

– Method initially used by ATLAS Split workload between grids




Parallel Infrastructures

• Site Driven– The site joins both grids

Deploys both interfaces

– User only sees their grid interface

– More work for the site Can only be supported by large sites

• Reduced resources

– Use By FZK Participating in EGEE, Nordugrid and D-Grid




Gateway

• A gateway is a bridge between grid infrastructures– Single point of failure– Gateway breaks, grid disappears– Scalability bottleneck– All the load through one service

• Useful as a proof concept and to demonstrate the need• NAREGI approach using glite-CE

Gateway




Adaptors and Translators

• Adaptors allow connection• Translators understand/modify information• They are built into the middleware

– The middleware can then work with both interfaces Useful feature even when using standards!

• Requires modification to the grid middleware– Existing service interfaces can still be used

• Using in the GIN information System; most portals A

PI P

lug

inP

lug

in




Worldwide Grids

APAC

DEISA

EGEE

Naregi

NDGF

NGSOSG

Pragma

Teragrid

GIN



How mature are we?

Gartner Group

Grid on the Computing in HighEnergy Physics conferences timeline

Padova2000

Beijing2001

San Diego2003

Interlaken2004

Mumbai2006

Victoria2007

Slide courtesy of Les Robertson, LCG Project Leader


EGEE-II INFSO-RI-031688

From e-Infrastructures to Knowledge Infrastructures

• Network infrastructure connects computing and data resources and allows their seamless usage via Grid infrastructures

• Federated resources and new technologies enable new application fields: – Distributed digital libraries– Distributed data mining– Digital preservation of cultural heritage– Data curation

→ Knowledge Infrastructure

NETWORK .

INFRASTRUCTURE

GRID .

–INFRASTRUCTURE

KNOWLEDGE .

INFRASTRUCTURE

24EuroVO Workshop - April 2008

••• 25

ICT for Science: e-Infrastructures

Linking at the speed of the lightLinking at the speed of the light

Sharing computers, instruments and applicationsSharing computers, instruments and applications

Sharing and federating scientific dataSharing and federating scientific data

. . . . . .

.

Astrophysics

community

WeatherForecast

community

Biomedics

community

Connecting the finest mindsSharing and federating the best scientific

resourcesBuilding global virtual communities

Mario Campolargo Acting Director Emerging Technologies and Infrastructures, EU European Information Space: Infrastructures, Services and Applications Workshop, Rome, 29-30 October 2007



Evolution

European e-Infrastructure

Testbeds Utility ServiceRoutine Usage

National

Global



European Grid Initiative

• Need to prepare permanent, common Grid infrastructure

• Ensure the long-term sustainability of the European e-Infrastructure independent of short project funding cycles

• Coordinate the integration and interaction between National Grid Infrastructures (NGIs)

• Operate the production Grid infrastructure on a European level for a wide range of scientific disciplines



Summary

• EGEE provides a dependable production quality Grid infrastructure to a wide variety of scientific disciplines.

• Collaborations on technical and political topics are key to implement a truly world-wide infrastructure

• Need to cover full spectrum: from individual sites, small scale Grids to world-wide infrastructures

• Grids are increasingly becoming an essential part of the scientific computing infrastructure – sustainability needs to be ensured

towards seamless grid computing the egee experience on interoperable grid infrastructures

Documents

applications workshop

ensuredeurovo workshop

production grid infrastructure

european grid initiativeneed

scientific dataconnecting

european level

summary egee

eu european information