ian foster argonne national laboratory university of chicago foster open grid services architecture...

Ian Foster

Argonne National Laboratory

University of Chicago

http://www.mcs.anl.gov/~foster

Open Grid Services Architecture

Plenary Talk at CHEP 2003, San Diego, March 26, 2003

2

[email protected] ARGONNE CHICAGO

Building an Open Grid

OpenStandards

OpenSource

OpenInfrastructure

OpenGrid

3


Grids and Open Standards

Incr

ease

d fu

nctio

nalit

y,st

anda

rdiz

atio

n

Time

Customsolutions

Open GridServices Arch

GGF: OGSI, …(+ OASIS, W3C)

Multiple implementations,including Globus Toolkit

Web services

Globus Toolkit

Defacto standardsGGF: GridFTP, GSI

X.509,LDAP,FTP, …

App-specificServices

4


Layered Grid Architecture

Application

Fabric“Controlling things locally”: Access to, & control of, resources

Connectivity“Talking to things”: communication (Internet protocols) & security

Resource“Sharing single resources”: negotiating access, controlling use

Collective“Coordinating multiple resources”: ubiquitous infrastructure services, app-specific distributed services

InternetTransport

Application

Link

Inte

rnet P

roto

col

Arch

itectu

re

“The Anatomy of the Grid: Enabling Scalable Virtual Organizations”, Foster, Kesselman, Tuecke, Intl J. High Performance Computing Applications, 15(3), 2001.

5


Globus Toolkit v2 Four key protocols and APIs

– Grid Security Infrastructure (GSI)

– Grid Resource Allocation & Mgmt (GRAM)

– Grid Resource Information Protocol (GRIP) and Index Information Protocol (GIIP)

– Grid File Transfer Protocol (GridFTP) Implementations on many platforms

– Resources, security systems, data models,… Various collective layer protocols & tools

– Info services, replica management, etc. A basis for many Grid-enabled tools & apps

– FTP, SSH, Condor, SRB, MPI, EDG, GridPort, …

6


From Resources to Services:Managing Virtual Services

Trying to manage total system properties– E.g. Dependability, end-to-end QoS

“Resource” tends to connote a tangible entity to be consumed: CPU, storage, bandwidth, …

But many interesting services may be decoupled from any particular resource– E.g. virtual data service, data analysis service

– A service consumes resources, but how that happens is irrelevant to the client

“Service” forms a better base abstraction– Can apply to physical or virtual

7


Open Grid Services Architecture

Service-oriented architecture– Key to virtualization, discovery, composition,

local-remote transparency Leverage industry standards

– Internet, Web services Distributed service management

– A “component model for Web services” (or: a “service model for the Grid”)

A framework for the definition of composable, interoperable services

“The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration”, Foster, Kesselman, Nick, Tuecke, 2002

8


Web Services

A simple but powerful distributed system paradigm, that allows one to:– Describe a service (WSDL)

– Invoke a service (SOAP)

– Discover a service (various) Web services appears to offer a fighting

chance at ubiquity (unlike CORBA)– Sophisticated tools emerging from industry

But Web services does not go far enough to serve a common base for the Grid …

9


Transient Service Instances “Web services” address discovery & invocation of

persistent services– Interface to persistent state of entire enterprise

In Grids, must also support transient service instances, created/destroyed dynamically– Interfaces to the states of distributed activities

– E.g. workflow, video conf., dist. data analysis Significant implications for how services are managed,

named, discovered, and used– In fact, much of Grid is concerned with the management of

service instances

10


OGSA Structure A standard substrate: the Grid service

– Standard interfaces and behaviors that address key distributed system issues

– A refactoring and extension of the Globus Toolkit protocol suite

… supports standard service specifications– Resource management, databases, workflow,

security, diagnostics, etc., etc.

– Target of current & planned GGF efforts … and arbitrary application-specific services

based on these & other definitions

11


Open Grid Services Infrastructure

Implementation

Hosting environment/runtime(“C”, J2EE, .NET, …)

Dataaccess

12



Implementation


Dataaccess

Grid ServiceHandle

Grid ServiceReference

handleresolution

13



Implementation


Servicedata

element

Servicedata

element

GridService(required)

Dataaccess

Grid ServiceHandle


handleresolution

14



Implementation


Servicedata

element

Servicedata

element


Dataaccess

Lifetime management• Explicit destruction• Soft-state lifetime

Introspection:• What port types?• What policy?• What state?

Client

Grid ServiceHandle


handleresolution

15



Implementation

Servicedata

element

Other standard interfaces:factory,

notification,collections


Servicedata

element

Servicedata

element


Dataaccess

Lifetime management• Explicit destruction• Soft-state lifetime

Introspection:• What port types?• What policy?• What state?

Client

Grid ServiceHandle


handleresolution

16



GWD-R (draft-ggf-ogsi- gridservice-23) Editors:Open Grid Services Infrastructure (OGSI) S. Tuecke, ANLhttp://www.ggf.org/ogsi-wg K. Czajkowski, USC/ISI

I. Foster, ANLJ. Frey, IBMS. Graham, IBMC. Kesselman, USC/ISID. Snelling, Fujitsu LabsP. Vanderbilt, NASAFebruary 17, 2003

Open Grid Services Infrastructure (OGSI)

17


Realizing a Service-Oriented Architecture: How Do I

Create, name, manage, discover services? Render resources, data, sensors as services? Negotiate service level agreements? Express & negotiate policy? Organize & manage service collections? Establish identity, negotiate authentication? Manage VO membership & communication? Compose services efficiently? Achieve interoperability?

18


The OGSA Platform

OGSI

TransportProtocolHosting EnvironmentHosting Environment

Host. Env. & Protocol Bindings

OGSA Platform services: registry,authorization, monitoring, data

access, etc., etc.

More specialized &domain-specific

services

Models for resources &

other entitiesO

therm

odels

Environment-specificprofiles

Domain-specificprofiles

OGSAPlatform

GWD-R (draft-ggf-ogsa-platform-3) Editors:Open Grid Services Architecture Platform I. Foster, Argonne & U.Chicagohttp://www.ggf.org/ogsa-wg D. Gannon, Indiana U.

19


OGSA Definition Activities(Underway or Pending)

Data Access and Integration Data Replication Security SLA Negotiation Common Management Model And others…

20


Open Grid Service Architecture:Next Steps

Technical specifications– Open Grid Services Infrastructure is complete

– Security, data access, Java binding, common management models, etc., in the pipeline

Implementations and compliant products– OGSA-based Globus Toolkit v3, pyGlobus, …

– IBM, Avaki, Platform, Sun, NEC, Oracle, … Rich set of service defns & implementations

– Time to start on OGSI-compliant services!

21


Example:Reliable File Transfer Service

Performance

Policy

Faults

servicedataelements

Pending

FileTransfer

InternalState

GridService

Notf’nSource

Policy

interfacesQuery &/orsubscribe

to service data

FaultMonitor

Perf.Monitor

Client Client Client

Request and manage file transfer operations

Data transfer operations

22


Globus Toolkit v3 (GT3)Open Source OGSA Technology

Implement core OGSI interfaces Support primary GT2 interfaces

– High degree of backward compatibility Multiple platforms & hosting environments

– J2EE, Java, C, .NET, Python New services

– SLA negotiation (GRAM-2), registry, replica location, community authorization, data, …

Growing external contributions & adoption

23


GT2 Evolution To GT3

What happened to the GT2 key protocols?– Security: Adapting X.509 proxy certs to integrate

with emerging WS standards

– GRIP/LDAP/MDS: Abstractions integrated into OGSI as serviceData

– GRAM: ManagedJobFactory and related service definitions

– GridFTP: Unchanged in 3.0, but will evolve into OGSI-compliant service in 2003

Also rendering collective services in terms of OGSI: RFT, RLS, CAS, etc.

24


GT Timeline

GT 1.0: 1998– GRAM, MDS

GT 2.0: 2001– GridFTP, packaging, reliability

GT3 Technology Preview: Apr-Dec 2002– Tracking OGSI definition

GT3.0 Alpha: Jan 2003– OGSI Base, GT2 functionality

GT3.0 Production: June 2003– Tested, documented, etc.

25


Globus Toolkit Contributors: GT2

Grid Packaging Technology (GPT) NCSA Persistent GRAM Jobmanager Condor GSI/Kerberos interchangeability Sandia Documentation NASA, NCSA Ports IBM, HP, Sun, SDSC, … MDS stress testing EU DataGrid Support IBM, Platform, UK eScience Testing and patches Many! Interoperable tools Many! $$ DARPA, DOE, NSF, NASA, Microsoft, EU

26


Globus Toolkit Contributors: GT3

Replica location service EU DataGrid Python hosting environment LBNL Data access & integration UK eScience Data mediation services SDSC Tooling, Xindice, JMS IBM ... ... ...

27


The Grid Technology Repository

Community repository

Clearing house for service definitions, code, documentation

Encourage collaboration & avoid redundant work

http://gtr.globus.org

International advisory committee: Ian Foster (Chair), Malcolm Atkinson, John Brooke, Fabrizio Gagliardi, Dennis Gannon, Wolfgang Gentzsch, Andrew Grimshaw, Keith Jackson, Gregor von Laszewski, Satoshi Matsuoka, Jarek Nabrzyski, Bill St. Arnaud, Jay Unger

28


GT Processes Are Evolving Rapidly(Thanks to Partners & the GT Team!)

Before 2000– Email-based problem tracking,

aka “req”

2000– Detailed documentation,

release notes (Q1)

– Legal framework for external contributions (Q1)

2001– Packaging; module & binary

releases (Q4)

– Substantial regression tests (Q4)

2002– Bugzilla problem reporting &

tracking (Q2)– Processes for external contrib

(Q2)– Distributed testing infrastructure

(Q3) 2003 (in progress)

– Distributed support infrastructure: GT “support centers”

– Standardized Grid testing framework(s)

– GT “contrib” components– Grid Technology Repository

29


OGSA Misconceptions

OGSA means you have to code in Java– No: C client bindings now, C server side

eventually (but not needed for current apps) OGSA means all programs must be services

– No: You can write services if you want, but GT2-style GRAM behavior is still supported (GRAM is just a server)

OGSA is a silver bullet for Grid computing– No, it makes some things easier, but it’s only

interfaces and behaviors, after all!

30


Summary

OGSA: standards-based Grid technology– From Web services: standard IDL, discovery, binding

independence, other desirable features

– From Grid: naming, state, lifetime management, etc., etc.

Rapid progress on definition & implementation– OGSI is defined, GT3 implements it (and other things),

multiple groups coding to it

– Much more happening, much more to be done! No silver bullet, but a good incremental step forward

to our ultimate Grid software goals

ian foster argonne national laboratory university of chicago foster open grid services architecture...

Documents

virtual services

argonne chicago web

interoperable services

interesting services

appspecific services

service various

service model

virtual slide