building and managing an open grid

Building and Managingan Open Grid

Ian Foster

Argonne National Laboratory

University of Chicago

http://www.mcs.anl.gov/~foster

Keynote talk at 8th IFIP/IEEE Intl Symp. on Integrated Network Management, March 26, 2003

2

[email protected] ARGONNE CHICAGO

Grids andIntegrated Network Management

2003: Production use in science & industry for distributed resource & service integration– Grids as management problem: definitely

– Grid technologies as management solution?

1995: I-WAY– Integration of HPC

resources for advanced scientific applications

– Management not quite on the agenda

3


Overview

The Grid: why and what Building an open Grid

– Open standards

– Open source

– Open infrastructure Summary

4


The Grid:On-Demand Access to Electricity

Time

Qua

lity,

eco

nom

ies

of s

cale

5


By Analogy, A Computing Grid

Decouple production and consumption– Enable on-demand access

– Achieve economies of scale

– Enhance consumer flexibility

– Enable new devices On a variety of scales

– Department

– Campus

– Enterprise

– Internet

6


Not Exactly a New Idea …

“The time-sharing computer system can unite a group of investigators …. one can conceive of such a facility as an … intellectual public utility.”– Fernando Corbato and Robert Fano , 1966

“We will perhaps see the spread of ‘computer utilities’, which, like present electric and telephone utilities, will service individual homes and offices across the country.”– Len Kleinrock, 1967

7


But, Things are Different Now Networks are far faster (and cheaper)

– Faster than computer backplanes “Computing” is very different than pre-Net

– Our “computers” have already disintegrated

– E-commerce increases size of demand peaks

– Entirely new applications & social structures We’ve learned a few things about software

8


Computing isn’t Really Like Electricity

I import electricity but must export data “Computing” is not interchangeable but highly

heterogeneous: data, sensors, services, … This complicates things; but also means that the sum

can be greater than the parts – Real opportunity: Construct new capabilities dynamically

from distributed services Raises three fundamental questions

– Can I really achieve economies of scale?

– Can I achieve QoS across distributed services?

– Can I identify apps that exploit synergies?

9


Why the Grid?(1) Revolution in Science

Pre-Internet– Theorize &/or experiment, alone

or in small teams; publish paper Post-Internet

– Construct and mine large databases of observational or simulation data

– Develop simulations & analyses

– Access specialized devices remotely

– Exchange information within distributed multidisciplinary teams

10


Why the Grid?(2) Revolution in Business

Pre-Internet– Central data processing facility

Post-Internet– Enterprise computing is highly distributed,

heterogeneous, inter-enterprise (B2B)

– Business processes increasingly computing- & data-rich

– Outsourcing becomes feasible => service providers of various sorts

11


New OpportunitiesDemand New Technology

“Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations”

12


Building an Open Grid

13



OpenStandards

14



OpenStandards

OpenSource

15



OpenStandards

OpenSource

OpenInfrastructure

16



OpenStandards

OpenSource

OpenInfrastructure

OpenGrid

17



OpenStandards

OpenSource

OpenInfrastructure

OpenGrid

18


Grids and Open Standards

Incr

ease

d fu

nctio

nalit

y,st

anda

rdiz

atio

n

Time

Customsolutions

Open GridServices Arch

GGF: OGSI, …(+ OASIS, W3C)

Multiple implementations,including Globus Toolkit

Web services

Globus Toolkit

Defacto standardsGGF: GridFTP, GSI

X.509,LDAP,FTP, …

App-specificServices

19


Open Grid Services Architecture

Service-oriented architecture– Key to virtualization, discovery,

composition, local-remote transparency Leverage industry standards

– Internet, Web services Distributed service management

– A “component model for Web services” A framework for the definition of

composable, interoperable services

“The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration”, Foster, Kesselman, Nick, Tuecke, 2002

20


Realizing a Service-Oriented Architecture: How Do I

Create, name, manage, discover services? Render resources, data, sensors as services? Negotiate service level agreements? Express & negotiate policy? Organize & manage service collections? Establish identity, negotiate authentication? Manage VO membership & communication? Compose services efficiently? Achieve interoperability?

21


Web Services

XML-based distributed computing technology Web service = a server process that exposes typed

ports to the network Described by the Web Services Definition Language, an

XML document that contains– Type of message(s) the service understands & types of

responses & exceptions it returns

– “Methods” bound together as “port types”

– Port types bound to protocols as “ports” A WSDL document completely defines a service and

how to access it

22


Open Grid Services Infrastructure

Implementation

Servicedata

element

Other standard interfaces:factory,

notification,collections

Hosting environment/runtime(“C”, J2EE, .NET, …)

Servicedata

element

Servicedata

element

GridService(required)

Dataaccess

Lifetime management• Explicit destruction• Soft-state lifetime

Introspection:• What port types?• What policy?• What state?

Client

Grid ServiceHandle

Grid ServiceReference

handleresolution

23


Open Grid Services Infrastructure

GWD-R (draft-ggf-ogsi- gridservice-23) Editors:Open Grid Services Infrastructure (OGSI) S. Tuecke, ANLhttp://www.ggf.org/ogsi-wg K. Czajkowski, USC/ISI

I. Foster, ANLJ. Frey, IBMS. Graham, IBMC. Kesselman, USC/ISID. Snelling, Fujitsu LabsP. Vanderbilt, NASAFebruary 17, 2003

Open Grid Services Infrastructure (OGSI)

24


The OGSA Platform

OGSI

TransportProtocolHosting EnvironmentHosting Environment

Host. Env. & Protocol Bindings

OGSA Platform services: registry,authorization, monitoring, data

access, etc., etc.

More specialized &domain-specific

services

Models for resources &

other entitiesO

therm

odels

Environment-specificprofiles

Domain-specificprofiles

OGSAPlatform

GWD-R (draft-ggf-ogsa-platform-3) Editors:Open Grid Services Architecture Platform I. Foster, Argonne & U.Chicagohttp://www.ggf.org/ogsa-wg D. Gannon, Indiana U.

25


Example:Reliable File Transfer Service

Performance

Policy

Faults

servicedataelements

Pending

FileTransfer

InternalState

GridService

Notf’nSource

Policy

interfacesQuery &/orsubscribe

to service data

FaultMonitor

Perf.Monitor

Client Client Client

Request and manage file transfer operations

Data transfer operations

26


Open Grid Service Architecture:Next Steps

Technical specifications– Open Grid Services Infrastructure is complete

– Security, data access, Java binding, common resource models, etc., etc., in the pipeline

Implementations and compliant products– OGSA-based Globus Toolkit v3, pyGlobus, …

– IBM, Avaki, Platform, Sun, NEC, Oracle, … Rich set of service defns & implementations

– Time to start on OGSI-compliant services!

27


OGSA Manageability

Management is a high-priority OGSA effort– GGF Common Management Model (CMM) WG

– Ellen Stokes (IBM) co-chair Goal:

– Define standard schema and interfaces for a manageable resource (modeled as a service)

And also:– Allow existing models to be used & exposed

– Leverage CIM schema when applicable

– Define how CMM interfaces can integrate with higher-level interfaces (e.g., provisioning)

28


Common Management Model A manageable resource is a Grid service, thus

– Global resource names: Grid service handles

– State data modeling + access: SDEs

– Lifetime management

– Service Group for grouping resources

– Interface definition language: WSDL Plus additional schema & operations

– Standard manageable resource SDE schema

– Interfaces for extensible lifecycle and relationship management

BaseManageableResource interface

29


Base Manageable Port Types

GridService

Locate Relationship LifecycleModel

BaseManageableResource

ServiceGroupHandleResolver

OGSA port types

CRM port types

30


CMM Schema

WSDL (open content model) & XSD describe resource’s manageable attributes (as SDEs)

Models are CIM-based where applicable– Re-factored for service efficiency; appropriate for

higher-level management applications

– Additional XML attributes: change control, measuring, lifecycle

– Additional XML data types CIM as basis for schemas: but may modify

– E.g. use constructs from XML/XSD where similar ones exist in CIM

31


“Use CIM Models Where Applicable”

Class is port type, properties of class are port type service data, methods of class are port type operations– Some refactoring of classes -> portTypes

Express in WSDL/GSDL as grid service– Managed resource port type from which other

resource port types are derived

– Mix in the base GridService port type

– Mix in other CMM port types as needed (Identity, relationship, lifecycleState)

32


CMM Bindings and Operations

A resource service may have more than one binding to access that resource’s manageability information

A resource may support more than one service implementation– e.g., Native, SNMP, CIM, RMC, etc

A resource can still be accessed (outside OGSA) using its existing implementation, resource definition, and access interface, e.g., SNMP, CIM, etc.

33


Basic Manageability Port Types

Needed to manage the resource– lifecycleModel

– relationship

– locate Others defined

– From CIM schema, e.g., opererational operations (start/stop/resume/pause)

– By OGSO spec, e.g. grid service, notification

– By other OGSA components, e.g. policy, logging/metering

34


lifecycleModel Port Type:A Container for Lifecycle States

There may be multiple models, but only one for a given resource’s port type

Example: Get/set resource’s lifecycle state – down, starting, up, stopping, failed

– Each state has additional info, e.g.,> up state: idle, busy, degraded

Down Starting

UpStopping

Failed

35


Relationships & Dependencies

Relationships describe existence and type of connections between resource instances– Relationships are discovered through the

relationship port type and its relatedResource service data element

– Relationship port type allows a view of relationships as they are known by the resources at each end of the relationship

Set of predefined relationship types– Hosts, Contains, Federates, Aggregates, Uses,

Implements

36


Locate Port Type

Provided to help in the location of manageable resources of a given type

Contains two service data elements– serviceGroupType, port type that provides the

ServiceGroup function for manageable resources of this type

– searchProperty, zero or more service data elements that are likely to be used for searching for a manageable resource, and are thus worth caching in the manageable resource registry

37



OpenStandards

OpenSource

OpenInfrastructure

OpenGrid

38


Open Source

Encourage adoption of standards by reducing barriers to entry– Overcome new technology Catch-22

Enable broad Grid technology ecosystem– Key to international cooperation

– Key to science-commerce partnership Jumpstart Grid industry and allow vendors to

focus on value-add– E.g., IBM, Avaki use GT3; Platform Globus

– Open source is industry friendly!

39


Globus Toolkit v3 (GT3)Open Source OGSA Technology

Implement core OGSI interfaces Support primary GT2 interfaces

– High degree of backward compatibility Multiple platforms & hosting environments

– J2EE, Java, C, .NET, Python New services

– SLA negotiation (GRAM-2), registry, replica location, community authorization, data, …

Growing external contributions & adoption

40


Globus Toolkit Contributors: GT2

Grid Packaging Technology (GPT) NCSA Persistent GRAM Jobmanager Condor GSI/Kerberos interchangeability Sandia Documentation NASA, NCSA Ports IBM, HP, Sun, SDSC, … MDS stress testing EU DataGrid Support IBM, Platform, UK eScience Testing and patches Many! Interoperable tools Many! $$ DARPA, DOE, NSF, NASA, Microsoft, EU

41


Globus Toolkit Contributors: GT3

Replica location service EU DataGrid Python hosting environment LBNL Data access & integration UK eScience Data mediation services SDSC Tooling, Xindice, JMS IBM ... ... ...

42


Open Source is More than Software!

Success of Grid concept demands effective community mechanisms for coordinated– R&D for core technologies

– Testing, packaging, documentation

– Support and training of user communities All three must become collaborative activities

– Based on standards

– Centered on a common code base

– Supported by appropriate tools

– United by appropriate processes & governance

43


For Example:Distributed Test Facility

(Proposed activity for NSF GRIDS Center: grids-center.org)

44



OpenStandards

OpenSource

OpenInfrastructure

OpenGrid

45


The Gridas Enabler of 21st Century Science

Entirely new approaches to enquiry based on– Deep analysis of huge quantities of data

– Interdisciplinary collaboration

– Large-scale simulation

– Smart instrumentation Enabled by an infrastructure that enables

access to, and integration of, resources & services without regard for location

46


Grid Infrastructure Broadly deployed services in support of fundamental

collaborative activities– Formation & operation of virtual organizations

– Authentication, authorization, discovery, … Services, software, and policies enabling on-demand

access to critical resources– Computers, databases, networks, storage, software

services,… Operational support for 24x7 availability Integration with campus and commercial

infrastructures

47


Tier0/1 facility

Tier2 facility

10 Gbps link

2.5 Gbps link

622 Mbps link

Other link

Tier3 facility

The Foundations are Being Laid

Cambridge

Newcastle

Edinburgh

Oxford

Glasgow

Manchester

Cardiff

Soton

London

Belfast

DL

RAL Hinxton

48


Summary

An open Grid technology has been created– Standards, software, infrastructure

– All three must be collaborative activities We are now scaling this technology base, to

– Meet the needs of the science community

– Achieve synergy between open & commercial, scientific & industrial contributors

Management is a crucial part of this effort– One might argue, “Grid = management”

– We need help from your community

building and managing an open grid

Documents

globus toolkit

chicagothe grid

chicagooverviewthe grid

economies of scale grid

distributed servicesraises

new capabilities

new idea

management problem