grid computing

Grid Computing

Ian Foster

Mathematics and Computer Science Division

Argonne National Laboratory

and

Department of Computer Science

The University of Chicago

http://www.mcs.anl.gov/~foster

2

[email protected] ARGONNE CHICAGO

Partial Acknowledgements Open Grid Services Architecture design

Carl Kesselman, Karl Czajkowski @ USC/ISI Steve Tuecke @ANL Jeff Nick, Steve Graham, Jeff Frey @ IBM

Grid services collaborators at ANL Kate Keahey, Gregor von Laszewski Thomas Sandholm, Jarek Gawor, John Bresnahan

Globus Toolkit R&D also involves many fine scientists & engineers at ANL, USC/ISI, and elsewhere (see www.globus.org)

Strong links with many EU, UK, US Grid projects Support from DOE, NASA, NSF, IBM, Microsoft

3


Goals Communicate the purpose, significance,

state, adoption, & future of Grid technology Persuade you that Grid technology

represents an opportunity Grids aren’t (particularly) about science or

servers—themes of virtualization, service discovery, service management, and QoS delivery are universal

Rapid uptake in industry & science represents an exceptional opportunity for impact

4


Overview

Origins: Resource sharing within scientific collaborations Science drivers & science Grid projects Globus Toolkit

Evolution: Resource virtualization Commercial drivers OGSA: Grid meets Web services

5


Overview



6


E-Science: The Original Grid Driver

Pre-electronic science Theorize &/or experiment, in small teams

Post-electronic science Construct and mine very large databases Develop computer simulations & analyses Access specialized devices remotely Exchange information within distributed

multidisciplinary teams Need to manage dynamic, distributed

infrastructures, services, and applications

7

[email protected] ARGONNE CHICAGO www.griphyn.org/chimera

Size distribution ofgalaxy clusters?

1

10

100

1000

10000

100000

1 10 100

Num

ber

of C

lust

ers

Number of Galaxies

Galaxy clustersize distribution

Chimera Virtual Data System+ GriPhyN Virtual Data Toolkit

+ iVDGL Data Grid (many CPUs)

eScience Application: Sloan Digital Sky Survey Analysis

8


•Lift Capabilities•Drag Capabilities•Responsiveness

•Deflection capabilities•Responsiveness

•Thrust performance•Reverse Thrust performance•Responsiveness•Fuel Consumption

•Braking performance•Steering capabilities•Traction•Dampening capabilities

Crew Capabilities- accuracy- perception- stamina- re-action times- SOPs

Engine Models

Airframe Models

Wing Models

Landing Gear Models

Stabilizer Models

Human Models

NASA’s Information Power Grid: Aviation Safety

9


NETWORK

IMAGINGINSTRUMENTS

COMPUTATIONALRESOURCES

LARGE DATABASES

DATA ACQUISITIONPROCESSING,

ANALYSISADVANCED

VISUALIZATION

Life Sciences: Telemicroscopy

10


And Thus: The Grid

“Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations”

11


Underlying Technical Requirements

Dynamic formation and management of virtual organizations

Online negotiation of access to services: who, what, why, when, how

Configuration of applications and systems able to deliver multiple qualities of service

Autonomic management of distributed infrastructures, services, and applications

12


The Grid World: Current Status Dozens of major Grid projects in scientific &

technical computing/research & education Open source Globus Toolkit™ a de facto standard

for major protocols & services Simple protocols & APIs for authentication, discovery,

access, etc.: infrastructure Information-centric design Large user and developer base Multiple commercial support providers Enabler of numerous tools and applications

Global Grid Forum: community & standards

13


Overview



14


Resource Sharing within “VOs” is Not Unique to Science!

Fragmentation of enterprise infrastructure Driven by cheap servers, fast nets, ubiquitous

Internet, eBusiness workloads Need to configure distributed collections of services

to deliver specified QoS Virtualization

Emerging service infrastructure, utility computing models, economies of scale

Services dynamically instantiated across device spectrum

B2B, B2C, C2C interactions

15


Virtualization andDistributed Service Management

Less capable, integratedLess connected

User service locus

Larger, more integratedMore connected

Dynamically provisioned Device Continuum

Resource &service

aggregation

Delivery of virtualized services with QoS

guaranteesDynamic, secureservice discovery

& composition

Distributed servicemanagement

16


Realizing the PromiseRequires Significant Innovation

Automation of infrastructure operation to achieve economies of scale

Management and component models for service discovery, composition, provisioning

New applications and tools powered by distributed services and resources

Business and service models to support specialization of function

17


Grid Evolution:Open Grid Services Architecture

Refactor Globus protocol suite to enable common base and expose key capabilities

Service orientation to virtualize resources and unify resources/services/information

Embrace key Web services technologies: WSDL as IDL, leverage commercial efforts

Result: standard interfaces & behaviors for distributed system management

18


OGSA System Structure

A standard substrate: the Grid service Standard interfaces and behaviors that address

key distributed system issues The “Grid Service Specification”

… supports standard service specifications Resource management, databases, workflow,

security, diagnostics, etc., etc. Target of current & planned GGF efforts

… and arbitrary application-specific services based on these & other definitions

19


Transient Service Instances “Web services” address discovery & invocation of

persistent services Interface to persistent state of entire enterprise

In Grids, must also support transient service instances, created/destroyed dynamically Interfaces to the states of distributed activities E.g. workflow, video conferencing, distributed data

analysis, workload management Significant implications for how services are

named, discovered, managed, and used

20


OGSI, OGSA, and Web Services

OGSI (I = Infrastructure) Small extensions to WSDL

Nested serviceType & serviceDataDescription

Conventions for naming service instances Handles and references

portTypes for common behavior Instance creation, lifetime management, introspection and

monitoring, registration, notification, …

OGSA (A = Architecture) built on OGSI A collection of Grid service interfaces

Resource description & provisioning Higher-level services: messaging services, logging, etc.

21


The Grid Service =Interfaces/Behaviors + Service Data

Servicedata

element

Servicedata

element

Servicedata

element

Implementation

GridService(required)Service data access

Explicit destructionSoft-state lifetime

… other interfaces …(optional) Standard:

- Notification- Authorization- Service creation- Service registry- Manageability- Concurrency

+ application-specific interfaces

Binding properties:- Reliable invocation- Authentication

Hosting environment/runtime(“C”, J2EE, .NET, …)

22


Service Data A Grid service instance maintains a set of service data

elements Described in WSDL extension XML element encapsulated in standard container: name,

type, lifetime, etc. Includes basic introspection information, interface-specific

data, and application state Pull and push models for information query

GridService::FindServiceData operation Pull: queries this information via extensible query language

NotificationSource::SubscribeServiceData Push: Subscribe to notification of changes to information

23


Notification Interfaces NotificationSource for client subscription

Subscription expression describes which service data element changes are of interest

Creates a subscription manager service Manages the lifetime and properties of subscription

NotificationSink for asynchronous delivery of notification messages

Simple, flexible base with wide variety of uses Dynamic discovery/registry services, monitoring, application

error notification, etc. Intermediaries: filter, aggregate, archive, et.c Can integrate commercial messaging services

24


Grid Service Example:Database Service

A DBaccess Grid service will support at least two portTypes GridService DBaccess

Each has service data GridService: basic introspection information, lifetime,

… DBaccess: database type, query languages

supported, current load, …, … Maybe other portTypes as well

E.g., NotificationSource

GridService DBaccess

DB info

Name, lifetime, etc.

25


Lifetime Management GS instances created by factory or manually;

destroyed explicitly or via soft state Negotiation of initial lifetime with a factory

(=service supporting Factory interface) GridService interface supports

Destroy operation for explicit destruction SetTerminationTime operation for keepalive

Soft state lifetime management avoids Explicit client teardown of complex state Resource “leaks” in hosting environments

26


Factory

Factory interface’s CreateService operation creates a new Grid service instance Reliable creation (once-and-only-once)

CreateService operation can be extended to accept service-specific creation parameters

Returns a Grid Service Handle (GSH) A globally unique URL, resolves to GSR Uniquely identifies the instance for all time Based on name of a handle resolver

Or Grid Service Reference (GSR)

28


Example:Transient Database Services

GridService

DBaccessFactory

Factory info

Instance name, etc.

GridService Registry

Registry info

Instance name, etc.


DB info



DB info


“What services can you create?”

“What database services exist?”

“Create a database service”

29


OGSA Design & Implementation

OGSI (I=Infrastructure) WG in GGF defining core Grid service specification (At least) three implementation efforts

Globus Toolkit => GT3 (alpha end 2002) GT3 Core: Grid service specification GT3 Base: Globus Toolkit behaviors CIM resource model, GRAM-2 SLA negotiation,

database services, … Other GGF WGs address OGSA security, OGSA-

compliant database services, etc.

30


Recap: Goals

Communicate the purpose, significance, state, adoption, & future of Grid technology

Persuade you that Grid technology represents a significant opportunity Grids aren’t only (or particularly) about

science and servers—themes of virtualization, service discovery, service management, and QoS delivery are universal

Rapid uptake in industry & science represents an exceptional opportunity for impact

31


For More Information

The Globus Project™ www.globus.org

Context & research articles www.mcs.anl.gov/~foster

Open Grid Services Architecture www.globus.org/ogsa

Global Grid Forum www.gridforum.org Edinburgh, July 22-24 Chicago, Oct 15-17

32


OGSA Implementation

Hosting EnvironmentResource virtualization

and QoS support

StandardOGSI

container

Webservicesvarious

OGSI/OGSA Interfacesservice description,

service provisioning, …

2) to enable virtualization via Service description Service provisioning

OGSI/OGSA Interfacesservice description,

service provisioning, …

3) Standard container avoids implementing OGSI features in every service instance

StandardOGSI

container

Hosting EnvironmentResource virtualization

and QoS support

Webservicesvarious

1) OGSA builds on infrastructure Plumbing: WSDL, WS-Security, WS-

Routing/Referral, reliable messaging, transactions, etc.

Hosting environments

33


Building an OGSI Container

Service data mgmt, query, subscription Container should provide simple interface for

interacting with an instance’s implementation to get and manage dynamic service data

Service instance = CLR object

Container should handle query processing .NET support for XPath & Xquery allows for rich functionality

Container manages notification subscriptions, and drives asynchronous notification messages

Soft-state lifetime management Soft-state registration

grid computing

Documents

grid resource

grid evolution

grid world

service models

distributed services

web services

ivdgl data grid

future of grid technology