promoting and standardizing grid computing defining the grid: open grid services architecture...

Promoting and Standardizing Grid Computing

Defining the Grid: Open Grid Services Architecture

Current and Future Generation Grid Technology

Summer School 2006

24th - 28th July, Bonn (Germany)

Hiro Kishimoto, Ph.D.

OGF OGSA-WG, co-chair

Research Fellow, Fujitsu

Visiting Professor, National Institute of Informatics

OGSA is a trademark of the Open Grid Forum

July 24, 2006 Open Grid Forum - www.ogf.org 2

Agenda• Open Grid Services Architecture

Overview

• A Closer Look at OGSA

• OGSA and the Standards Landscape

• The OGSA Working Group

• Summary


The Open Grid ServicesArchitecture

Open Grid Forum - www.ogf.org 4July 24, 2006

Three Generations of Grid

Source: Charlie Catlett

StandardizationStandardization is key for third-generation grids! is key for third-generation grids!StandardizationStandardization is key for third-generation grids! is key for third-generation grids!

• Local “metacomputers“– Distributed file systems– Site-wide single sign-on

• "Metacenters" explore inter-organizational integration• Totally custom-made, top-to-bottom: proofs of concept

• Local “metacomputers“– Distributed file systems– Site-wide single sign-on

• "Metacenters" explore inter-organizational integration• Totally custom-made, top-to-bottom: proofs of concept

• Utilize software services and communications protocols developed by grid projects:

– Condor, Globus, UNICORE, Legion, etc.

• Need significant customization to deliver complete solution• Interoperability is still very difficult!

• Utilize software services and communications protocols developed by grid projects:

– Condor, Globus, UNICORE, Legion, etc.

• Need significant customization to deliver complete solution• Interoperability is still very difficult!

• Common interface specifications support interoperability of discrete, independently developed services

• Competition and interoperability among applications, toolkits, and implementations of key services

• Common interface specifications support interoperability of discrete, independently developed services

• Competition and interoperability among applications, toolkits, and implementations of key services

We are here!

We are here!


The Open Grid Services Architecture

• An open, service-oriented architecture (SOA)− Resources as first-class entities− Dynamic service/resource creation and destruction

• Built on a Web services infrastructure

• Resource virtualization at the core

• Build grids from small number of standards-based components− Replaceable, coarse-grained− e.g. brokers

• Customizable− Support for dynamic, domain-specific content…− …within the same standardized framework


• Logical view of capabilities

• Relatively coarse-grained functions

• Reusable and composable behaviors

• Encapsulation of complex operations

• Naturally extendable framework

• Platform-neutral– machine and OS

Why Use an SOA?


SOA & Web Services: Key Benefits

SOA• Flexible

− Locate services on any server− Relocate as necessary− Prospective clients find services using

registries

• Scalable− Add & remove services as demand

varies

• Replaceable− Update implementations without

disruption to users

• Fault-tolerant− On failure, clients query registry for

alternate services

SOA• Flexible

− Locate services on any server− Relocate as necessary− Prospective clients find services using

registries

• Scalable− Add & remove services as demand

varies

• Replaceable− Update implementations without

disruption to users

• Fault-tolerant− On failure, clients query registry for

alternate services

Web Services• Interoperable

− Growing number of industry standards

• Strong industry support• Reduce time-to-value

− Harness robust development tools for Web services

− Decrease learning & implementation time

• Embrace and extend− Leverage effort in developing and

driving consensus on standards− Focus limited resources on

augmenting & adding standards as needed

Web Services• Interoperable

− Growing number of industry standards

• Strong industry support• Reduce time-to-value

− Harness robust development tools for Web services

− Decrease learning & implementation time

• Embrace and extend− Leverage effort in developing and

driving consensus on standards− Focus limited resources on

augmenting & adding standards as needed


Virtualizing Resources

Resources

Webservices

AccessAccess

StorageStorage SensorsSensors ApplicationsApplications InformationInformationComputersComputers

Resource-specific InterfacesResource-specific Interfaces

Common Interfaces

Type-specific interfaces


A Service-Oriented Grid

Virtualized resources

Grid middleware

services

Brokering Service

Brokering Service

Registry Service

Registry Service

DataService

DataService

CPU ResourceCPU ResourcePrinter Service

Printer Service

Job-Submit Service

Job-Submit Service

ComputeService

ComputeService

No

tify

Ad

vertise

ApplicationService

ApplicationService


A Closer Look at OGSA


OGSA Capabilities

Security• Cross-organizational users• Trust nobody• Authorized access only

Security• Cross-organizational users• Trust nobody• Authorized access only

Information Services• Registry• Notification• Logging/auditing

Information Services• Registry• Notification• Logging/auditing

Execution Management• Job description & submission• Scheduling• Resource provisioning

Execution Management• Job description & submission• Scheduling• Resource provisioning

Data Services• Common access facilities• Efficient & reliable transport• Replication services

Data Services• Common access facilities• Efficient & reliable transport• Replication services

Self-Management• Self-configuration• Self-optimization• Self-healing

Self-Management• Self-configuration• Self-optimization• Self-healing

Resource Management• Discovery• Monitoring• Control

Resource Management• Discovery• Monitoring• Control

OGSAOGSA

OGSA “profiles”OGSA “profiles”

Web services foundation Web services foundation


CDLCDL

3. Select from or deployrequired resources

3. Select from or deployrequired resources

Execution Management

• The basic problem− Execute and manage jobs/services in the grid− Select from or provision required resources

• The basic problem− Execute and manage jobs/services in the grid− Select from or provision required resources

2. Submit the job2. Submit the job

1. Describe the job1. Describe the job

JSDLJSDL

Job

4. Manage the job4. Manage the job


Describing a Job Submission: JSDL• Job Submission Description Language (JSDL)

− A language for describing the requirements of jobs for submission− Declarative description

• A JSDL document describes the job requirements − Job identification information − Application (e.g., executable, arguments)− Required resources (e.g., CPUs, memory)− Input/output files

JobJob

IT InfrastructureIT Infrastructure

JSDLJSDL


Configuration & Deployment: CDL• Prepare the infrastructure so that the job can execute

− Provide a right-shaped slot to fit the job

• Main parts:− Configuration Description Language (CDL) provides declarative definition of

system configuration− Deployment service carries out configuration requests to deploy and

configure the system

IT InfrastructureIT Infrastructure

CDLCDL

Prepare


IssuesIssues

Find Describe

Access

DataDataDataData

Formats

ProtocolsProtocols

Use casesUse cases

DataDataDataData

DataData

DataDataMove/Copy/ReplicateMove/Copy/Replicate

Metadata Metadata

DataData

ManageManage

Common

accessCommon

access

Data Services

The basic problem

• Manage, transfer and access distributed data services and resources

The basic problem

• Manage, transfer and access distributed data services and resources

Derived dataCatalog

Sensor Data stream

Text file

Relational database


Basic Data Services

Data Resources

Data Resources

Managed StorageManaged Storage

Data Resources

Data Resources Transfer ProtocolsTransfer Protocols

Storage Management

Storage Management

Data Management

Data Management

Other Data Services

Other Data Services

TransferTransfer RegistriesRegistries

Other OGSA services and non-OGSA client APIsOther OGSA services and non-OGSA client APIs

Serviceinterface

Resourceinterface


Data Services

Data Service n

Data Service n

Data Service 1

Data Service 1

Data Service 2

Data Service 2

Composite Data Services

ReplicationReplication

CacheCache

FederationFederation


Basic Data Interfaces

• Storage Management− e.g. Storage Resource

Management (SRM)

• Storage Management− e.g. Storage Resource

Management (SRM)

• Data Access− ByteIO− Data Access & Integration

(DAI)

• Data Access− ByteIO− Data Access & Integration

(DAI)

• Data Transfer− Data Movement Interface

(DMI)− Protocols (e.g. GridFTP)

• Data Transfer− Data Movement Interface

(DMI)− Protocols (e.g. GridFTP)

• Replica management

• Metadata catalog

• Cache management

• Replica management

• Metadata catalog

• Cache management


Resource Management

• Provides a framework to integrate resource management functions− interfaces, services, information models, etc.

• Enables integrated discovery, monitoring, control, etc.

• Provides a framework to integrate resource management functions− interfaces, services, information models, etc.

• Enables integrated discovery, monitoring, control, etc.

High-levelmanagementservices(GGF)

Domain-specific capabilitiesDomain-specific capabilities

OGSA

Access tomanageability(OASIS, DMTF)

Informationmodels (DMTF,SNIA, etc.)

Resources

WSDM, WS-ManagementWSDM, WS-Management

WSRF/WSN, WS-Transfer/EventingWSRF/WSN, WS-Transfer/Eventing

Dataservices

Securityservices

ExecutionManagement

services

Application-specific


Self-Management

Self-

Management

Self-

Management

MonitoringMonitoring

ProjectionProjection

AnalysisAnalysisActionAction

PolicyPolicyPolicyPolicyPolicyPolicy

PolicyPolicyPolicyPolicySLASLA

• Self-configuration: Automatically adapt to changes in the environment:− e.g. Deploy/undeploy resources as load changes

• Self-configuration: Automatically adapt to changes in the environment:− e.g. Deploy/undeploy resources as load changes

• Self-optimization: Automatically tune system to best meet user or business needs− Uses service-level agreements (SLAs)

• Self-optimization: Automatically tune system to best meet user or business needs− Uses service-level agreements (SLAs)

• Self-healing: Automatically detect & correct problems− Component failures− Security violations− etc.

• Self-healing: Automatically detect & correct problems− Component failures− Security violations− etc.


Information Services

Executionmanagement

Resourcereservation

Problemdetermination

Accounting

Applicationmonitoring

Loadbalancing

Servicediscovery

Consumers ConsumersInformationServices

InformationServices

• Reliable

• Secure• Efficient

Provide management and access facilities for information about applications and resources in the grid environment

Provide management and access facilities for information about applications and resources in the grid environment

ProducersProducers

Asynchronous notification

Retrieval

RegistryRegistry

LoggerLogger


Security Services• Authorization, roles, and access privileges

− Locally (site) managed− Based on SAML and XACML security standards− Implementations provide credential mapping

• Working with GGF Security Area groups− Authorization attributes for grids

• Developing OGSA basic security profiles

PKIcertificate

PKIcertificate

WS-SecurityWS-Security

WS-AddressingWS-Addressing

OGSA

security profilesOGSA

security profiles


OGSA Profiles• The normative definition of OGSA

− Styled on WS-I profiles to promote interoperability− Define specific usage patterns

• e.g. execution management

− Basis for claims of conformance• “My scheduler conforms to the OGSA Execution Management Profile…”

• Include specifications developed by OGF and by other bodies− Issue: How mature and widely adopted?− OGSA Profile Definition document provides guidelines

In the futureEarly stagesIn the pipeline

Data ProfileData Profile

OGSA WSRF Basic Profile OGSA WSRF Basic Profile

OGSA Basic Security Profile – CoreOGSA Basic Security Profile – Core

OGSA Basic Security Profile – Secure ChannelOGSA Basic Security Profile – Secure Channel

Execution Mgmt ProfileExecution Mgmt Profile

Others…Others…

HPC ProfileHPC Profile


OGSA and theStandards Landscape


Specifications Landscape: July 2006

SYSTEMSMANAGEMENT

UTILITYCOMPUTING

GRIDCOMPUTING

Core Services

Web ServicesFoundation WS-Addressing

Privacy

WS-BaseNotification

CIM/JSIM

WSRF-RAP

WSDM

WS-Security

Naming

OGSA-EMSOGSA Self Mgmt

GFD-C.16

GGF-UR

Data Model

HTTP(S)/SOAP

Discovery

SAML/XACML

WSDL

WSRF-RL

Trust

WS-DAI

VO Management

Information

Distributed query processing

ASP

Data CentreUse Cases &Applications Collaboration Multi MediaPersistent Archive

Data Transport

WSRF-RP

X.509

StandardEvolvingGapHole

Warning: Volatile data!Warning: Volatile data!


Related Standards Bodies

• OGF (Open Grid Forum)− Overall architecture for grid computing− OGSA, CDDLM, WS-Agreement, …

• OASIS (Organization for the Advancement of Structured Information Standards)− Middleware/Web services focused− WSRF, WS-Notification, WSDM, WS-Security…

• DMTF (Distributed Management Task Force)− Management and Information models (CIM)− Server management− WS-Management

− WS-CIM

• W3C (WWW Consortium)

− WS-Addressing

− WS-PolicyNot a complete list!Not a complete list!

Logos are trademarks or service marks of their respective owners.


OGF SCRM-WG

• Formed at GGF14

• Online reference guide of standards− Taxonomy− Glossary

• Live wiki document available

SDOs Collaboration on networked Resourced Management- WG

DMTF


Making It Happen:The OGSA Working Group


The OGSA Working Group

• History− Established September 2002− Declared OGF’s “flagship architecture” March 2004

• Getting it done− Over 300 mailing-list subscribers− Twice-weekly teleconferences− Regular face-to-face meetings (at least 3+3 per year)− Collaboration with other working groups and standards organizations

• Contributors− Industry− Government− Academia


Aims & Perspective

• Goals− Interoperable solutions for grid-based applications− Addressing loosely coupled distributed computing

• Approach− Standardization at the architectural level

• Understand & describe the elements of grid systems and models• Develop architectural framework for standards in service-oriented

grids• Similar to profiling

− Use existing standards and technology where possible• Validate current standards for applicability in grids

− Use case-driven gap analysis – gaps filled proactively• Provide direction/motivation for new standards activity• Leverage & collaborate with other standards organizations


The OGSA Process

• Use case-driven− 21 detailed use cases

• “Design team” working model− Facilitate cross-fertilization within and outside OGF− Focus mind-share (the most valuable commodity!)

• e.g. DAIS and OGSA-Data working groups

− Design teams: Execution Management, Resource Management, Security

• OGSA fellow WGs− Formal independent OGF WG with “OGSA-” prefix− Work on some particular service interface

• Design teams may move up to OGSA fellow WGs

− Tightly collaborate and synchronize with OGSA-WG

• Iterative refinement− Abstract service evolving to concrete specifications


OGSA Document Structure

OGSARoadmap

OGSAArchitecture

OGSAUse Cases

Scenario

ServiceDescription

Candidate Profile

Profile

Actualspecs

consistent

inform andguide

inform andguide

feedback

refer

Proposedrecommendation

Informational

OGSA-WG documents

Documents producedby other OGF WGsor other SDOs

Base document

Informationmodels

Modelingguidelines

ProfileDefinition

Guidelines


OGSA and Fellow WGs

2002 2003 2004 2005 2006

OGSAdebut

WSRF TC

WSRF

WSRF 1.0

Arch 1.0

WSRF BP 1.0

OGSA-Naming WG

OGSA-BES WG

OGSA-ByteIO WG

OGSA-RSS WG

OGSA-WG

OGSA-AuthZ WG

Arch 1.5

OGSA-Data WG

OGSA-DMI WG

OGSA-HPCP WG

OGSI 1.0OGSI-WG

Roadmap1.0

Use Cases Roadmap1.1


Published OGSA Documents

20062005

Base document

Service description

Profile

Specification

2004

Architecture V1.0 (GFD-I.30) Glossary V1.0 (GFD-I.44) Resource Management

(GFD-I.45)

Architecture V1.0 (GFD-I.30) Glossary V1.0 (GFD-I.44) Resource Management

(GFD-I.45)

Use Cases (GFD-I.29) Use Cases (GFD-I.29)

Roadmap (GFD-I.53) Roadmap (GFD-I.53)

JSDL (GFD-R-P.56)

Profile Definition (GFD-I.59) Guideline


OGSA Schedule

2005 2006

EMS arch scenarios

HPC Profile

ByteIOBasic Execution Service

Architecture V1.5Architecture V1.5Glossary V1.5Glossary V1.5

Roadmap V1.1Roadmap V1.1Base document

Service description

Profile

Specification

Guideline Modeling guideline

WS-Naming

Public comment start GFD publication

Data architecture

WSRF Basic Profile

Basic Security Profiles

Container info-model


The OGSA Roadmap

• Defines OGSA as:− An architectural process− A set of specifications & profiles− OGSA software

• For each OGSA-related document:− Schedule− Dependencies

• Publication:− v1.0 published September ’05 (GFD.53)− Next version v1.1 in progress and plan to

publish Jan. 2007


OGSA-Based Open-Source Grids

Numerous grid projects are implementingOGSA components


Summary

• An open, service-oriented architecture


• Virtualize resources to hide complexity

• An open, service-oriented architecture


• Virtualize resources to hide complexity

• Interoperability is key!

• OGSA is defining a set of core services− Designed to work together…− …but with implementation flexibility

• Interoperability is key!

• OGSA is defining a set of core services− Designed to work together…− …but with implementation flexibility

• The current set of services is not the end of the road – just the beginning!

• The current set of services is not the end of the road – just the beginning!


More materials

• Architecture and Glossary v1.5− https://forge.gridforum.org/sf/go/doc13553?nav=1 − https://forge.gridforum.org/sf/go/doc13555?nav=1

• OGSA-WG project page− http://forge.ggf.org/projects/ogsa-wg

• OGSA-WG mailing list− Send email to "[email protected]" with the body text

"subscribe [email protected]".


Acknowledgement

With special thanks to Jem Treadwell, Andrew Grimshaw,

Jay Unger, Andreas Savva, David Snelling, Tom Maguire,

Dave Berry, Chris Jordan, Frank Siebenlist, Fred Maciel,

Michael Behrens, Allen Luniewski, Andrea Westerinen,

and other OGSA-WG members


OGF Full Copyright Notice

Copyright (C) Open Grid Forum (2005, 2006). All Rights Reserved.

This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the OGF or other organizations, except as needed for the purpose of developing Grid Recommendations in which case the procedures for copyrights defined in the OGF Document process must be followed, or as required to translate it into languages other than English.

The limited permissions granted above are perpetual and will not be revoked by the OGF or its successors or assigns.

This document and the information contained herein is provided on an "AS IS" basis and THE OPEN GRID FORUM DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE."

promoting and standardizing grid computing defining the grid: open grid services architecture...

Documents

open grid forum

open grid forum slide

remove services

software services

soa web services

grid projects

developed services competition

ogsa slide