grid computing environments

25
Grid Computing Environments Grid: a system supporting the coordinated resource sharing and problem- solving in dynamic, multi-institutional virtual organizations. Note: some of this material was taken from the Globus tutorial at www.globus.org

Upload: mieko

Post on 12-Feb-2016

47 views

Category:

Documents


0 download

DESCRIPTION

Grid Computing Environments. Grid: a system supporting the coordinated resource sharing and problem-solving in dynamic, multi-institutional virtual organizations. Note: some of this material was taken from the Globus tutorial at www.globus.org. A Scenario. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Grid Computing Environments

Grid Computing Environments

Grid: a system supporting the coordinated resource sharing and problem-solving in dynamic, multi-institutional virtual organizations.

Note: some of this materialwas taken from the Globus tutorial at www.globus.org

Page 2: Grid Computing Environments

A ScenarioA fundamental change in the problems needed to be solved:

• multidisciplinary nature composition of expertise• larger problem scale composition of computing resources

Mary is a university researcher in North America who is collaborating withtwo colleagues - Hans at a corporate R&D facility in Europe and Ling at a researchinstitute in Asia. Mary has a new research model that she would like tovalidate using data collected by Ling and compare the results to that of theproprietary model developed by Hans for his corporation. Each of the three collaborators have previously constructed software components that provide access to the resources they are contributing to the collaboration. Mary and Hans are using a shared workspace system to interact synchronously in performing the computations, viewing the results, and saving the results for later study. Ling is not available at the time of the interaction and needs to provide appropriate access so that her data is available to Mary and Hans. The results, stored by Mary, should also be available later forstudy by Ling.

Page 3: Grid Computing Environments

Grid Computing Environments

Virtual Organization: a set of individuals and/or institutions collaborating to achieve a common goal within a set of rules defining the controlled sharing of computational resources.

Characteristics of virtual organizations:

•flexible relationships (client-server, P2P, bockered,…)•complex sharing rules (access control, delegation,…)•varied resources (programs, storage, devices, …)•diverse usage modes (single vs. multi-user, performance vs. cost sensitive, synchronous vs. asynchronous,…)

Page 4: Grid Computing Environments

On-demand creation of powerful virtual organizations

Grid vs.Web

http://

http://

Web: Uniform naming/access to documents

Grid: Uniform, high-performance access to computational resources

Sensor nets

Data archives

Computers

Softwarecatalogs

Colleagues

Page 5: Grid Computing Environments

Grid vs. Web

Characteristics Grid Web

Seamless naming Yes YesUniform security/authentication

Yes No

Information services Yes Yes/noCo-scheduling Yes NoAccounting/authorization Yes NoUser services Yes NoEvent services Yes NoGlobal shell yes no

Page 6: Grid Computing Environments

Layered Grid Architecture

Application

Fabric“Controlling things locally”: Access to, & control of, resources

Connectivity“Talking to things”: communication (Internet protocols) & security

Resource“Sharing single resources”: negotiating access, controlling use

Collective“Coordinating multiple resources”: ubiquitous infrastructure services, app-specific distributed services

InternetTransport

Application

Link

Internet Protocol Architecture

Page 7: Grid Computing Environments

“Hourglass” principle

Application

Fabric

Connectivity

Resource

Collective

Local OS

Core Gridservices

Page 8: Grid Computing Environments

Protocols, Services,and Intefaces

Languages/Frameworks

Fabric Layer

Applications

Local Access APIs and Protocols

Collective Service APIs and SDKsCollective Services

Collective Service Protocols

Resource APIs and SDKsResource Services

Resource Service Protocols

Connectivity APIsConnectivity Protocols

Page 9: Grid Computing Environments

Protocols, Services, Interfaces:

• Protocol-mediated access to resources– Mask local heterogeneities– Extensible to allow for advanced features– Negotiate multi-domain security, policy– “Grid-enabled” resources speak protocols– Multiple implementations are possible

• Broad deployment of protocols facilitates creation of Services that provide integrated view of distributed resources

• Intefaces (APIs/SKDs) use protocols and services to enable specific classes of applications

Page 10: Grid Computing Environments

Globus

•NSF’s National Technology Grid•NASA’s Information Power Grid

•Internet protocols•Globus Security Infrastructure (GSI)

•Grid Resource Information Service (GRIS)•Grid Resource Access and Management (GRAM)•GridFTP

•Grid Information Index Servers (GIIS)•Replica management•Certificate repository (MyProxy)•Co-allocation library (DUROC)

Fabric

Connectivity

Resource

Collective

Page 11: Grid Computing Environments

Fabric Layer

• A diverse mix of resources that may be shared– Individual computers, Condor pools, file systems,

archives, metadata catalogs, networks, sensors, etc., etc.• Few constraints on low-level technology:

connectivity and resource level protocols form the “neck in the hourglass”

• Defined by interfaces not physical characteristics

Page 12: Grid Computing Environments

Connectivity Layer

• Communication– Internet protocols: IP, DNS, routing, etc.

• Security: Grid Security Infrastructure (GSI)– Uniform authentication & authorization

mechanisms in multi-institutional setting– Single sign-on, delegation, identity mapping– Public key technology, SSL, X.509, GSS-API– Supporting infrastructure: Certificate Authorities,

key management, etc.

Page 13: Grid Computing Environments

Why Grid Security is Hard

• Resources being used may be extremely valuable & the problems being solved extremely sensitive

• Resources are often located in distinct administrative domains– Each resource may have own policies & procedures

• Set of resources used by a single computation may be large, dynamic, and unpredictable– Not just client/server

• It must be broadly available & applicable– Standard, well-tested, well-understood protocols– Integration with wide variety of tools

Page 14: Grid Computing Environments

1) Easy to use2) Single sign-on3) Run applications

ftp,ssh,MPI,Condor,Web,…4) User based trust model5) Proxies/agents (delegation)

User View1) Specify local access control2) Auditing, accounting, etc. 3) Integration w/ local system

Kerberos, AFS, license mgr.4) Protection from compromised

resources

Resource Owner View

API/SDK with authentication, flexible message protection, flexible communication, delegation, ...

Direct calls to various security functions (e.g. GSS-API)Or security integrated into higher-level SDKs:

E.g. GlobusIO, Condor-G, MPICH-G2, HDF5, etc.

Developer View

Grid Security Requirements

Page 15: Grid Computing Environments

Secure Remote Startup

key

cert

gatekeeperclient

1. Exchange certificates, authenticate, delegate

2. Check gridmap file3. Lookup service4. Run service program

(e.g. jobmanager)

jobmanager

key

cert

1.

2.

map

4.

services3.

Page 16: Grid Computing Environments

Resource Layer

• Grid Resource Allocation Mgmt (GRAM) – Remote allocation, reservation, monitoring, control of

compute resources• GridFTP protocol (FTP extensions)

– High-performance data access & transport• Grid Resource Information Service (GRIS)

– Access to structure & state information• Network reservation, monitoring, control• All integrated with GSI: authentication,

authorization, policy, delegation

Page 17: Grid Computing Environments

Metacomputing Directory Services• Resources run a standard information service (GRIS) which speaks LDAP and

provides information about the resource (no searching).• GIIS provides a “caching” service much like a web search engine. Resources

register with GIIS and GIIS pulls information from them when requested by a client and the cache as expired.

• GIIS provides the collective-level indexing/searching function.

GIISCache contains info from A and B

Resource AGRIS

GIIS requests information fromGRIS services as needed.

Client 1

Client 2

Client 3

Resource BGRIS

Clients 1 and 2 request infodirectly from resources.

Client 3 uses GIIS for searchingcollective information.

Page 18: Grid Computing Environments

Resource Management

• The Globus Resource Allocation Manager (GRAM) protocol and client API allows programs to be started on remote resources, despite local heterogeneity

• Resource Specification Language (RSL) is used to communicate requirements

• A layered architecture allows application-specific resource brokers and co-allocators to be defined in terms of GRAM services– Integrated with Condor, PBS, MPICH-G2, …

Page 19: Grid Computing Environments

GRAM Components

Grid SecurityInfrastructure

Job Manager

GRAM client API calls to request resource allocation

and process creation.

MDS client API callsto locate resources

Query current statusof resource

Create

RSL Library

Parse

Request Allocate &create processes

Process

Process

Process

Monitor &control

Site boundary

Client MDS: Grid Index Info Server

Gatekeeper

MDS: Grid Resource Info Server

Local Resource Manager

MDS client API callsto get resource info

GRAM client API statechange callbacks

Page 20: Grid Computing Environments

GridFTP

• Suite of communication libraries and related tools that support– GSI, Kerberos security– Third-party transfers– Parameter set/negotiate– Partial file access– Reliability/restart– Large file support– Data channel reuse

• All based on a standard, widely deployed protocol

– Integrated instrumentation– Loggin/audit trail– Parallel transfers– Striping (cf DPSS)– Policy-based access control– Server-side computation– Proxies (firewall, load bal)

Page 21: Grid Computing Environments

GridFTP

• Why FTP?– Ubiquity enables interoperation with many commodity

tools– Already supports many desired features, easily extended

to support others– Well understood and supported

• We use the term GridFTP to refer to– Transfer protocol which meets requirements– Family of tools which implement the protocol

• Note GridFTP > FTP

Page 22: Grid Computing Environments

GridFTP: Basic Approach

• FTP protocol is defined by several IETF RFCs• Start with most commonly used subset

– Standard FTP: get/put etc., 3rd-party transfer• Implement standard but often unused features

– GSS binding, extended directory listing, simple restart• Extend in various ways, while preserving

interoperability with existing servers– Striped/parallel data channels, partial file, automatic &

manual TCP buffer setting, progress monitoring, extended restart

Page 23: Grid Computing Environments

Collective Layer

• Index servers aka metadirectory services– Custom views on dynamic resource collections

assembled by a community • Resource brokers (e.g., Condor Matchmaker)

– Resource discovery and allocation• Replica catalogs• Co-reservation and co-allocation services

Page 24: Grid Computing Environments

Replica Management

• Maintain a mapping between logical names for files and collections and one or more physical locations

• Important for many applications– Example: CERN HLT data

• Multiple petabytes of data per year• Copy of everything at CERN (Tier 0)• Subsets at national centers (Tier 1)• Smaller regional centers (Tier 2)• Individual researchers will have copies

Page 25: Grid Computing Environments

DUROC

• Simultaneous allocation of a resource set– Handled via optimistic co-allocation based on free

nodes or queue prediction– In the future, advance reservations will also be

supported• globusrun will co-allocate specific multi-requests

– Uses a Globus component called the Dynamically Updated Request OnlineCo-allocator (DUROC)