the globus grid programming toolkit: a user-level tutorial the globus project team anl and usc/isi
Post on 11-Jan-2016
243 Views
Preview:
TRANSCRIPT
The Globus Grid Programming Toolkit:A User-level Tutorial
The Globus Project Team
ANL and USC/ISI
http://www.globus.org
Introduction 2
AbstractThis tutorial is a practical introduction to programming for high-performance distributed computing systems, or "computational grids," and the capabilities of the Globus grid toolkit.
Emerging high-performance networks promise to enable a wide range of emerging application concepts such as remote computing, distributed supercomputing, tele-immersion, smart instruments, and data mining. However, the development and use of such applications is in practice very difficult and time consuming, because of the need to deal with complex and highly heterogeneous systems. The Globus grid programming toolkit is designed to help application developers and tool builders overcome these obstacles to the construction of "grid-enabled” scientific and engineering applications. It does this by providing a set of standard services for authentication, resource location, resource allocation, configuration, communication, file access, fault detection, and executable management. These services can be incorporated into applications and/or programming tools in a "mix-and-match" fashion to provide access to needed capabilities.
Our goal in this tutorial is both to introduce the capabilities of the Globus toolkit and to show attendees how Globus services can be applied in specific applications. Hence, the tutorial covers a mixture of grid programming principles and detailed case studies of real applications.
Introduction 3
Tutorial Goals
Provide an introduction To the structure of the Globus computational
grid To the capabilities of the Globus toolkit To pragmatic issues associated with using
the toolkit Enable attendees
To start building & using Globus applications To utilize Globus services
Introduction 4
Overview Introduction to computational grids High-level overview of the Globus toolkit Four components:
Security and remote process creation Running programs across multiple resources Information services Dynamic configuration and resource management
Case studies Other Globus services, and future directions Globus installation & administration
Introduction 5
Why “The Grid”?
New applications based on high-speed coupling of people, computers, databases, instruments, etc. Computer-enhanced instruments Collaborative engineering Browsing of remote datasets Use of remote software Data-intensive computing Very large-scale simulation Large-scale parameter studies
Introduction 6
Los Angeles Chicago
E.g.: Computer-Enhanced Instruments for Microtomography
“100 Gflop/sec, 50 Mb/sec, 5 minutes; rendering, 10 GB storage”
APSbeamline
@ Argonne
Coupling with supercomputers
Interactive use of beamline
Collaboration on results
Parameter studies for experiment planning
Coupling with mass store systems
50 Mb/s -> 5 Gb/s -> 100 Gb/s
5 Mb/s -> 1 Gb/s -> 10 Gb/s
Introduction 7
Control Text Video Audio DatabaseLeigh et al., UofI, Electronic Visualization Lab.
E.g.: Tele-immersion
“5 Gflop/sec, flowspecs, design db”
Multiple access modalitiesMultiple flows
Simulation Tracking Haptics Rendering
Introduction 8
SF-Express: Distributed Interactive Simulation
P. Messina et al., Caltech
Issues: Resource discovery, scheduling Configuration Multiple comm methods Message passing (MPI) Scalability Fault tolerance
NCSAOrigin
CaltechExemplar
ArgonneSP
MauiSP
“200 GB memory, 100 BIPs”
Introduction 9
The Grid
“Dependable, consistent, pervasive access to
[high-end] resources” Dependable: Can provide
performance and functionality guarantees
Consistent: Uniform interfaces to a wide variety of resources
Pervasive: Ability to “plug in” from anywhere
Introduction 10
Evolution of a Concept
Metacomputing: late 80s Focus on distributed computation
Gigabit testbeds: early 90s Research, primarily on networking
I-WAY: 1995 Demonstration of application feasibility
PACIs (National Technology Grid): 1998 NASA Information Power Grid: 1999 ASCI DISCOM: 1999; SSI: 2000?
Introduction 11
National and International Grid Testbeds
I-WAY
NASA’s Information Power Grid
The Alliance National Technology Grid
Introduction 12
Technical Challenges
Complex application structures, combining aspects of parallel, multimedia, distributed, collaborative computing
Dynamic varying resource characteristics, in time and space
Need for high & guaranteed “end-to-end” performance, despite heterogeneity and lack of global control
Interdomain issues of security, policy, payment
Introduction 13
Domain 2
Domain 1
•Authenticate once
•Specify simulation
(code, resources, etc.)
•Locate resources
•Negotiate authorization,
acceptable use, etc.
•Acquire resources
•Initiate computation
•Steer computation
•Access remote datasets
•Collaborate on results
•Account for usage
Issues
Introduction 14
Architectural Approaches
Distributed systems: DCE, CORBA, Jini, etc. Rich functionality eases app. development Complexity hinders deployment
especially in absence of global control
Performance difficulties Internet Protocol, Web tools
Simple protocols facilitate deployment Missing functionality hinders app. development Performance difficulties
Introduction 15
Standards & Commodity Tech
Where appropriate, exploit standards and commodity technology in core infrastructure LDAP, SSL, X.509, GSS-API, GAA-API, http, ftp,
XML, etc. Provides leverage
Interface with other common standards CORBA, Java/Jini, DCOM, Web, etc While our core infrastructure may not be built on
one of these distributed architectures, we must cleanly interface with them
Introduction 16
The Globus Project
Basic research in grid-related technologies Resource management, QoS, networking,
storage, security, adaptation, policy, etc. Development of Globus toolkit
Core services for grid-enabled tools & applns Construction of large grid testbed: GUSTO
Largest grid testbed in terms of sites & apps Application experiments
Tele-immersion, distributed computing, etc.
Introduction 17
Globus Approach
A toolkit and collection of services addressing key technical problems Bag of services model Not a vertically integrated solution
Inter-domain issues, rather than clustering Integration of intra-domain solutions
Distinguish between local and global services “IP hourglass” model
Introduction 18
Technical Focus & Approach
Information-rich environment Basis for configuration and adaptation
Enable incremental development of grid-enabled tools and applications Support many programming models, tools,
applications Deploy toolkit on national-scale testbed to
allow large-scale applications Evolve in response to user requirements
Introduction 19
Globus Approach
Focus on architecture issues Propose set of core services
as basic infrastructure Use to construct high-level,
domain-specific solutions Design principles
Keep participation cost low Enable local control Support for adaptation
Diverse global services
Core Globusservices
Local OS
A p p l i c a t i o n s
Introduction 20
Layered Architecture
Applications
Core ServicesMetacomputing
Directory Service
GRAMGlobus
Security Interface
Heartbeat Monitor
Nexus
Gloperf
Local Services
LSF
Condor MPI
NQEEasy
TCP
SolarisIrixAIX
UDP
High-level Services and Tools
DUROC globusrunMPI Nimrod/GMPI-IO CC++
GlobusView Testbed Status
GASS
Introduction 21
Core Globus Services
Communication infrastructure (Nexus, IO) Information services (MDS) Network performance monitoring (Gloperf) Process monitoring (HBM) Remote file and executable management
(GASS and GEM) Resource management (GRAM) Security (GSI)
Introduction 22
Sample of High-Level Services I
Communication & I/O libraries MPICH, PAWS, RIO (MPI-IO), PPFS, MOL
Parallel languages CC++, HPC++
Collaborative environments CAVERNsoft, ManyWorlds
Others MetaNEOS, NetSolve, LSA, AutoPilot, WebFlow
Introduction 23
Sample High-Level Services II
Resource brokers and co-allocators DUROC: co-allocation of multiple systems Nimrod: high-throughput computing
Graphical system status display elements GlobusView MDS Browsers Health & Status Monitors (HBM) Network Monitors (Gloperf)
Introduction 24
“GUSTO”Globus Ubiquitous
Supercomputing Testbed Organization
A collection of organizations committed to creating a persistent computational grid infrastructure
As of November 1998, 70 organizations in 3 continents and 8 countries
Introduction 25
16 sites, 330 computers, 3600 nodes, 2 Teraflop/s, 10 application partners
Introduction 26
GUSTO Testbed During SC’97
Introduction 27
GUSTO Computational Grid Testbed: November 1998
Introduction 28
Where We Are (November 1998)
New results in security, resource management, tools, fault detection, etc.
Globus v1.0 completed All core services complete, relatively robust,
and documented Available on most Unix platforms Many tool projects are leveraging this
considerable investment in infrastructure Interesting applications are emerging, although
mostly still in “demo” mode
Introduction 29
Where We Are (June 1999)
New results in QoS, security, resource management, data management, tools, etc.
Globus v1.1 nearing completion Available on most Unix platforms and Win32 Many tool projects are leveraging this
considerable investment in infrastructure Documentation and deployment underway at
NCSA and NASA IPG Always looking for interesting applications
Introduction 30
Changes from 1.0 to 1.1
Tutorial changes for 1.1 are denoted by Name changes from Globus to Grid
Security and Information Service adopted as core Grid infrastructure by several organizations
Globus Security Infrastructure -> Grid Security Infrastructure Metacomputing Directory Service -> Grid Information Service
Affects naming of APIs and tools Numerous small API fixes, additions, changes Cleanup of programs/tools A few new modules (I/O, error objects)
Introduction 31
Example Application Projects
Computed microtomography (ANL, ISI) Real-time, collaborative analysis of data from
X-Ray source (and electron microscope) Hydrology (ISI, UMD, UT; also NCSA, Wisc.)
Interactive modeling and data analysis Collaborative engineering (“tele-immersion”)
CAVERNsoft @ EVL, Metro @ ANL X-Ray crystallography (ANL, SUNY)
High-throughput computing for Shake ‘n Bake
Introduction 32
Example Application Expts (contd)
Distributed interactive simulation (CIT, ISI) Record-setting SF-Express simulation
Remote visualization and steering for astrophysics Including trans-Atlantic experiments
Data-intensive computing experiments (with LBNL and SLAC: “Clipper” project)
Introduction 33
For More Information on Globus
http://www.globus.org Papers on all components Tutorial and documents Software Application descriptions
Introduction 34
The Grid:Blueprint for a New Computing Infrastructure
I. Foster, C. Kesselman (Eds), Morgan Kaufmann, 1999
Available July 1998;
ISBN 1-55860-475-8 22 chapters by expert
authors including Andrew Chien, Jack Dongarra, Tom DeFanti, Andrew Grimshaw, Roch Guerin, Ken Kennedy, Paul Messina, Cliff Neuman, Jon Postel, Larry Smarr, Rick Stevens, and many others
http://www.mkp.com/grids
“A source book for the historyof the future” -- Vint Cerf
Introduction 35
Tutorial Approach
Four sections, each illustrates a basic Globus technique
Laboratory material is available to allow practice with the use of each technique See http://www.globus.org/tutorial
top related