the globus grid programming toolkit: a user-level tutorial the globus project team anl and usc/isi
TRANSCRIPT
![Page 1: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/1.jpg)
The Globus Grid Programming Toolkit:A User-level Tutorial
The Globus Project Team
ANL and USC/ISI
http://www.globus.org
![Page 2: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/2.jpg)
Introduction 2
AbstractThis tutorial is a practical introduction to programming for high-performance distributed computing systems, or "computational grids," and the capabilities of the Globus grid toolkit.
Emerging high-performance networks promise to enable a wide range of emerging application concepts such as remote computing, distributed supercomputing, tele-immersion, smart instruments, and data mining. However, the development and use of such applications is in practice very difficult and time consuming, because of the need to deal with complex and highly heterogeneous systems. The Globus grid programming toolkit is designed to help application developers and tool builders overcome these obstacles to the construction of "grid-enabled” scientific and engineering applications. It does this by providing a set of standard services for authentication, resource location, resource allocation, configuration, communication, file access, fault detection, and executable management. These services can be incorporated into applications and/or programming tools in a "mix-and-match" fashion to provide access to needed capabilities.
Our goal in this tutorial is both to introduce the capabilities of the Globus toolkit and to show attendees how Globus services can be applied in specific applications. Hence, the tutorial covers a mixture of grid programming principles and detailed case studies of real applications.
![Page 3: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/3.jpg)
Introduction 3
Tutorial Goals
Provide an introduction To the structure of the Globus computational
grid To the capabilities of the Globus toolkit To pragmatic issues associated with using
the toolkit Enable attendees
To start building & using Globus applications To utilize Globus services
![Page 4: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/4.jpg)
Introduction 4
Overview Introduction to computational grids High-level overview of the Globus toolkit Four components:
Security and remote process creation Running programs across multiple resources Information services Dynamic configuration and resource management
Case studies Other Globus services, and future directions Globus installation & administration
![Page 5: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/5.jpg)
Introduction 5
Why “The Grid”?
New applications based on high-speed coupling of people, computers, databases, instruments, etc. Computer-enhanced instruments Collaborative engineering Browsing of remote datasets Use of remote software Data-intensive computing Very large-scale simulation Large-scale parameter studies
![Page 6: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/6.jpg)
Introduction 6
Los Angeles Chicago
E.g.: Computer-Enhanced Instruments for Microtomography
“100 Gflop/sec, 50 Mb/sec, 5 minutes; rendering, 10 GB storage”
APSbeamline
@ Argonne
Coupling with supercomputers
Interactive use of beamline
Collaboration on results
Parameter studies for experiment planning
Coupling with mass store systems
50 Mb/s -> 5 Gb/s -> 100 Gb/s
5 Mb/s -> 1 Gb/s -> 10 Gb/s
![Page 7: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/7.jpg)
Introduction 7
Control Text Video Audio DatabaseLeigh et al., UofI, Electronic Visualization Lab.
E.g.: Tele-immersion
“5 Gflop/sec, flowspecs, design db”
Multiple access modalitiesMultiple flows
Simulation Tracking Haptics Rendering
![Page 8: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/8.jpg)
Introduction 8
SF-Express: Distributed Interactive Simulation
P. Messina et al., Caltech
Issues: Resource discovery, scheduling Configuration Multiple comm methods Message passing (MPI) Scalability Fault tolerance
NCSAOrigin
CaltechExemplar
ArgonneSP
MauiSP
“200 GB memory, 100 BIPs”
![Page 9: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/9.jpg)
Introduction 9
The Grid
“Dependable, consistent, pervasive access to
[high-end] resources” Dependable: Can provide
performance and functionality guarantees
Consistent: Uniform interfaces to a wide variety of resources
Pervasive: Ability to “plug in” from anywhere
![Page 10: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/10.jpg)
Introduction 10
Evolution of a Concept
Metacomputing: late 80s Focus on distributed computation
Gigabit testbeds: early 90s Research, primarily on networking
I-WAY: 1995 Demonstration of application feasibility
PACIs (National Technology Grid): 1998 NASA Information Power Grid: 1999 ASCI DISCOM: 1999; SSI: 2000?
![Page 11: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/11.jpg)
Introduction 11
National and International Grid Testbeds
I-WAY
NASA’s Information Power Grid
The Alliance National Technology Grid
![Page 12: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/12.jpg)
Introduction 12
Technical Challenges
Complex application structures, combining aspects of parallel, multimedia, distributed, collaborative computing
Dynamic varying resource characteristics, in time and space
Need for high & guaranteed “end-to-end” performance, despite heterogeneity and lack of global control
Interdomain issues of security, policy, payment
![Page 13: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/13.jpg)
Introduction 13
Domain 2
Domain 1
•Authenticate once
•Specify simulation
(code, resources, etc.)
•Locate resources
•Negotiate authorization,
acceptable use, etc.
•Acquire resources
•Initiate computation
•Steer computation
•Access remote datasets
•Collaborate on results
•Account for usage
Issues
![Page 14: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/14.jpg)
Introduction 14
Architectural Approaches
Distributed systems: DCE, CORBA, Jini, etc. Rich functionality eases app. development Complexity hinders deployment
especially in absence of global control
Performance difficulties Internet Protocol, Web tools
Simple protocols facilitate deployment Missing functionality hinders app. development Performance difficulties
![Page 15: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/15.jpg)
Introduction 15
Standards & Commodity Tech
Where appropriate, exploit standards and commodity technology in core infrastructure LDAP, SSL, X.509, GSS-API, GAA-API, http, ftp,
XML, etc. Provides leverage
Interface with other common standards CORBA, Java/Jini, DCOM, Web, etc While our core infrastructure may not be built on
one of these distributed architectures, we must cleanly interface with them
![Page 16: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/16.jpg)
Introduction 16
The Globus Project
Basic research in grid-related technologies Resource management, QoS, networking,
storage, security, adaptation, policy, etc. Development of Globus toolkit
Core services for grid-enabled tools & applns Construction of large grid testbed: GUSTO
Largest grid testbed in terms of sites & apps Application experiments
Tele-immersion, distributed computing, etc.
![Page 17: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/17.jpg)
Introduction 17
Globus Approach
A toolkit and collection of services addressing key technical problems Bag of services model Not a vertically integrated solution
Inter-domain issues, rather than clustering Integration of intra-domain solutions
Distinguish between local and global services “IP hourglass” model
![Page 18: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/18.jpg)
Introduction 18
Technical Focus & Approach
Information-rich environment Basis for configuration and adaptation
Enable incremental development of grid-enabled tools and applications Support many programming models, tools,
applications Deploy toolkit on national-scale testbed to
allow large-scale applications Evolve in response to user requirements
![Page 19: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/19.jpg)
Introduction 19
Globus Approach
Focus on architecture issues Propose set of core services
as basic infrastructure Use to construct high-level,
domain-specific solutions Design principles
Keep participation cost low Enable local control Support for adaptation
Diverse global services
Core Globusservices
Local OS
A p p l i c a t i o n s
![Page 20: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/20.jpg)
Introduction 20
Layered Architecture
Applications
Core ServicesMetacomputing
Directory Service
GRAMGlobus
Security Interface
Heartbeat Monitor
Nexus
Gloperf
Local Services
LSF
Condor MPI
NQEEasy
TCP
SolarisIrixAIX
UDP
High-level Services and Tools
DUROC globusrunMPI Nimrod/GMPI-IO CC++
GlobusView Testbed Status
GASS
![Page 21: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/21.jpg)
Introduction 21
Core Globus Services
Communication infrastructure (Nexus, IO) Information services (MDS) Network performance monitoring (Gloperf) Process monitoring (HBM) Remote file and executable management
(GASS and GEM) Resource management (GRAM) Security (GSI)
![Page 22: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/22.jpg)
Introduction 22
Sample of High-Level Services I
Communication & I/O libraries MPICH, PAWS, RIO (MPI-IO), PPFS, MOL
Parallel languages CC++, HPC++
Collaborative environments CAVERNsoft, ManyWorlds
Others MetaNEOS, NetSolve, LSA, AutoPilot, WebFlow
![Page 23: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/23.jpg)
Introduction 23
Sample High-Level Services II
Resource brokers and co-allocators DUROC: co-allocation of multiple systems Nimrod: high-throughput computing
Graphical system status display elements GlobusView MDS Browsers Health & Status Monitors (HBM) Network Monitors (Gloperf)
![Page 24: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/24.jpg)
Introduction 24
“GUSTO”Globus Ubiquitous
Supercomputing Testbed Organization
A collection of organizations committed to creating a persistent computational grid infrastructure
As of November 1998, 70 organizations in 3 continents and 8 countries
![Page 25: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/25.jpg)
Introduction 25
16 sites, 330 computers, 3600 nodes, 2 Teraflop/s, 10 application partners
![Page 26: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/26.jpg)
Introduction 26
GUSTO Testbed During SC’97
![Page 27: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/27.jpg)
Introduction 27
GUSTO Computational Grid Testbed: November 1998
![Page 28: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/28.jpg)
Introduction 28
Where We Are (November 1998)
New results in security, resource management, tools, fault detection, etc.
Globus v1.0 completed All core services complete, relatively robust,
and documented Available on most Unix platforms Many tool projects are leveraging this
considerable investment in infrastructure Interesting applications are emerging, although
mostly still in “demo” mode
![Page 29: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/29.jpg)
Introduction 29
Where We Are (June 1999)
New results in QoS, security, resource management, data management, tools, etc.
Globus v1.1 nearing completion Available on most Unix platforms and Win32 Many tool projects are leveraging this
considerable investment in infrastructure Documentation and deployment underway at
NCSA and NASA IPG Always looking for interesting applications
![Page 30: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/30.jpg)
Introduction 30
Changes from 1.0 to 1.1
Tutorial changes for 1.1 are denoted by Name changes from Globus to Grid
Security and Information Service adopted as core Grid infrastructure by several organizations
Globus Security Infrastructure -> Grid Security Infrastructure Metacomputing Directory Service -> Grid Information Service
Affects naming of APIs and tools Numerous small API fixes, additions, changes Cleanup of programs/tools A few new modules (I/O, error objects)
![Page 31: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/31.jpg)
Introduction 31
Example Application Projects
Computed microtomography (ANL, ISI) Real-time, collaborative analysis of data from
X-Ray source (and electron microscope) Hydrology (ISI, UMD, UT; also NCSA, Wisc.)
Interactive modeling and data analysis Collaborative engineering (“tele-immersion”)
CAVERNsoft @ EVL, Metro @ ANL X-Ray crystallography (ANL, SUNY)
High-throughput computing for Shake ‘n Bake
![Page 32: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/32.jpg)
Introduction 32
Example Application Expts (contd)
Distributed interactive simulation (CIT, ISI) Record-setting SF-Express simulation
Remote visualization and steering for astrophysics Including trans-Atlantic experiments
Data-intensive computing experiments (with LBNL and SLAC: “Clipper” project)
![Page 33: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/33.jpg)
Introduction 33
For More Information on Globus
http://www.globus.org Papers on all components Tutorial and documents Software Application descriptions
![Page 34: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/34.jpg)
Introduction 34
The Grid:Blueprint for a New Computing Infrastructure
I. Foster, C. Kesselman (Eds), Morgan Kaufmann, 1999
Available July 1998;
ISBN 1-55860-475-8 22 chapters by expert
authors including Andrew Chien, Jack Dongarra, Tom DeFanti, Andrew Grimshaw, Roch Guerin, Ken Kennedy, Paul Messina, Cliff Neuman, Jon Postel, Larry Smarr, Rick Stevens, and many others
http://www.mkp.com/grids
“A source book for the historyof the future” -- Vint Cerf
![Page 35: The Globus Grid Programming Toolkit: A User-level Tutorial The Globus Project Team ANL and USC/ISI](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649e4c5503460f94b41ab8/html5/thumbnails/35.jpg)
Introduction 35
Tutorial Approach
Four sections, each illustrates a basic Globus technique
Laboratory material is available to allow practice with the use of each technique See http://www.globus.org/tutorial