national center for supercomputing applicationsnational computational science grid packaging...
Post on 20-Jan-2016
215 Views
Preview:
TRANSCRIPT
National Center for Supercomputing Applications National Computational Science
National Computational Science
Grid Packaging TechnologyTechnical Talk
University of Wisconsin Condor/GPTMeeting
January 3,4 2002
Michael Bletzingermbletzin@ncsa.uiuc.edu
Specialist, Software DeveloperAdvanced Computational Environment
and Security, NCSA
National Center for Supercomputing Applications National Computational Science
National Computational Science
GPT Purposes
Give developers an easy way to package software All packaging data is specified from a source
perspective. Packaging data is strongly typed.
Provides a means of building and installing collections of packages
Provides standard version negotiation method for managing updates to packages.
National Center for Supercomputing Applications National Computational Science
National Computational Science
GPT and Globus
GPT was designed to re-implement the deployment requirements of Globus plus shared libraries. Globus most of Globus is libraries. Globus has deployment requirements for client tools,
service daemons, and development/build environments. Globus wants to release individual components
separately Globus wants to add outside software (ie. openssl,
openldap) to these deployments. Globus wants to work on multiple platforms.
National Center for Supercomputing Applications National Computational Science
National Computational Science
Outline
Splitting up the source code Binary Package Types Dependencies Build Flavors Versioning Installations Bundling Setup The GPT Tools
National Center for Supercomputing Applications National Computational Science
National Computational Science
Splitting up the Source Code
Source Package Types Source packages: contain source code and
documentation. Setup packages: contain configuration/localization
scripts and files. Test Packages: contain component test code.
National Center for Supercomputing Applications National Computational Science
National Computational Science
Source Packages
Contains source code for programs, scripts, and libraries. All libraries are added to the link line.
Contains documentation and data files. All of these share the same version number.
National Center for Supercomputing Applications National Computational Science
National Computational Science
Other Packages
Setup Packages Contains scripts and templates. These are assumed to have no build flavor. Generate files outside of the control of the packaging
system.
Test Packages Contains test harnesses and code. Installed in a special directory.
National Center for Supercomputing Applications National Computational Science
National Computational Science
Outline
Splitting up the source code Binary Package Types Dependencies Build Flavors Versioning Installations Bundling Setup The GPT Tools
National Center for Supercomputing Applications National Computational Science
National Computational Science
Binary Package Types
Type Description Flavored
programs and scripts. Yes No
Yes Yes
headers and static libraries. Yes Yes
documentation. No No
data non-localized data files. No No
Multi-flavored installation
pgm
rtlshared libraries and flavored scripts.
dev
doc
National Center for Supercomputing Applications National Computational Science
National Computational Science
Package Transformations
TestPackage
SetupPackage
SourcePackage
pgm
data
doc pgm
data
devrtl
doc pgm
data
doc
Source Code
Installed Files localized files
National Center for Supercomputing Applications National Computational Science
National Computational Science
Outline
Splitting up the source code Binary Package Types Dependencies Build Flavors Versioning Installations Bundling Setup The GPT Tools
National Center for Supercomputing Applications National Computational Science
National Computational Science
Dependencies
A Dependency is a need relationship to another package RPM's "Requires:" does the same thing.
Different Dependency Types for Source an Binary Packages.
Types include both build and runtime dependencies.
National Center for Supercomputing Applications National Computational Science
National Computational Science
Runtime Dependencies
Source Runtime dependencies are transferred to the specific binary package.
Runtime dependencies require a binary package type in the specification .
National Center for Supercomputing Applications National Computational Science
National Computational Science
Build Dependencies
Are transferred to the binary dev packages Are used in the configure stage of the source
package build.
National Center for Supercomputing Applications National Computational Science
National Computational Science
Source Runtime Dependencies
pgm_runtime Dependencies for the programs.
lib_runtime Dependencies for the libraries.
doc_runtime Dependencies for the documentation.
data_runtime Dependencies for the data files.
National Center for Supercomputing Applications National Computational Science
National Computational Science
Source Build Dependencies
compile Requires header file from another package
pgm_link Programs require libraries from another package
lib_link Libraries depend on libraries from another package
National Center for Supercomputing Applications National Computational Science
National Computational Science
Dependencies
HeaderFile
Library
Script
DoxygenPage
Library SourceCode
Documentation
Program SourceCode
Source Package
Package Foo
Package Fee
Package Foe
Package Fum
National Center for Supercomputing Applications National Computational Science
National Computational Science
Binary Dependencies
Compile Only in dev packages
Build_Link Only in dev packages
Runtime_Link rtl and pgm packages
Regeneration Only in pgm packages
Runtime in any binary package
National Center for Supercomputing Applications National Computational Science
National Computational Science
Source to Binary Dep Conversion
compile
lib_link
pgm_link
runtime
compile
runtime
build_link
runtime_link
regeneration
Static Linking
Dynamic Linking
Pass Thru
SourceDependencies
BinaryDependencies
National Center for Supercomputing Applications National Computational Science
National Computational Science
Outline
Splitting up the source code Binary Package Types Dependencies Build Flavors Versioning Installations Bundling Setup The GPT Tools
National Center for Supercomputing Applications National Computational Science
National Computational Science
Build Flavors
A Build Flavor is a set of compilation and linking options which have to be defined to produce binaries that can be linked with each other.
A development environment issue for older versions of Globus.
Now a runtime issue with GT 2.0 because of shared libraries and plugins.
National Center for Supercomputing Applications National Computational Science
National Computational Science
Build Flavors
Build Flavor Choices Compiler Choice: gcc, vendor cc Size: 32 bit, 64 bit Debugging: Debug Symbols included or stripped Thread Package: pthreads, solaris, sproc, none message passing library: mpi, none
Standard Flavors in green Example
gcc32dbgpthr = gcc compiler, 32 bit, debugging symbols, pthreads
National Center for Supercomputing Applications National Computational Science
National Computational Science
Outline
Splitting up the source code Binary Package Types Dependencies Build Flavors Versioning Installations Bundling Setup The GPT Tools
National Center for Supercomputing Applications National Computational Science
National Computational Science
Versioning Purposes
Source code change identifier. Distribution identifier. Marketing tool. Interface compatibility identifier
for APIs. for protocols. for configuration files.
National Center for Supercomputing Applications National Computational Science
National Computational Science
Version Negotiation.
Agreement on compatibility between package developer and package user. Package developer makes a compatibility
recommendation. Package user choose to accept or override the
recommendation.
National Center for Supercomputing Applications National Computational Science
National Computational Science
Aging Version
Version scheme used by the package developer. Based on libtool and other shared library schemes.
4.1.3
major version numberminor version number - indicates a binary compatible change
age - indicatesbackward compatibility to themajor version number
National Center for Supercomputing Applications National Computational Science
National Computational Science
Version Requirements
Simple Version Requirement. Package user specifies major version number. Trusts the developer compatibility recommendation.
Version Range Requirement. Package user specifies a range of major and minor
version numbers. Does not use the compatibility recommendation from the
package developer.
Combinations of these can be used for more complex requirements.
National Center for Supercomputing Applications National Computational Science
National Computational Science
Versioning Example
Version for the package is 5.1.3
Version Requirement Is Compatible
3 Yes
1 No
4.2 to 5.4 Yes
4.2 to 5.0 No
National Center for Supercomputing Applications National Computational Science
National Computational Science
Outline
Splitting up the source code Binary Package Types Dependencies Build Flavors Versioning Installations Bundling Setup The GPT Tools
National Center for Supercomputing Applications National Computational Science
National Computational Science
Installation Objectives
Standard structure for both runtime and build installations.
Multiple build flavors can exist in the same location. Packaging data co-located with the installation.
National Center for Supercomputing Applications National Computational Science
National Computational Science
Installation Structure
$GLOBUS_LOCATION
programs
libraries
include
etc
setup
test
bin
liblibexec
gcc32headers
gcc32pthrheaderspackagingmetadata
sbin
National Center for Supercomputing Applications National Computational Science
National Computational Science
Multiple Installations
$GLOBUS_LOCATION
programs
libraries
include
etc
setup
test
bin
lib
libexec
gcc32headers
gcc32pthrheaders
packagingmetadata
sbin
Client Tools
$GLOBUS_LOCATION
programs
libraries
include
etc
setup
test
bin
lib
libexec
gcc32headers
gcc32pthrheaders
packagingmetadata
sbin
User Development
$GLOBUS_LOCATION
programs
libraries
include
etc
setup
test
bin
lib
libexec
gcc32headers
gcc32pthrheaders
packagingmetadata
sbin
Services
National Center for Supercomputing Applications National Computational Science
National Computational Science
How to move an installation.
Copy Directory. Re-run setup scripts.
National Center for Supercomputing Applications National Computational Science
National Computational Science
Outline
Splitting up the source code Binary Package Types Dependencies Build Flavors Versioning Installations Bundling Setup The GPT Tools
National Center for Supercomputing Applications National Computational Science
National Computational Science
Bundling Objective
Be able to install a collection of packages or even an entire distribution with one command. Packages are not installed if compatible ones already
exist. All setups are consolidated into one post install script.
Be able to build a binary bundle with one command.
National Center for Supercomputing Applications National Computational Science
National Computational Science
Bundling Data
Name and version of the package. Binary package types that are needed. Build options to be avoided ie. no threads. Build Flavors are chosen during builds for each
platform.
National Center for Supercomputing Applications National Computational Science
National Computational Science
Outline
Splitting up the source code Binary Package Types Dependencies Build Flavors Versioning Installations Bundling Setup The GPT Tools
National Center for Supercomputing Applications National Computational Science
National Computational Science
Setup Objectives
Allow installed packages to be configured with a script. Each script packaged in a setup package.
Script execution is recorded in the packaging data. Setup packages can be replaced by distributing
organizations to tailor a distribution.
National Center for Supercomputing Applications National Computational Science
National Computational Science
Setup dependencies
Source package indicate a setup need by a special setup dependency.
Setup packages have two names and versions. One name/version for the specific package.
different setup packages have different name/version.
One name/version to fulfill the setup dependency. All setup packages will have the same name/version for this.
National Center for Supercomputing Applications National Computational Science
National Computational Science
Setup Package Example
Regular PackageFoo
SetupPackageFum
SetupPackageFie
SetupPackageFoe
National Center for Supercomputing Applications National Computational Science
National Computational Science
Outline
Splitting up the source code Binary Package Types Dependencies Build Flavors Versioning Installations Bundling Setup The GPT Tools
National Center for Supercomputing Applications National Computational Science
National Computational Science
What does GPT offer?
An easy way for developers to package their software.
An easy way for people to build large sets of source packages.
Provides a package manager for those systems that don't have one.
Provides packaging metadata in an XML format. Compatible with other package managers
National Center for Supercomputing Applications National Computational Science
National Computational Science
An easy way for developers to package their software
gpt_setup - Sets up Makefiles and autoconf to use packaging system.
gpt_edit - GUI to allow developers to edit the metadata.
patch-n-build - for those developers that want to keep their own build system.
globus-makefile-header - for those who want to build applications using Globus components.
National Center for Supercomputing Applications National Computational Science
National Computational Science
An easy way for people to build large sets of source packages.
globus-build -swiss army knife for building packages, bundles, and development directories.globus-build [ \ -verbose \ -help \ -installdir=path_to_installation \ -builddir=path_to_build directory \ -log=logfile \ -srcfile=source_metadata_file \ -srcdir=source_directory \ -static=<build number> \ -install-only \ -force \ -std-flavors \ -all-flavors \ ] \[macro=value] [list of flavors to build] [list of source packages/bundles]
National Center for Supercomputing Applications National Computational Science
National Computational Science
Provides a package manager for those systems that don't have one.
globus-install - Installs/uninstalls binary packages gpt_verify - Verifies that a Globus installation is
complete.
National Center for Supercomputing Applications National Computational Science
National Computational Science
NMI Changes to GPT.
Changes for April NMI release. New schema/dtd for packaging data.
conversion script to update existing packages
Add package signatures and other installation verification mechanisms.
Add tools for installation management. Centralized file which records installations and installed bundles.
De-globus-ize the tools replace "globus" with something else for script names variables
etc. probably keep old globus names for compatibility purposes.
top related