Performance AnalysisNecessity or Add-on in Grid Computing
Michael Gerndt
Technische Universität München
LRR at Technische Universität München
•Chair for Computer Hardware & Organisation / Parallel Computer Architecture (Prof. A. Bode)
•Three groups in parallel & distributed architectures
• Architectures– SCI Smile project– DAB– Hotswap
• Tools– CrossGrid– APART
• Applications– CFD– Medicine– Bioinformatics
Performance Analysis for Parallel Systems
•Development cycle• Assumption: Reproducibility
•Instrumentation• Static vs Dynamic• Source-level vs object-level
•Monitoring• Software vs Hardware• Statistical profiles vs Event
traces
•Analysis• Source-based tools• Visualization tools• Automatic analysis tools
Coding
Performance Monitoringand Analysis
Production
Program Tuning
Grid Computing
•Grids• enable communities (“virtual organizations”) to share
geographically distributed resources as they pursue common goals -- assuming the absence of…
– central location,– central control, – omniscience, – existing trust relationships.
[Globus Tutorial]
•Major differences to parallel systems• Dynamic system of resources• Large number of diverse systems• Sharing of resources• Transparent resource allocation
Scenarios for Performance Monitoring and Analysis
• Post-mortem application analysis• Self-tuning applications• Grid scheduling• Grid management
[GGF performance working group, DataGrid, CrossGrid]
Post-Mortem Application Analysis
• Requires• either resources with known performance
characteristics (QoS)• or system-level information to assess performance data• scalability of performance tools
• Focus will be on interacting components
1. George submits job to the Grid2. Job is executed on some resources3. George receives performance data4. George analyzes performance
Self-Tuning Applications
• Requires• Integration of system and application monitoring• On-the-fly performance analysis• API for accessing monitor data (if PA by application)• Performance model and interface to steer adaptation
(If PA and tuning decision by external component.)
1. Chris submits job2. Application adapts to assigned
resources3. Application starts4. Application monitors performance and
adapts to resource changes
Grid-Scheduling
• Requires• PA of the grid application• Possibly benchmarking the application• Access to current performance capabilities of
resources• Even better to predicted capabilities
1. Gloria determines performance critical application properties
2. She specifies a performance model3. Grid scheduler selects resources4. Application is started
Grid-Management
• Requires• PA of historical system information• Need to be done in a distributed fashion
1. George claims to see bad performance since one week.
2. The helpdesk runs the Grid performance analysis software.
3. Periodical saturation of connections is detected.
New Aspect of Performance Analysis
•Transparent resource allocation•Dynamism in resource availability
•Approaches in the following projects:• Damien• Datagrid• Crossgrid• GrADS
Analyzing Meta-Computing Applications
•DAMIEN (IST-25406), 5 partnerswww.hlrs.de/organization/pds/projects/damien/
•Goals• Analysis of GRID-enabled applications
– using MpCCI (www.mpcci.org)– using PACX-MPI
(www.hlrs.de/organization/pds/projects/pacx-mpi)
• Analysis of GRID components– PACX-MPI and MpCCI
• Extend Vampir/Vampirtrace technology
MetaVampirtrace for Application Analysis
GRID-MPI profiling routine (PPACX_Send)
Native MPI GRID communication layer
Compiled code (PACX_Send)
Routine call
Tracefile
MetaVT wrapper (PACX_Send)
Routine call
Name shift (CPP)
Application code (MPI_Send)
MetaVampirtrace for GRID Component Analysis
Name shift (CPP)
Application code (MPI_Send)
TracefileMetaVT wrapper (MPI_Send)
MPI profiling routine (PMPI_Send)
Compiled code (PACX_Send)
Routine call
GRID-MPI layer (PACX_Send)
Routine call
TCP/IP
GRID-MPI communication layer
MetaVampir
•General counter support • Grid component metrics
•Hierarchical analysis • Analysis at each level• Aggregate data for groups• Improves scalability
•Structured tracefiles• Subdivided into frames• Stripe data across multiple
files
Metacomputer
Node 2Node 1
SMP node 1
P_1
GRID–DaemonsMPI processes
Send RecvSMP node 2
P_n
All MPI Processes
P_1 P_n
Grid Monitoring Architecture
•Developed by GGF Performance working group•Separation of data discovery and data transfer
• Data discovery via (possibly distributed) directory service
• Data transfer among producer – consumer
•GMA interactions• Publish/subscribe• Query/response• Notification
•Directory includes• Types of events• Accepted protocols• Security mechanisms
Consumer
Producer
Directory
Service
eventpublicationinformation
eventpublicationinformation
R-GMA in DataGrid
•DataGrid www.eu-datagrid.org•R-GMA www.cs.nwu.edu/~rgis•DataGrid WP3 hepunx.rl.ac.uk/edg/wp3
•Relational approach to GMA• Producers announce: SQL “CREATE TABLE”
publish: SQL “INSERT”• Consumers collect: SQL “SELECT”• Approach to use the relational model in a distributed
environment• It can be used for information service as well as
system and application monitoring.
P-Grade and R-GMA
•P-GRADE Environment developed at MTA SZTAKI
• GRM (Distributed monitor)• Prove (Visualization tool)
•GRM creates two tables in R-GMA• GRMTrace (String appName, String event): all events• GRMHeader (String appName, String event): important
header events only
•GRM Main Monitor• SELECT “*” FROM GRMHeader WHERE appName=“...”• SELECT “*” FROM GRMTrace WHERE appName=“...”
Main Monitor
Site
User’s
Host
Host 1 Host 2
ApplicationProcess
Appl.Process
Appl.Process
R-GMA
PROVE
Connection to R-GMA
Analyzing Interactive Applications in CrossGrid
•CrossGrid funded by EU: 03/2002 – 02/2005www.eu-crossgrid.org
•Simulation of vascular blood flow• Interactive visualization and simulation
– response times are critical– 0.1 sec (head movement) to 5 min (change in simulation)
• Performance analysis– response time and its breakdown– performance data for specific interactions
CrossGrid Application Monitoring Architecture
•OCM-G = Grid-enabled OMIS-Compliant Monitor•OMIS = On-line Monitoring Interface Specification
•Application-oriented• Information about running applications
•On-line• Information collected at runtime• Immediately delivered to consumers
•Information collected via instrumentation• Activated / deactivated on demand• Information of interest defined at runtime (lower
overhead)
OMIS
Performance Tool
Service Manager
LM
P1 P2
LM
P4 P5
LM
P3
th_stop(Sim)
th_stop(P1,P2) th_stop(P4,P5)th_stop(P3)
StopStop StopStopStop
Application Specific Measurement
•G-PM offers standard metrics• CPU time, communication time, disk I/O, ...
•Application programmer provides • Relevant events inside application (probes)• Relevant data computed by the application• Association between events in different processes
•G-PM allows to define new metrics• Based on existing ones and application specific
information• Metric Definition Language under development• Compilation or interpretation will be done by High-Level
Analysis Component.
Managing Dynamism: The GrADS Approach
•GrADS (Grid Application Development Software)• Funded by National Science Foundation, started 2000
•Goal:Provide application development technologies that make it easy to construct and execute applications with reliable [and often high] performance in the constantly-changing environment of the Grid.
•Major techniques to handle transparency and dynamism:
• Dynamic configuration to available resources (configurable object programs)
• Performance contracts and dynamic reconfiguration
GrADS Software Architecture
PSE
Config.object
program
wholeprogramcompiler
Source appli-cation
libraries
Realtimeperf
monitor
Dynamicoptimizer
Grid runtime System
(Globus)
negotiation
Software Components
Scheduler/Service
Negotiator
Performance feedback
Program Preparation System Execution Environment
Configurable Object Programs
•Integrated mapping strategy and cost model
•Performance enhanced by context-depend. variants
•Context includes potential execution platforms
•Dynamic Optimizer performs final binding
• Implements mapping strategy
• Chooses machine-specific variants
• Inserts sensors and actuators
• Perform final compilation and optimization
Performance Contracts
A performance contract specifies the measurable performance of a grid application.
Given• set of resources,• capabilities of resources,• problem parameters
the application will• achieve a specified, measurable performance
Creation of Performance Contracts
Program
PerformanceModel
Resource Broker
ResourceAssignment
PerformanceContract
• Developer• Compiler• Measurements
MDS
NWS
History-Based Contracts
•Resources given by broker•Capabilities of resources given by
• Measurements of this code on those resources• Possibly scaled by the Network Weather Service• e.g. Flops/second and Bytes/second
•Problem parameters• Given by the input data set
•Application intrinsic parameters • Independent of execution platform• Measurements of this code with same problem parameters• e.g. floating point operation count, message count,
message bytes count
•Measurable Performance Prediction• Combining application parameters and resource
capabilities
Application and System Space Signature
Application Signature• trajectory of values through
N-dimensional metric space• one trajectory per process• e.g. one point per iteration• e.g. metric: iterations/flop
M1
M2
M3
M1
M2
M3
System Signature• trajectory of values through
N-dimensional metric space• will vary across application
executions, even on the sameresources
• e.g. metric iterations/second
resource capabilities
Verification of Performance Contracts
Execution
ContractMonitor
Rescheduling
Sensor Data
SteerDynamic Optimizer
• Violation detection• Fault detection
APART
•ESPRIT IV Working Group, 01/1999 – 12/2000
•IST Working Group, 08/2001 – 07/2004
www.fz-juelich.de/apart
Focus:
• Network European development projects for
automatic performance analysis tools
– Testsuite for automatic analysis tools
• Automatic Performance Analysis and Grid Computing
(WP3 – Peter Kacsuk)
Summary
•Scenarios• Post-mortem Application Tuning• Self-tuning applications• Grid scheduling• Grid management
•How to handle transparency and dynamism?
•Approaches here:• Damien: Provide static environment.• Datagrid: Combining system and application
monitoring• Crossgrid: On-line analysis• GrADS: Performance models and contracts