architectural models for resource management in the grid
DESCRIPTION
$grid. Architectural Models for Resource Management in the Grid. Rajkumar Buyya Monash University, Australia Steve Chapin, Syracuse University, USA David DiNucci Elepar.com, USA. Outline. Changes in Computing Landscape Resource Management Issues Architectural Models - PowerPoint PPT PresentationTRANSCRIPT
1
Architectural Models for Resource Management in the Grid
Rajkumar BuyyaMonash University, Australia
Steve Chapin, Syracuse University, USA
David DiNucciElepar.com, USA
2
Outline
Changes in Computing Landscape Resource Management Issues Architectural Models Hierarchical Resource Management Abstract Owner Market Model Economy Grid Nimrod/G Resource Broker Conclusions
3
Computing Power (HPC) Drivers
Solving grand challenge applications using computer modeling, simulation and analysis
Life SciencesLife Sciences
CAD/CAMCAD/CAM
AerospaceAerospace
Military ApplicationsDigital BiologyDigital Biology Military ApplicationsMilitary Applications
Internet & Ecommerce
4
2100
2100 2100 2100 2100
2100 2100 2100 2100
Desktop SMPs or SuperComputers
LocalCluster
GlobalCluster/Grid
PERFORMANCE
Inter PlanetCluster/Grid ??
•Individual•Group•Department•Campus•State•National•Globe•Inter Planet•Universe
Breaking Administrative Barriers
EnterpriseCluster/Grid
?
Computing Platforms
5
Towards Grid Computing
Unification of geographically distributed resources
6
What is Grid ?
An infrastructure that couples: Computers – PCs, workstations, clusters,
supercomputers, laptops, notebooks, mobile devices, PDA, etc;
Software – e.g., ASPs renting expensive special purpose applications on demand;
Catalogued data and databases – e.g. transparent access to human genome database;
Special devices – e.g., radio telescope – SETI@Home searching for life in galaxy, Austrophysics@Swinburne for pulsars)
People/collaborators.
& offers dependable, consistent, & pervasive access to resources.
7
A Example Grid Infrastructure
8
Sources of Complexity in Grid Resource Management
No single administrative control. No single ownership policy:
Each resource owner has their own policies or scheduling mechanisms;
Users must honour them (particularly external Grid users).
Heterogeneity of resources. Dynamic availability – may appear
and disappear…
9
Sources of Complexity in Grid Resource Management
Unreliable resource – disappear from view!
No uniform cost model - varies from one user’s resource to another and from time of day.
No single access mechanism – Web, custom interfaces, command line…
10
Grid Resource Management Issues
Ack: Globus..
•Authentication (once).
•Specify (code, resources,
etc.).
•Discover resources.
•Negotiate authorization,
acceptable use, Cost, etc.
•Acquire resources.
•Schedule Jobs.
•Initiate computation.
•Steer computation.
•Access remote data-sets.
•Collaborate with results.
•Account for usage.
•Discover resources.
•Negotiate authorisation,
acceptable use, Cost, etc.
•Acquire resources.
•Schedule jobs.
•Initiate computation.
•Steer computation.
Domain 2
Domain 1
11
Architectural Models
MODEL REMARKS SystemsHierarchical It captures model
followed in most contemporary systems.
Globus, Legion, CCS, Apples, NetSolve, Ninf.
Abstract Owner (AO)
Order and delivery model and focuses on long term goals.
Expected to emerge and most peer-2-peer computing systems likely to be based on this.
Market Model It follows economic model for resource discover, sharing, & scheduling.
GRACE, Nimrod/G, JavaMarket, Mariposa.
12
Hierarchical Resource Management
Connection Cloud
UserGlobal
Scheduler
Control Domain
Monitor
Resource
Grid Information Service
Access/Admission Control Agent
Domain Resource Manager or Control Agent
GlobalScheduler
LocalScheduler
- Task
Persistent Job ControlAgent
GlobalSchedule
r
GlobalSchedule
r
GlobalSchedule
r
Deployment Agent
13
Who owns the GRID?
Talk topeople
Powerappliances
Use GRIDresources
Phone co. Electric co.Abstract
Owner (AO)
I want to:
My interface is:
I arrange serviceand payments with a:(may be many choices)
But resources I “get”may belong to others:
GeneratorsPower linesTransformers
AntennaeCable/fiberSwitches
HPCNetworksInstrumentsPeople
14
Order Pickup
AO is owner or broker User negotiates
with AO through “order window”
That AO may own some resources, and/or it may broker with other AOs for those resources
After negotiation, resources are delivered through “pickup window”
OrderWindow
PickupWindow
PhysicalResource
UserRequests Resources
AO
Order Pickup
ResourceManager
AO1
Manager
DeliverySales
AO2
AO3
15
AO Resources
Resources are objects Classes are
Instrument» Data source, sink,
transform» e.g. programs, people,
files, data collection devices
Channel» Moves data among
instruments Complexes of above
Attributes define sizes, times, connections, etc.
Instrument(File)
Instrument(Program)
Instrument(File)
Instrument(Program)
Channels
Instrument(Telescope)
Instrument(Person)
16
Negotiating with an AO
Make dummy resource(with attributes set to
constants, variables, or “don’t care”)
+ bid + delivery plan+ variable constraints
Resource candidates(values for variables/attributes
+ asking price for each)
Pick one,Try again,Or give up
DeliveryWindow
Resource
Order Window
Assign tasksto resource,use, relinquish
Perhapslater...
17
Many Testbeds ? & who pays ?
GUSTO
Legion TestbedNASA IPG
EcoGrid
18
Testbeds so far -- observations
Who contributed to resources & why ? Volunteers: for fun, challenge, fame, public good
like SETI@Home & distributed.net projects. Collaborators: sharing resources while
developing new technologies of common interest – Globus, Legion, Ecogrid.
How long ? Short duration: GUSTO decommissioned.
What do we need ? Grid Marketplace! Regulates demand and supply, offers incentive
for being players, simple, scalable solution, quasi-deterministic – proven model in real-world.
19
Users in Grid Economy & Strategy Grid Consumers
Execute jobs for solving varying problem size and complexity
Benefit by selecting and aggregating resources wisely Tradeoff timeframe and cost
» Strategy: minimise expenses
Grid Providers Contribute “idle” resource for executing consumer
jobs Benefit by maximizing resource utilisation Tradeoff local requirements & market opportunity
» Strategy: maximise returns on services
20
Building of a Economy Grid “brokerage” system…..
Foundation for the Grid Economy
21
Grid Node N
Grid Architecture for Computational Economy
Grid User
Application
Grid Resource Broker
Grid Service Providers
Grid Explorer
Schedule Advisor
Trade Manager
Job ControlAgent
Deployment Agent
Trade Server
Resource Allocation
ResourceReservation
R1
Misc. services
Information Server(s)
R2 Rm…
Pricing Algorithms
Accounting
Grid Node1
…
Grid Middleware Services
…
…
HealthMonitor
Grid Market Services
JobExec
Info ?
Secure
Trading
QoS
Storage
Sign-on
22
Economic Models for Trading Commodity Market Model Posted Prices Models Bargaining Model Tendering (Contract Net) Model Auction Model
English, first-price sealed-bid, second-price sealded-bid (Vickrey), and Dutch.
Proportional Resource Sharing Model
Shareholder Model Partnership Model
Economy Grid = Globus + GRACE
Applications
MDS
GRAMGlobus Security Interface
Heartbeat MonitorNexus
Local Services
LSF
Condor GRD QBank
PBS
TCP
SolarisIrixLinux
UDP
High-level Services and Tools
DUROC globusrunMPI-G Nimrod/GMPI-IO CC++
GlobusView Grid Status
GASS
GRACE-TS
GARA
GridFabric
GridApps.
GridMiddleware
GridTools
GBankGMD
eCash
JVM
DUROC
Core Services
Science
Engineering Commerce Portals ActiveSheet……
24
A resource broker for managing and steering task farming (parametric sweep) applications on computational Grids based on deadline and computational economy.
Key Features A single window to manage & control experiment Resource Discovery Trade for Resources Scheduling Steering & data management
It allows to study the behaviour of some of the output variables against a range of different input scenarios.
What is Nimrod/G ?
25
Nimrod/G Grid Broker Architecture
Grid Middleware
Nimrod/G Client Nimrod/G ClientNimrod/G Client
Grid Information Server(s)
Schedule Advisor
Trading Manager
Nimrod/G Engine
GridStore
Grid Explorer
GE GISTM TS
RM & TS
Grid Dispatcher
RM: Local Resource Manager, TS: Trade Server
Globus,Legion, Condor-g,, Ninf,etc.
G
G
CL
Globus enabled node.Legion enabled node.
CL
Condor enabled node.
RM & TSRM & TS
26
A Nimrod/G Client
A Nimrod/G Client
CostCostDeadlineDeadline
Legion hosts
Globus Hosts
Bezek is in both Globus and Legion Domains
Arlington
Alexandria
Richmond
HamptonNorfolk
Virginia BeachChesapeakePortsmouth
Newport News
Roanoke
Ap p om a toxRive r
Ja m esRive r
Shena nd oa hRive r
Ra p p a ha nnoc kRive r
Potom a cRive r
VIRGINIA77
81
64
64
66
85
27
User Requirements: Deadline/Budget User Requirements: Deadline/Budget
28
Globus+Legion+Condor/G
Australia
Monash Uni.:
Linux cluster
Solaris WS
Nimrod/G
Globus +GRACE_TS
Europe
ZIB/FUB: T3E/Mosix Cardiff: Sun E6500Paderborn: HPCLineLecce: Compaq SCCNR: ClusterCERN: Cluster
Globus +GRACE_TS
Asia/Japan
Tokyo I-Tech.:ETL, Tuskuba
Linux cluster
Globus/LegionGRACE_TS
North America
ANL: SGI/Sun/SP2USC-ISI: SGIUVa: Linux ClusterManitoba: Cluster
Internet
Global Economy Grid
29
Graph 5 - GUSTO Usage for 10 Hour Deadline
0
5
10
15
20
25
30
35
0 2.5 5 7.5 10 12.5 15 17.5 20
Time
No
Pro
cess
es
5 CUs
10 CUs
15 CUs
20 CUs
50 CUs
10 Cost Units
50 Cost Units
20 Cost Units
5 Cost Units
15 Cost Units
Graph 4 - GUSTO Usage for 15 Hour Deadline
0
2
4
6
8
10
12
14
16
18
20
0 2.5 5 7.5 10 12.5 15 17.5 20
Time
Av
era
ge
No
Pro
ce
ss
ors
5 CUs
10 CUs
15 CUs
20 CUs
50 CUs
5 Cos t Units
50 Cos t Units
15 Cos t Units
10 Cos t Units
Graph 3 - GUSTO Usage for 20 Hour Deadline
0
2
4
6
8
10
12
14
16
18
20
0 2.5 5 7.5 10 12.5 15 17.5 20
Time
Ave
rag
e N
o P
roce
sso
rs 5 CUs
10 CUs
15 CUs
20 CUs
50 CUs
5 Cost Units
10 Cost Units
Graph 2 - GUSTO Usage for Ionization Chamber Study
0
10
20
30
40
50
60
70
80
0 2.5 5 7.5 10 12.5 15 17.5 20
Time
Ave
rag
eN
o. P
roce
sso
rs20 Hour deadline15 hour deadline10 hour deadline
30
Conclusions
Proposed three models for Grid resource management architecture Hierarchical, AO, & Market-model
The future systems are likely follow a model that combines all these models.
The future computing (HPC) infrastructure is going to be a Grid of Clusters.
Peer-to-Peer/Grid has already become a darling of venture capitalists.
The impact of Grid on 21st century economy will be the same as electricity on 20th century economy.