iceni overview & grid scheduling
DESCRIPTION
ICENI Overview & Grid Scheduling. Laurie Young London e-Science Centre Department of Computing, Imperial College. ICENI. IC e -Science N etworked I nfrastructure Developed by LeSC Grid Middleware Group Collect and provide relevant Grid meta-data - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/1.jpg)
ICENI Overview&
Grid Scheduling
Laurie Young
London e-Science CentreDepartment of Computing, Imperial College
![Page 2: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/2.jpg)
2
ICENI
IC e-Science Networked InfrastructureDeveloped by LeSC Grid Middleware GroupCollect and provide relevant Grid meta-data Use to define and develop higher-level services Interaction with other frameworks: OGSA, Jxta etc.
The Iceni, under Queen Boudicca, united the tribes of South-East England in
a revolt against the occupying Roman forces
in AD60.
![Page 3: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/3.jpg)
3
ICENI (The Big Picture)
Private
Resource Manager
Policy Manager
CR
SR
Identity Manager
Domain Manager
CR
SR
Gateway between private and public regions
Public
Resource Browser
Public Computational Community
SR CR
Public Computational Community
SR
Private
Administrative
Domain
SR
CR
Resource Broker
Application Design Tools
Component Design Tools
Application Mapper
Web ServicesGateway
Application
Portal
Computational Resource
SoftwareResources
NetworkResources
StorageResources
JavaCoGGlobus
![Page 4: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/4.jpg)
4
ICENI Stack
Portal Interface
Application Construction & Deployment
ICENI Middleware
Grid Fabric
![Page 5: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/5.jpg)
5
Web Portals
• Handheld wireless devices become ubiquitous– Personal Digital Assistants, Mobile Phones– Secure access any time, any place, any where
• Use X.509 certificates embedded in a browser to authenticate user’s identity
• Integration portal infrastructure with ICENI– EPIC: Use component meta-data to
build portal application• Goal: Provide secure ‘one stop shop’ for e-
science
![Page 6: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/6.jpg)
6
EPIC: e-Science Portal at Imperial College
• Collaborative LeSC industrial project with Sun Microsystems
• Develop a secure portal infrastructure to:– Access your own personal environment– Applications to support day-to-day e-science– Interaction with other Grid infrastructures
• Allow role based access to resources– Anonymous: public web pages– Students: internal pages, email, compute resources– Staff: restricted pages
![Page 7: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/7.jpg)
7
ICENI Application Model
• Legacy code!• Component Applications
– Compose applications from many components– Component does work on data– Component communicates data
![Page 8: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/8.jpg)
8
Component Motivation
• Logical application model• Collaborative software authoring • Promote component reuse and sharing• Simplify application construction• Enable deployment to diverse Grid resources:
– Communication Selection– Implementation Selection
![Page 9: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/9.jpg)
9
Layered Abstraction
Meaning
BehaviourBehaviour
dataflowabstract data types
Implementation
control flowthreads etc.
performance,architectures,concrete data type
may have many
may each have many
Implementation
![Page 10: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/10.jpg)
10
Component’s View of the Grid
OtherCode
SOAP
More Code
RMI
My Code
You must implementa provided interface
You may call methods
provided by the middleware
Context object
![Page 11: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/11.jpg)
11
Visual Component Composition
![Page 12: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/12.jpg)
12Grid
Container
Deployment of Components
ComponentDesignTools
Scientist
ApplicationDesignTools
EndUser
Application DescriptionDocument
Developer
ImplementationAnnotating
Tools
Code CodeCode
Code
Run-TimeRepresentation
ApplicationMapper
RTR
Code
Access ResourceInformation
APO
Application Proxy Object
Repository
![Page 13: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/13.jpg)
13
SOAPRMI
Component Execution
Compute Resource Hardware
RTR
CodeCode Code
RTR RTR
NetworkResource
MPI
APO
Jini Jini
OGSA, Jxta, etc. OGSA, Jxta, etc.
![Page 14: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/14.jpg)
14
Components as Services
Component
Service interface
SOAP (or other)protocol
Context object
![Page 15: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/15.jpg)
15
ICENI & Jini: P2P
Service requester
Service LocatorService Provider
LookupService
ServiceMatches
Two- wayInteractionwith service
Register service
Discover services
Conceptual Model of peer- to- peer architecture
![Page 16: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/16.jpg)
16
Web Services Architecture
Service requester(SOAP Client)
Service Locator(UDDI Registry)
Service Provider(SOAP Server)
UDDILookup
UDDIResponse, WSDL location
SOAPmessageexchange
UDDI registration
Web Services Crawler?
Web Service Model
![Page 17: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/17.jpg)
17
Synergy
The Synergy
Service requester
Service Locator Web Service Proxy
Service Provider Web Service Proxy
Service RequesterWeb Service Client
P2P(J ini)
Web Services
J ini Lookup J ini Object
J ava App
RMI RMI
SOAP SOAP
SOAP
SOAPProxy
![Page 18: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/18.jpg)
18
Grid Service Contracts
JiniLookupService
DRMAAResource
DRMAAClient
![Page 19: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/19.jpg)
19
Grid Service Contracts
JiniLookupService
DRMAAResource
User: A+BDuration:1hr
DRMAAClient
User:B
ResourceBrowser
![Page 20: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/20.jpg)
20
Grid Service Contracts
JiniLookupService
DRMAAResource
User: A+BDuration:1hr
DRMAAResource
User:ADuration:10m
DRMAAClient
User:B
DRMAAClient
User:A
![Page 21: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/21.jpg)
21
OGSA & Jini Integration
JiniLookupService
GatewayManager
GSI enabledWeb Service
HostingEnvironment
DRMAAResource
User: A+BDuration:1hr
DRMAAResource
User:ADuration:10m
![Page 22: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/22.jpg)
22
OGSA & Jini Integration
JiniLookupService
GatewayManager
GSI enabledWeb Service
HostingEnvironment
Jini ClientInterface
WSDLInterface
DRMAAResource
User: A+BDuration:1hr
DRMAAResource
User:ADuration:10m
![Page 23: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/23.jpg)
23
OGSA & Jini Integration
JiniLookupService
GatewayManager
GSI enabledWeb Service
HostingEnvironment
GSI + SOAPConnection Jini Client
Interface
WSDLInterface
DRMAAResource
User: A+BDuration:1hr
DRMAAResource
User:ADuration:10m
![Page 24: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/24.jpg)
24
OGSA & Jini Integration
JiniLookupService
GatewayManager
GSI enabledWeb Service
HostingEnvironment
GSI + SOAPConnection
User Info
SOAP->Java
Jini ClientInterface
WSDLInterface
DRMAAResource
User: A+BDuration:1hr
DRMAAResource
User:ADuration:10m
![Page 25: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/25.jpg)
25
Application Mapping (Scheduling)
• Architecture– How meta-data is collected– What meta-data is required
• Scheduling Algorithms– Map components onto resources for “best” results– Meta-data dependent decisions
![Page 26: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/26.jpg)
26
Scheduling Architecture
Resources
ICENI
App Builder (GUI) Component Repository Performance Models
Scheduler Launcher
![Page 27: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/27.jpg)
27
Multiple Metrics (1)
• “It is the goal of a scheduler to optimise one or more metrics” (Feitelson & Rudolph)
• Generally one metric is most important– Application Optimisation
• Execution time• Execution cost
– Host Optimisation• Host utilisation• Host throughput• Interaction Latency
![Page 28: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/28.jpg)
28
• In a Grid Environment there are three application optimisation based important metrics– Start time ( )– End time ( )– Cost ( )
• Relative importance varies on a user by user and application by application basis
Multiple Metrics (2)
be
![Page 29: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/29.jpg)
29
• A Benefit Function maps the metrics we are interested in to a single Benefit Value metric
• Different benefit functions represent different optimisation preferences
Combining Metrics – Benefit Fn
),,( ebBB
![Page 30: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/30.jpg)
30
Optimisation Preferences
• Cost Optimisation
• Time Optimisation
• Cost/Time Optimisation
max max e and if
eB
max max e and if
eB
max max e and if
eB
![Page 31: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/31.jpg)
31
Schedule Benefit
• Each component and communication has a benefit function
• Each resource and network connection has a predicted time & cost for each component or communication that could be deployed
• Fit the tasks onto the resources to get the maximum Total Predicted Benefit
),,( ebt BB
![Page 32: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/32.jpg)
32
Graph Oriented Scheduling (1)
• Applications are described as a graph– Nodes represent application components– Edges represent component communication
• Resources are described as a graph– Nodes represent resources– Edges represent network connections
![Page 33: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/33.jpg)
33
Graph Oriented Scheduling (2)
Condor pool
Atlas Saturn
Viking
Design Analyse
Scatter
Gather
Mesh
DRACS
Mesh
DRACS
Mesh
DRACS
Factory
![Page 34: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/34.jpg)
34
Graph Oriented Scheduling (3)
Condor pool
ScatterGather
DesignAtlas
Factory
AnalyseSaturn
Viking
![Page 35: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/35.jpg)
35
Summary
• Component framework provides:– Rich application meta-data– Decoupled component definition and implementation
• Meta-Data:– Exploit performance information to map component implementation to
the ‘best’ resources• Resource Broker:
– Resource selection through user defined policies:• Minimise cost using computational economics• Minimise execution time using the application mapper
![Page 36: ICENI Overview & Grid Scheduling](https://reader035.vdocuments.net/reader035/viewer/2022062422/5681356d550346895d9cd2b8/html5/thumbnails/36.jpg)
36
Acknowledgements
• Director: Professor John Darlington• Technical Director: Dr Steven Newhouse• Research Staff:
– Anthony Mayer, Nathalie Furmento– Stephen McGough, James Stanton– Yong Xie, William Lee– Marko Krznaric, Murtaza Gulamali– Asif Saleem, Laurie Young, Gary Kong
• Contact:– http://www.lesc.ic.ac.uk/– e-mail: [email protected]