grid on demand
DESCRIPTION
Slides of the Presentation (30 july 2010) of Grid on DemandTRANSCRIPT
Welcome
Grid on Demand
Willem Toorop and Alain van Hoof
{wtoorop,ahoof}@os3.nl
June 30, 2010
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 1 / 39
Introduction Research Question
Research Question
Can Grid computing be offered as a Cloud service and whattype of applications benefit from this?
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 2 / 39
Introduction The Grid and The Cloud
The Grid and The Cloud: Sharing Resources
Sharing Resources to minimizing idle resources
Resource Capacity GoodsProcessor Flops Compute
Hard-disk memory Bytes Storage
Network Bytes/sec Communication Bandwidth
Grid Cloud (IaaS)Distribution By Collaboration By Cloud Provider
Resource Type Heterogeneous Homogeneous
Abstraction Middleware (open standards) Virtualization
Sharing Guarantee Security infrastructure Financial infra
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 3 / 39
Introduction The Grid and The Cloud
The Grid and The Cloud: Dividing the Resources
Cloud
Grid
GridGrid
User
User
User
VOUser User
VOUser User
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 4 / 39
A Grid in the Cloud Urgent Computing
A Grid in The Cloud: Urgent Computing
GRIDon
Demand
Using the elasticity and scalability of the Cloud and providing a Gridinterface, current e-Science (computationally intensive science)applications can use a Grid on Demand when there is a high demand forresources for a short period of time.
Example
UrbanFlood (http://urbanflood.eu) wants to calculate flooding predictionswhen actual flooding happens, Grid on Demand can provide this.
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 5 / 39
A Grid in the Cloud Grid on Demand
Grid on Demand
Compute Nodes
Head Node
Job Submission Client
OS: Ubuntu 10.04Grid: Globus 4.2.1PBS: Torque 2.6.8
Grid Interface: Globus Toolkit 4.2 (open standard, widely used)Queue Manager (Distribution of jobs to nodes): Torque 2.6.2Amazon EC2 Cloud Provider (de-facto standard with open API)
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 6 / 39
Implementation What makes the Grid, the Grid
What makes the Grid, the GridTypical Grid usage example spanning multiple organizations
GridUser
Scheduler Storage
Compute
Compute
Organization A
Organization B
Organization D
Organization C
store data
use my datato compute X
use 1st 1
2of data
to compute12X
use 2nd 12 of data
to compute 12 X
accessdata
access data
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 7 / 39
Implementation What makes the Grid, the Grid
Proxy certificates for delegation
The Grid is a collaboration between multiple organizations.
Grid users need to be authorized to use remote resources
Those resources should be authorized to perform actionson behalf of the Grid user
Proxy certificates enable delegation of authority
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 8 / 39
Implementation What makes the Grid, the Grid
Grid user
create proxy Scheduler
create cert.request
Compute
compute X
sign this
Okay
compute X
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39
Implementation What makes the Grid, the Grid
Grid user
create proxy Scheduler
create cert.request
Compute
compute X
sign this
Okay
compute XVirtual Organizations are based ondelegation by proxy certificates.
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39
Implementation What makes the Grid, the Grid
Grid user
create proxy Scheduler
create cert.request
Compute
compute X
sign this
Okay
compute X
Proxy certificates allow fordynamic resource allocation,
But the hosts still need aCA signed certificate.
This is a problem when hostname’sare assigned dynamically!
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39
Implementation What makes the Grid, the Grid
Grid user
create proxy Scheduler
create cert.request
Compute
compute X
sign this
Okay
compute X
Proxy certificates allow fordynamic resource allocation,
But the hosts still need aCA signed certificate.
This is a problem when hostname’sare assigned dynamically!
Fixed CA in image?
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39
Implementation What makes the Grid, the Grid
Grid user
create proxy Scheduler
create cert.request
Compute
compute X
sign this
Okay
compute X
Proxy certificates allow fordynamic resource allocation,
But the hosts still need aCA signed certificate.
This is a problem when hostname’sare assigned dynamically!
Fixed CA in image?
Dynamic CA on every Launch?
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39
Implementation Amazon Elastic Block Store
Easy Image management with Amazon Elastic Block Store
(A,S) PendingRunning
(IA,VS ,H)
Stopping
Stopped(IA,VS)
Pending
Shutting-down
TerminatedIA
(A′,S ′)
launchinstance IA
VS ← S
stop
star
t
terminate
terminate
create AMI A′
S ′ ← VS
configure
Life cycle of Amazon Machine Images using Elastic Block StorageA = Amazon Machine Image, S = Snapshot,I = Instance, V = Volume, H = Hostname
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 10 / 39
Implementation EBS for configuration stages
EBS for configuration stages
(A, S)Running
(IA,VS ,H)
Stopped(IA,VS)
(A′, S ′)
Stopped(IA,VS)
(A′′, S ′′)
Running(IA,VS ,H)
Stopped(IA,VS)
(A′′′, S ′′′)
launchinstance IA
VS ← S
shu
tdow
n
forreu
se star
t
create AMI A′
S ′ ← VS
create AMI A′′
S ′′ ← VS
create AMI A′′′
S ′′′ ← VSshutdow
n
forsharing
shut
dow
nfo
rsh
arin
g shu
tdow
n
forreu
se star
t
provide Amazon Access Credentials
(configure cluster)
launch IA′
VS ← S ′launch IA′′′
VS ← S ′′′
configure configure
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 11 / 39
Web Interface Launching Grid on Demand
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 12 / 39
Web Interface Launching Grid on Demand
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 13 / 39
Web Interface Launching Grid on Demand
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 14 / 39
Web Interface Launching Grid on Demand
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 15 / 39
Web Interface Launching Grid on Demand
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 16 / 39
Web Interface Launching Grid on Demand
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 17 / 39
Web Interface Launching Grid on Demand
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 18 / 39
Web Interface Launching Grid on Demand
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 19 / 39
Web Interface Configuring the cluster
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 20 / 39
Web Interface Configuring the cluster
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 21 / 39
Web Interface Configuring the cluster
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 22 / 39
Web Interface Configuring the cluster
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 23 / 39
Web Interface Real Grid certificate usage
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 24 / 39
Web Interface Real Grid certificate usage
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 25 / 39
Web Interface Real Grid certificate usage
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 26 / 39
Web Interface Elasticity of the cluster
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 27 / 39
Web Interface Elasticity of the cluster
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 28 / 39
Web Interface Elasticity of the cluster
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 29 / 39
Web Interface Saving your settings
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 30 / 39
Web Interface Saving your settings
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 31 / 39
Web Interface Saving your settings
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 32 / 39
“Real use” and Performance WS-VLAM and WAVE
“Real use” and Performance test
To test the possibilities and performance of Grid on Demand a BiomedicalApplication and a workflow manager are used
Biomedical Application: WAVE
Parallel (MonteCarlo simulation) application
Workflow Manager
A (graphical) tool to assist complex e-Science application creation
Creates a series of jobsubmissions
WS-VLAM created by UvA SNE Group
Connects to Globus Grid Interface
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 33 / 39
“Real use” and Performance WS-VLAM and WAVE
“Real use” and Performance test
To test the possibilities and performance of Grid on Demand a BiomedicalApplication and a workflow manager are used
Biomedical Application: WAVE
Parallel (MonteCarlo simulation) application
Workflow Manager
A (graphical) tool to assist complex e-Science application creation
Creates a series of jobsubmissions
WS-VLAM created by UvA SNE Group
Connects to Globus Grid Interface
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 33 / 39
“Real use” and Performance WS-VLAM and WAVE
“Real use” and Performance test
To test the possibilities and performance of Grid on Demand a BiomedicalApplication and a workflow manager are used
Biomedical Application: WAVE
Parallel (MonteCarlo simulation) application
Workflow Manager
A (graphical) tool to assist complex e-Science application creation
Creates a series of jobsubmissions
WS-VLAM created by UvA SNE Group
Connects to Globus Grid Interface
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 33 / 39
“Real use” and Performance Connecting the Grid on Demand to a Workflow Manager
Connecting the Grid on Demand to a Workflow Manager
Compute Nodes
Head Node
OS: Ubuntu 10.04Grid: Globus 4.2.1PBS: Torque 2.6.8
WAVE Biomedical Compute
Application
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 34 / 39
“Real use” and Performance Preliminary performance results: total execution time
Preliminary performance results: total execution time
Grid on demand compared to a 32 node physical UvA cluster (DAS3) withGlobus Grid interface
Grid Jobs Total execution timeDAS3 28 00:42:58
Grid on Demand 28 00:46:22
DAS3 98 02:46:36
Grid on Demand 98 00:52:26
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 35 / 39
“Real use” and Performance Preliminary performance results: 30 Jobs
Preliminary performance results of job pending andrunning time: 30 jobs
00:00:00
00:10:00
00:20:00
00:30:00
00:40:00
00:50:00
01:00:00
01:10:00
01:20:00
0 5 10 15 20 25 30
Tim
e (
HH
:MM
:SS
)
Job Number
Performance on DAS3 - 30 Jobs
Running timePending time
00:00:00
00:10:00
00:20:00
00:30:00
00:40:00
00:50:00
01:00:00
01:10:00
01:20:00
0 5 10 15 20 25 30
Tim
e (
HH
:MM
:SS
)
Job Number
Performance on Grid on Demand 30 Jobs
Running timePending time
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 36 / 39
“Real use” and Performance Preliminary performance results: 100 Jobs
Preliminary performance results of job pending andrunning time: 100 jobs
00:00:00
00:15:00
00:30:00
00:45:00
01:00:00
01:15:00
01:30:00
01:45:00
02:00:00
02:15:00
02:30:00
0 10 20 30 40 50 60 70 80 90 100
Tim
e (
HH
:MM
:SS
)
Job Number
Performance on DAS3 - 100 Jobs
Running timePending time
00:00:00
00:15:00
00:30:00
00:45:00
01:00:00
01:15:00
01:30:00
01:45:00
02:00:00
02:15:00
02:30:00
0 20 40 60 80 100
Tim
e (
HH
:MM
:SS
)
Job Number
Performance on Grid on Demand - 100 Jobs
Running timePending time
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 37 / 39
Conclusion
Conclusion
Grid on Demand can offer Grid as a Cloud service
Parallel applications run/scale very well on Grid on Demand
Grid on Demand can be a solution for Urgent Computing demands
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 38 / 39
The End
Questions
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 39 / 39