1 virtual ez grid aaa/switch project nabil abdennadher hes-so, hepia (geneva)
TRANSCRIPT
![Page 1: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/1.jpg)
1
Virtual EZ GridAAA/Switch project
Nabil Abdennadher
HES-SO, hepia (Geneva)
![Page 2: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/2.jpg)
2Swiss Grid Day, Bern, 26th Nov. 2009
Virtual EZ Grid at a glance
• AAA/Switch funding
• Partners: UniGE, USI, UniNE, HES-SO
• Start/End dates : February 2009 - July 2010
• Objectives :
• Build a sustainable desktop Grid platform based on a Volunteer Computing (VC) middleware
• Evaluate it in a real world setting with two medical applications
![Page 3: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/3.jpg)
3Swiss Grid Day, Bern, 26th Nov. 2009
Virtual EZ Grid objectives• Infrastructure: build a grid platform with more than 1000 PCs
• Non dedicated nodes
• Belong to several institutions.
• Reliability: manage the volatility of the “worker” nodes
• Check pointing, migration, restarting, etc.
• Security: guarantee the security of resources providers
• Negotiation Model: provide a resource-credit system
• Applications: two medical packages
• NeuroWeb: build neural maps extended from brain measurements
• MedGift: medical image analysis application
![Page 4: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/4.jpg)
4Swiss Grid Day, Bern, 26th Nov. 2009
PLAN
• Virtual EZ Grid ingredients
• JOpera
• XtremWeb-CH
• EZ Grid
• Applications
![Page 5: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/5.jpg)
5Swiss Grid Day, Bern, 26th Nov. 2009
Virtual EZ Grid ingredients
JOpera (developed at USI)
•A grid workflow management system: http://www.jopera.org
XtremWeb-CH (developed at HES-SO)
•A volunteer computing platform : www.xtremwebch.net
EZ grid (developed at UniNE and UniGE)
•Based on virtualization
•Job check-pointing
•Migration and restarting
ARC
![Page 6: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/6.jpg)
6Swiss Grid Day, Bern, 26th Nov. 2009
Virtual EZ Grid architecture
Infrastructure
EZ-Grid
XtremWeb-CH (XWCH)
JOpera
ARC
![Page 7: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/7.jpg)
7Swiss Grid Day, Bern, 26th Nov. 2009
JOpera
“Provide the scientist with a platform that takes care of all data handling and record keeping chores so that the user can concentrate on the science and not computer science”
Make it easy to build Grid and VC applications composed of multiple jobs
Based on slides by Cesare Pautasso
![Page 8: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/8.jpg)
8Swiss Grid Day, Bern, 26th Nov. 2009
Drag, Drop and Connect
Based on slides by Cesare Pautasso
![Page 9: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/9.jpg)
9Swiss Grid Day, Bern, 26th Nov. 2009
Run, Monitor, Steer and Debug
Based on slides by Cesare Pautasso
![Page 10: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/10.jpg)
10Swiss Grid Day, Bern, 26th Nov. 2009
XtremWeb-CHcoordinator
Service Request
result
PC
Work Request
Work Result
Consumer(Users’ applications)
Providers(Workers)
PC
PC
PC
PC
PCPC
PC
PC
XtremWeb-CH at a glance
Job0
Job2
Job1Job3
Application
![Page 11: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/11.jpg)
11Swiss Grid Day, Bern, 26th Nov. 2009
XtremWeb-CHcoordinator
Communication protocol
Work Request
Worker
Work Result
Work Alive
![Page 12: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/12.jpg)
12Swiss Grid Day, Bern, 26th Nov. 2009
Use
r’ se
rvic
es
Java
AP
IC
/C+
+ A
PI
Pyt
hon
AP
I
Scheduler
XWCHDB
Work request
Work Result
Warehouse
Worker
Work Alive
Wor
ker
and
War
ehou
se s
ervi
ces
Use
r ap
plic
atio
nXtremWeb-CHCoordinator
Admin. Service
![Page 13: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/13.jpg)
13Swiss Grid Day, Bern, 26th Nov. 2009
User application (jobs generator)int main ()
{
int Job0, Job1, Job2, Job3
…
Job0 = addjob (…);
while (getJobStatus (Job0,…)) ≠ COMPLETE);
Job0
Job1 = addjob (…);Job2 = addjob (…);while ((getJobStatus (Job1,…)) ≠ COMPLETE) && (getJobStatus (Job2,…)) ≠ COMPLETE));
Job2
Job1
Job3 = addjob (…); while (getJobStatus (Job3,…)) ≠ COMPLETE);
getJobFileOut (Job3,…);}
Job3
![Page 14: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/14.jpg)
14Swiss Grid Day, Bern, 26th Nov. 2009
User application: waves generator
![Page 15: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/15.jpg)
15Swiss Grid Day, Bern, 26th Nov. 2009
EZ Grid
Isolates XWCH jobs from “local” jobs
Why?: guarantee the privacy of the providers
How?: use virtualization technology
Supports check-point, and migration
Why?: manage the volatility of nodes
How?: remotely monitor the XWCH virtual machine
![Page 16: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/16.jpg)
16Swiss Grid Day, Bern, 26th Nov. 2009
Work request
Work Result
Warehouse
Worker
Work Alive
XWCHcoordinator
OS (Windows)
VM Manager
OS (Linux)
XWCH
Application
EZ
Grid
mo
dul
e
OS (Linux)
XWCH
ApplicationL
oca
l app
lica
tion
XWCH worker
XWCH VM1 XWCH VM2
EZ Gridcoordinator
![Page 17: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/17.jpg)
18Swiss Grid Day, Bern, 26th Nov. 2009
PLAN
• Virtual EZ Grid ingredients
• JOpera
• XtremWeb-CH
• EZ Grid
• Applications
![Page 18: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/18.jpg)
19Swiss Grid Day, Bern, 26th Nov. 2009
Applications
NeuroWeb : build neuronal maps extracted from brain measurements
MedGift : medical image analysis and retrieval application
![Page 19: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/19.jpg)
20Swiss Grid Day, Bern, 26th Nov. 2009
NeuroWebObjective
• Reconstruction of the electromagnetic brain map
• Which neuron is responsible of what?— Epileptic crisis (*)
— Parkinson
— Alzheimer
— Etc.
Why?
• Avoid invasive surgeries
![Page 20: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/20.jpg)
21Swiss Grid Day, Bern, 26th Nov. 2009
Neuronal activity
MEG Scanner
# Sensors = 256dt = 1 millisecond
MEG scannerMagneto-EncephaloGraphy (MEG) scanner:provides temporal information (functional data)
Based on slides by Cédric Bilat
![Page 21: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/21.jpg)
22Swiss Grid Day, Bern, 26th Nov. 2009
?NeuroWeb
# of sensors: 256
A large scale optimization problem
# of neurons: 60’000
Based on slides by Cédric Bilat
![Page 22: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/22.jpg)
23Swiss Grid Day, Bern, 26th Nov. 2009
Mapping of the MEG signals
on voxels (IRM):
IRM
MEG
How does the system work?
Based on slides by Cédric Bilat
![Page 23: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/23.jpg)
24Swiss Grid Day, Bern, 26th Nov. 2009
Start with a random matrix A0
In each step, calculate Ai .
• Ai = F (Ai-1, functional data, anatomic data)
Stop when Ai = Ai-1
~16 Mb
~360 Mb
The Algorithm
![Page 24: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/24.jpg)
25Swiss Grid Day, Bern, 26th Nov. 2009
.
.
.
.
.
.
.
.
.
…
A0A1 Ai-1
Gridification
Ai
![Page 25: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/25.jpg)
26Swiss Grid Day, Bern, 26th Nov. 2009
HD
XWCH
Worker
Worker
node
Main Memory
Data persistence
Persistent
server
XWCH
Job
Why ?• Avoid loading/storing data
from/to the HD
How ?• Data remain in main memory
even after the end of the task
![Page 26: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/26.jpg)
27Swiss Grid Day, Bern, 26th Nov. 2009
Virtual EZ Grid today• Infrastructure: Around 500 workers
HES-SO (Geneva + Yverdon): ~250
UNiGe: ~ 200
UniNe: ~ 20
Univ. Franche Comté (France): ~30
• Reliability: The implementation of EZ Grid is in progress (50%)
• Security: Fully operational. based on SWITCHaai: Shibboleth-based AAI
• Negotiation Model: In progress (30%)
• Applications• NeuroWeb: a prototype (proof-of-concept) is already deployed
• MedGift: gridification will start soon (January 2010)
![Page 27: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/27.jpg)
28Swiss Grid Day, Bern, 26th Nov. 2009
Swiss Grid Portal
Virtual EZ-GridSwiss Multi-Science
Computing Grid
Infrastructure
Virtual EZ Grid : Links with other projects
![Page 28: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/28.jpg)
29Swiss Grid Day, Bern, 26th Nov. 2009
Model 1
• Prices are fixed by a “central” agency.
• All the “institutions” receives initial credit which:
• depends on the quantity of resources provided by the institution
• can be used against the usage.
• The client (from a given institution) chooses the priority of jobs. The goal is to:
• Minimize the cost of execution
• Execute the application ASAP
• The model determines the “best” prices which optimize the use of the platform
![Page 29: 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)](https://reader036.vdocuments.net/reader036/viewer/2022062409/56649f165503460f94c2c8c9/html5/thumbnails/29.jpg)
30Swiss Grid Day, Bern, 26th Nov. 2009
Model 2
XWCHcoordinator
Worker (1) Proposed price
(2) feedback(3) Price updating