custom split-merge support in a hierarchical gridrpc middleware
TRANSCRIPT
. . . . . .
Custom Split-Merge Support in a Hierarchical GridRPCMiddleware
Cristian KLEIN, Christian PÉREZ
Avalon / GRAAL, INRIA / LIP, ENS de Lyon
COOP Meeting, 11-12 April 2011, Rennes
Cristian KLEIN (INRIA) DIET Split-Merge COOP, Rennes 1 / 14
. . . . . .
DIET (Rappel)
.What is DIET?..
.. ..
.
.
Distributed Interactive Engineering ToolkitIn the context of COOP: Resource Management System
.Features..
.. ..
.
.
Hierarchical schedulingData managementClient-side schedulingData-flow scheduling
Cristian KLEIN (INRIA) DIET Split-Merge COOP, Rennes 2 / 14
. . . . . .
GridTLSE Use-Cases.Solving AX = B..
.. ..
.
.
A︸︷︷︸m×n
X︸︷︷︸n×p
= B︸︷︷︸m×p
Columns of B can be solved independentlyAX = B1︸︷︷︸
m×p1
AX = B2︸︷︷︸m×p2
· · ·
How to choose p1, p2, . . .?
Cristian KLEIN (INRIA) DIET Split-Merge COOP, Rennes 3 / 14
. . . . . .
GridTLSE Use-Cases.Solving AX = B..
.. ..
.
.
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
1 10 100 1000Tim
e p
er c
olu
mn
(s)
Number of columns solved simultaneously
Solving AX=B on a single host
Example: let ncols = 100 andsolve 1 column/requestsolve 10 columns/requestsolve 20 columns/request
Problem: How to generate resource-dependent requests in DIET?
Cristian KLEIN (INRIA) DIET Split-Merge COOP, Rennes 4 / 14
. . . . . .
GridTLSE Use-Cases.Solving AX = B..
.. ..
.
.
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
1 10 100 1000Tim
e p
er c
olu
mn
(s)
Number of columns solved simultaneously
Solving AX=B on a single host
Example: let ncols = 100 and multiple hostssolve 1 column/requestsolve 10 columns/requestsolve 20 columns/request
Problem: How to generate resource-dependent requests in DIET?
Cristian KLEIN (INRIA) DIET Split-Merge COOP, Rennes 4 / 14
. . . . . .
GridTLSE Use-Cases.Solving AX = B..
.. ..
.
.
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
1 10 100 1000Tim
e p
er c
olu
mn
(s)
Number of columns solved simultaneously
Solving AX=B on a single host
Example: let ncols = 100 and nH = 10solve 1 column/requestsolve 10 columns/requestsolve 20 columns/request
Problem: How to generate resource-dependent requests in DIET?
Cristian KLEIN (INRIA) DIET Split-Merge COOP, Rennes 4 / 14
. . . . . .
GridTLSE Use-Cases.Solving AX = B..
.. ..
.
.
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
1 10 100 1000Tim
e p
er c
olu
mn
(s)
Number of columns solved simultaneously
Solving AX=B on a single host
Example: let ncols = 100 and nH = 10solve 1 column/request 10 requests / host t = 7.50 ssolve 10 columns/requestsolve 20 columns/request
Problem: How to generate resource-dependent requests in DIET?
Cristian KLEIN (INRIA) DIET Split-Merge COOP, Rennes 4 / 14
. . . . . .
GridTLSE Use-Cases.Solving AX = B..
.. ..
.
.
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
1 10 100 1000Tim
e p
er c
olu
mn
(s)
Number of columns solved simultaneously
Solving AX=B on a single host
Example: let ncols = 100 and nH = 10solve 1 column/request 10 requests / host t = 7.50 ssolve 10 columns/request 1 request / host t = 2.15 ssolve 20 columns/request
Problem: How to generate resource-dependent requests in DIET?
Cristian KLEIN (INRIA) DIET Split-Merge COOP, Rennes 4 / 14
. . . . . .
GridTLSE Use-Cases.Solving AX = B..
.. ..
.
.
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
1 10 100 1000Tim
e p
er c
olu
mn
(s)
Number of columns solved simultaneously
Solving AX=B on a single host
Example: let ncols = 100 and nH = 10solve 1 column/request 10 requests / host t = 7.50 ssolve 10 columns/request 1 request / host t = 2.15 ssolve 20 columns/request 5 requests t = 5.00 s
Problem: How to generate resource-dependent requests in DIET?
Cristian KLEIN (INRIA) DIET Split-Merge COOP, Rennes 4 / 14
. . . . . .
GridTLSE Use-Cases.Solving AX = B..
.. ..
.
.
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
1 10 100 1000Tim
e p
er c
olu
mn
(s)
Number of columns solved simultaneously
Solving AX=B on a single host
Example: let ncols = 100 and nH = 10solve 1 column/request 10 requests / host t = 7.50 ssolve 10 columns/request 1 request / host t = 2.15 ssolve 20 columns/request 5 requests t = 5.00 s
Problem: How to generate resource-dependent requests in DIET?
Cristian KLEIN (INRIA) DIET Split-Merge COOP, Rennes 4 / 14
. . . . . .
.. .1 Introduction
.. .2 Solution: Split-Merge Support
.. .3 Example
.. .4 Conclusion
Cristian KLEIN (INRIA) DIET Split-Merge COOP, Rennes 5 / 14
. . . . . .
.. .1 Introduction
.. .2 Solution: Split-Merge Support
.. .3 Example
.. .4 Conclusion
Cristian KLEIN (INRIA) DIET Split-Merge COOP, Rennes 6 / 14
. . . . . .
DIET w/ Client-Side Scheduling
Cristian KLEIN (INRIA) DIET Split-Merge COOP, Rennes 7 / 14
. . . . . .
DIET w/ Client-Side Split-Merge
Cristian KLEIN (INRIA) DIET Split-Merge COOP, Rennes 8 / 14
. . . . . .
.. .1 Introduction
.. .2 Solution: Split-Merge Support
.. .3 Example
.. .4 Conclusion
Cristian KLEIN (INRIA) DIET Split-Merge COOP, Rennes 9 / 14
. . . . . .
Example: Computing the Sum of an Array
.Problem..
.. ..
.
.
“Efficiently” computing the (associative) sum of a given arrayOn heterogeneous resources
.Solution..
.. ..
.
.
SeDs send relative computation power in their estimation vectorsplit() assigns each SeD a sub-vector→ proportional to relative computation power
merge() sums the partial results
Cristian KLEIN (INRIA) DIET Split-Merge COOP, Rennes 10 / 14
. . . . . .
bool CustomSM::split(diet_profile_t *profile, corba_response_t *response,SubRequests &subrequests)
{if (strcmp(profile->pb_name, "sum_vector") != 0) return false;
/* Get original vector */long *vector;size_t vectorLength;diet_vector_get(diet_parameter(profile, 0), &vector, NULL, &vectorLength);
/* Compute total computation power */int numDiscoveredSeDs = response->servers.length();double totalPower = 0;for (int i = 0; i < numDiscoveredSeDs; i++) {
double thisPower = diet_est_get(&response->servers[i].estim, ESTTAG_RELPOWER, 0);totalPower += thisPower;
}
/* Scatter subrequests */size_t subVectorStart = 0;for (int i = 0; i < numDiscoveredSeDs; i++) {
double thisPower = diet_est_get(&response->servers[i].estim, ESTTAG_RELPOWER, 0);char *SeDName = response->servers[i].loc.SeDName;
long subVectorLength = vectorLength * thisPower / totalPower;diet_profile_t* subprofile = diet_profile_alloc(profile->pb_name, 0, 0, 1);diet_vector_set(diet_parameter(subprofile, 0), &vector[subVectorStart]);subVectorStart += subVectorLength;
subrequests.push_back(SubRequests::value_type(subprofile, SeDName));}
return true;}
Cristian KLEIN (INRIA) DIET Split-Merge COOP, Rennes 11 / 14
. . . . . .
bool CustomSM::merge(diet_profile_t *profile, corba_response_t *response,SubRequests &subrequests)
{/* Gather responses */long *out = new long(0);for (SubRequests::iterator it = subrequests.begin();
it != subrequests.end(); it++) {long *subOutValue;
diet_profile_t *subprofile = it->first;diet_scalar_get(diet_parameter(subprofile, 1), &subOutValue, NULL);diet_profile_free(subprofile);
*out += *subOutValue;}diet_scalar_set( diet_parameter( profile, 1 ), out, DIET_VOLATILE, DIET_LONGINT );
return true;}
int main(){
...CustomSM clientScheduler;SpecificClientScheduler::setScheduler(&clientScheduler);...
}
Cristian KLEIN (INRIA) DIET Split-Merge COOP, Rennes 12 / 14
. . . . . .
.. .1 Introduction
.. .2 Solution: Split-Merge Support
.. .3 Example
.. .4 Conclusion
Cristian KLEIN (INRIA) DIET Split-Merge COOP, Rennes 13 / 14
. . . . . .
Conclusion
.DIET Client Split-Merge Support..
.. ..
.
.
We extended the DIET’s client-side schedulingApplication can easily hook their custom split-mergeAllows to adapt DIET requests to available resources(even in the heterogeneous case)
.Future Work..
.. ..
.
.
Apply to GridTLSE use-cases:I What information from SeDs? What algorithm?
More tests, experiments, . . .Study split-merge support in DIET data-flow engine
Cristian KLEIN (INRIA) DIET Split-Merge COOP, Rennes 14 / 14