[ieee comput. soc 12th ieee international symposium on high performance distributed computing -...

Trace-Based Simulations ofProcessor Co-Allocation Policies in Multiclusters

A.I.D. Bucur and D.H.J. EpemaFaculty of Information Technology and Systems

Delft University of TechnologyP.O. Box 5031, 2600 GA Delft, The Netherlandse-mail: A.I.D.Bucur, [email protected]

Abstract

In systems consisting of multiple clusters of proces-sors which employ space sharing for scheduling jobs,such as our Distributed ASCI1 Supercomputer (DAS), co-allocation, i.e., the simultaneous allocation of processors tosingle jobs in multiple clusters, may be required. In this pa-per we study the performance of several scheduling policiesfor co-allocating unordered requests in multiclusters witha workload derived from the DAS. We find that beside thepolicy, limiting the total job size significantly improves theperformance, and that for a slowdown of jobs due to globalcommunication bounded by ��, co-allocation is a viablechoice.

1 Introduction

Over the last decade, clusters and distributed-memorymultiprocessors consisting of hundreds or thousands ofstandard CPUs have become very popular. In addition, re-cent work in computational and data grids [2] enables appli-cations to access resources in different and possibly widelydispersed locations simultaneously—that is, to employ pro-cessor co-allocation—to accomplish their goals, effectivelycreating single multicluster systems. Scheduling paralleljobs in single-cluster systems has received very much at-tention. In this paper we study scheduling in multiclustersystems, and we restrict ourselves to rigid jobs (which havepredefined, fixed sizes) and to space sharing, in which jobsrun to completion on exclusively allocated processors.

We evaluate the performance (in terms of average re-sponse times and maximal utilization) of several schedulingpolicies for co-allocating unordered requests—these spec-

1In this paper, ASCI refers to the Advanced School for Computing andImaging in The Netherlands, which came into existence before, and is un-related to, the US Accelerated Strategic Computing Initiative.

ify the numbers of processors to be allocated in differentclusters, but not the actual clusters—in multiclusters with aworkload derived from a real system. For comparison, weassess the performance of total requests, which only specifythe total numbers of processors needed, equal to the num-bers required by unordered requests, in a single cluster ofthe same total size. The best multicluster policy turns out tobe LS (for each cluster there is a local queue and all jobs goto those queues; multi-component jobs are spread across theclusters, single-component jobs are scheduled on the localclusters); if we consider the gross utilization (includes thewide-area communication), in some cases LS even comesclose to using FCFS for total requests in a single cluster.However, in multiclusters a large amount of utilization isspent on global communication, and when comparing thenet utilization (takes into account only the computationsand the local communication) the single-cluster system per-forms better. Although the choice of policy does matter forthe performance, we found that the largest improvement canbe obtained from simply limiting the total job size.

In previous papers [6, 7, 8], we have assessed the influ-ence on the mean response time and maximal utilization ofthe job structure and size, the sizes of the clusters in thesystem, the ratio of the speeds of local and wide-area com-munications, and of the presence of a single or of multiplequeues in the system.

Our five-cluster second-generation Distributed ASCI Su-percomputer (DAS) [1, 12] (and its predecessor), which wasan important motivation for this work, was designed to as-sess the feasibility of running parallel applications acrosswide-area systems [14]. In the most general setting, gridresources are very heterogeneous; in this paper we restrictourselves to homogeneous multicluster systems such as theDAS. Showing the viability of co-allocation in such sys-tems may be regarded as a first step in assessing the benefitof co-allocation in more general grid environments.

Proceedings of the 12th IEEE International Symposium on High Performance Distributed Computing (HPDC’03)

1082-8907/03 $17.00 © 2003 IEEE

2 The model

In this section we describe our model of multicluster sys-tems based on the DAS system.

2.1 The DAS system

The DAS (in fact the DAS2, the second-generation sys-tem which was installed at the end of 2001 when the first-generation DAS1 system was discontinued) [1, 12] is awide-area computer system consisting of five clusters ofdual-processor nodes, one with 72, the other four with 32nodes each. The clusters are interconnected by the Dutchuniversity backbone for wide-area communications, whilefor local communications inside the clusters Myrinet LANsare used. The system was designed for research on paralleland distributed computing. On single DAS clusters the PBSscheduler [4] is used, while jobs spanning multiple clusterscan be submitted with Globus [3].

2.2 The structure of the system

We model a multicluster system consisting of C clustersof processors, of possibly different sizes. We assume that allprocessors have the same service rate. By a job we under-stand a parallel application requiring some number of pro-cessors, possibly in multiple clusters (co-allocation). Jobsare rigid, so the numbers of processors requested by and al-located to a job are fixed. We call a task the part of a jobthat runs on a single processor. We assume that jobs onlyrequest processors and we do not include in the model othertypes of resources. For interarrival times we use exponentialdistributions.

2.3 The structure of job requests and the place-ment policies

Jobs that require co-allocation have to specify the num-ber and the sizes of their components, i.e., of the sets oftasks that have to go to the separate clusters. A job is repre-sented by a tuple of C values, at least one of which is strictlypositive.

We consider unordered requests, where by the compo-nents of the tuple the job only specifies the numbers of pro-cessors it needs in the separate clusters, allowing the sched-uler to choose the clusters for the components. For com-parison we introduce total requests, where there are onlysingle-component jobs, scheduled in a single-cluster sys-tem with FCFS. Unordered requests model applications likeFFT, where tasks in the same job component share data andneed intensive communication, while tasks from differentcomponents exchange little or no information. To deter-mine whether an unordered request fits, we try to scheduleits components in decreasing order of their sizes on distinct

Table 1. The fractions of jobs with sizes pow-ers of two

total job size fraction of the jobs

1 0.0912 0.1304 0.0878 0.06616 0.09032 0.03964 0.190

128 0.012

clusters. We use Worst Fit (WF) to place the componentson clusters.

2.4 The workload

Although co-allocation is possible on the DAS, so far ithas not been used enough to let us gather statistics on thesizes of the jobs’ components. Before we started our simu-lations we could not obtain a relevant log from the recentlyinstalled DAS2. However, from the log of the largest cluster(128 processors) of the DAS1 we found that over a periodof three months, the cluster was used by 20 different userswho ran �� jobs. The sizes of the job requests took 58values in the interval �� , for an average of �� anda coefficient of variation of ��; their density is presentedin Fig. 1. The results are in line with those presented inthe literature for other systems, in that there is an obviouspreference for small numbers and powers of two [10] (seealso Table 1).

By sampling the job-size distribution as measured on theDAS1 we derive two distributions which we use in our sim-ulations for the total job sizes. DAS-s-128 is based on theentire log, while DAS-s-64 is obtained from the log cut at� (the average job size for DAS-s-64 is �� and the co-efficient of variation is ��); the maximum size of a jobis reduced to half excluding only � of the jobs from thelog—the percentage of jobs that require more than � pro-cessors. We introduce DAS-s-64 to check whether limitingthe total job size improves the performance.

When simulating total requests in a single cluster the to-tal job-size distribution is directly used. In the multiclustercase, a size limit is set for the job components; the numberof components of a job is computed depending on the sizelimit so that no component is larger than it, as long as thenumber of components does not exceed the number of clus-ters. Once we decide on the number of components of thejob, we split it into components of sizes as equal as possi-ble. We consider three size limits for the components: �,�� and ��; with the size limit we vary the numbers and thesizes of the job components (see Table 2).


1082-8907/03 $17.00 © 2003 IEEE

0

1000

2000

3000

4000

5000

6000

0 20 40 60 80 100 120

Num

ber

of J

obs

Nodes Requested

powers of 2other numbers

Figure 1. The density of the job-request sizesfor the largest DAS1 cluster (128 processors)

Table 2. The fractions of jobs with the differentnumbers of components for the DAS-s-128distribution and the three job-component-size limits

Job-component-size limit Number of job components1 2 3 4

16 0.513 0.267 0.090 0.21124 0.738 0.051 0.194 0.01732 0.780 0.200 0.003 0.017

In the DAS1 log, �� jobs were recorded with boththeir starting and ending times, and we could compute theirservice times. Influenced by the fact that during work-ing hours jobs are restricted to �� minutes of service (theyare automatically killed after that period), �� of therecorded jobs ran for less than ��minutes. Figure 2 presentsthe density of service time values on the DAS1, as it wasobtained from the log. The average service time is ��

seconds and the coefficient of variation is ��. In our simu-lations, we use for the service-time distribution the distribu-tion derived from the log of the DAS, cut off at � seconds(DAS-t-900). The average service time for the jobs in thecut log is �� and the coefficient of variation is ��.

We include wide-area communication in the model byextending the service times of all multi-component jobswith a factor, independent on the number of components.In [14], the performance of four parallel applications inwide-area systems is assessed by comparing the speedupsof the original applications on a 64-cluster system with thespeedups of versions of the applications optimized for wide-area execution on a 4x16 multicluster system. For three ofthe four applications, these speedups are �� and ��, ��

and �� (which is counter-intuitive, but the reason is ex-plained in [14]), and �� and ��, respectively; the fourthapplication (All-pair Shortest Paths) has very poor multi-cluster performance. So for these three applications, the

0

200

400

600

800

1000

1200

1400

0 100 200 300 400 500 600 700 800 900

Num

ber

of J

obs

Service Time (s)

Figure 2. The density of the service times forthe largest DAS1 cluster (128 processors)

extension factor for multicluster operation does not exceed��. In [5], experiences are presented with distributing alarge, wide-area optimized application solving Einstein’sequations across four supercomputers in two locations us-ing Globus. For a total of 1140 processors, an efficiency of�� is cited, which amounts to an extension factor of ��.In addition, in [11] it was concluded that it pays to use co-allocation when the extension factor is ��. We use �� asthe extension factor of the service times of multi-componentjobs, which is a realistic upper bound for many application.

We call ”gross utilization” the utilization obtained forthe system with extended service times. Since there is nopreemption for communication, the processors are consid-ered busy also when the jobs running on them are involvedin inter-cluster communication. The ”net utilization” takesinto account only the processor usage for computations andfast local communication, i.e., the non-extended servicetimes. In Sect. 4 we compare the two types of utilizationfor several policies and component-size limits.

2.5 The scheduling policies

In a multicluster system where co-allocation is used, jobscan be either single-component or multi-component, and ina general case both types are simultaneously present in thesystem. A scheduler dealing with the first type of jobs canbe local to a cluster and does not need any knowledge aboutthe rest of the system. For multi-component jobs, the sched-uler needs global information for its decisions.

Treating both types of jobs equally or keeping single-component jobs local and scheduling only multi-componentjobs globally over the entire system, having a single globalscheduler or schedulers local to each cluster, all these aredecisions that influence the performance of the system. In[8] we have studied several policies, some of which withmultiple variations; in this paper we consider the followingapproaches:

1. [GS] The system has one global scheduler with one


1082-8907/03 $17.00 © 2003 IEEE

global queue, for both single- and multi-componentjobs. All jobs are submitted to the global queue. Theglobal scheduler knows at any moment the number ofidle processors in each cluster and based on this infor-mation chooses the clusters for each job.

2. [LS] Each cluster has its own local scheduler witha local queue. All queues receive both single- andmulti-component jobs and each local scheduler hasglobal knowledge about the numbers of idle proces-sors. However, single-component jobs are scheduledonly on the local cluster. The multi-component jobsare co-allocated over the entire system. When schedul-ing is performed all enabled queues are repeatedly vis-ited, and in each round at most one job from eachqueue is started. When the job at the head of a queuedoes not fit, the queue is disabled until the next jobdeparts from the system. At each job departure thequeues are enabled in the same order in which theywere disabled.

3. [LP] Each cluster has its own local scheduler with alocal queue and all the single-component jobs are dis-tributed among the local queues, and there is a globalscheduler with a global queue where all the multi-component jobs are placed. The local schedulers havepriority: the global scheduler can schedule jobs onlywhen at least one local queue is empty. When a jobdeparts, if one or more of the local queues are emptyboth the global queue and the local queues are enabled.If no local queue is empty only the local queues areenabled and repeatedly visited; the global queue is en-abled and added to the list of queues which are visitedwhen at least one of the local queues gets empty. Whenboth the global queue and the local queues are enabledat job departures, they are always enabled starting withthe global queue. The order in which the local queuesare enabled does not matter since the jobs in them areonly started on the local clusters.

In all the cases considered, both the local and the globalschedulers use the FCFS policy to choose the next job torun.

For comparison, we consider the single-cluster casewhere there are only single-component jobs and we useFCFS as scheduling policy ([SC]).

3 Performance evaluation

In this section we assess the performance of multiclustersystems for the scheduling policies introduced (Sect. 2.5),for the DAS-t-900, DAS-s-128 and DAS-s-64 workload dis-tributions, for several job-component-size limits and for dif-ferent ways of distributing the jobs across the local queues.

We compare the results to those for a single cluster withFCFS.

The simulations are for a multicluster with � clusters of�� processors each and a single cluster with �� processors.For the policies where local queues are defined, we com-pare the balanced case (all local queues receive the samepercentage of jobs) to the unbalanced case when one localqueue receives �� and the other three �� of the jobs sub-mitted locally. The simulation programs were implementedusing the CSIM simulation package [13].

In this section we only look at the gross utilization, anddepict the response time as a function of this utilization be-cause that is a fair basis for comparing the policies. For SCthere is no wide-area communication and the net utilizationis equal to the gross utilization. In the figures, the legendsdescribe the curves in decreasing order of their performance(right-left).

Section 3.1 makes a general comparison of the policies.In Sect. 3.2 we study the effect of limiting the total jobsize. Section 3.3 discusses the impact on performance ofthe job-component-size limit. Section 4 compares for allthe policies and size limits the gross and the net utilization,which shows how efficient the global applications use thegross utilization offered.

3.1 Comparing the policies

In this section we compare the three policies defined formulticlusters to the FCFS policy in a single cluster. Formulticlusters, with the maximum job-component size wevary the numbers and sizes of job components; it influencesthe performance by modifying the way jobs fit together and,through the percentage of single-component jobs, the per-centage of jobs with extended service times.

3.1.1 Multiclusters versus single clusters

In this section we compare the four policies for each of thethree job-component-size limits. For SC the size limit doesnot influence the results, so the SC curves can be used asa reference for the graphs in Fig. 3. With the workloadconsidered the performance is poor for all policies and allsize limits. Even for total requests the maximal utilization isbelow ��. This seems to indicate that the bad performanceis caused by the total sizes of the jobs; we further investigatethis aspect in Sect. 3.2.

Even though it restricts single-component jobs to thelocal clusters and uses co-allocation to schedule its manymulti-component jobs (��) for which the service timesare extended due to the global communication, LS performsmuch better than the other multicluster policies for a sizelimit of ��; it also provides a higher maximal utilizationthan SC, but a part of that utilization is spent waiting forthe wide-area communication, so SC is still significantly


1082-8907/03 $17.00 © 2003 IEEE

0

2000

4000

6000

8000

10000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Ave

rage

Res

pons

e T

ime

Utilization

LSSCGSLP

0

2000

4000

6000

8000

10000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Ave

rage

Res

pons

e T

ime

Utilization

SCLSGSLP

0

2000

4000

6000

8000

10000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Ave

rage

Res

pons

e T

ime

Utilization

SCLSGSLP

0

2000

4000

6000

8000

10000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Ave

rage

Res

pons

e T

ime

Utilization

SCLSGSLP

0

2000

4000

6000

8000

10000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Ave

rage

Res

pons

e T

ime

Utilization

SCLSGSLP

0

2000

4000

6000

8000

10000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Ave

rage

Res

pons

e T

ime

Utilization

SCGSLPLS

Figure 3. The performance of the policies for job-component-size limits of ��, �� and �� (left-right); forLS and LP we depict results with balanced local queues (top) and unbalanced local queues (bottom)

0

2500

5000

7500

10000

12500

15000

17500

20000

GS LS LP SC

LocalTotal AverageGlobal

Gross Util. 0.552Net Util. 0.453

0

2500

5000

7500

10000

12500

15000

17500

20000

GS LS LP SC



30443

0

2500

5000

7500

10000

12500

15000

17500

20000

GS LS LP SC



60932

0

2500

5000

7500

10000

12500

15000

17500

20000

GS LS LP SC



21360

0

2500

5000

7500

10000

12500

15000

17500

20000

GS LS LP SC



48651

0

2500

5000

7500

10000

12500

15000

17500

20000

GS LS LP SC



52726

Figure 4. The response times for job-component size limits of ��, �� and �� (left-right) close to LP’ssaturation point; for LS and LP the local queues are balanced (top) and unbalanced (bottom)

better (see also Sect. 4). The main advantage of LS isthat it distributes the multi-component jobs among the localqueues allowing the local schedulers to spread them acrossthe clusters; at each moment a job can be chosen from anyof the local queues, which generates a form of backfillingwith a window equal to the number of clusters. For size

limits of �� and ��, where there are only �� and ��

multi-component jobs respectively, the performance of LSis worse. In all the graphs LP displays the worst resultsbecause all the multi-component jobs are placed in a sin-gle global queue, and all the single-component jobs are re-stricted to the local clusters. Although GS has only a sin-


1082-8907/03 $17.00 © 2003 IEEE

gle global queue, it is consistently better than LP, and fora size limit of �� (when there are many local jobs) it evenapproaches LS; this is due to the fact that it has the freedomto choose the clusters for the single-component jobs.

In itself, the fact that a multicluster policy can have sim-ilar performance to SC in terms of gross utilization makesco-allocation a good option, showing that the fragmenta-tion caused by having multiple clusters can be overcome.Even when possible, admitting only single-component jobswould result in a lower utilization as our previous resultshave shown (see [7]). Combining these results, it seemsthat a multicluster system without co-allocation deals worsewith the fragmentation. The internal loss of utilizationcaused by the slow global communication should be han-dled by structuring the applications to use the high utiliza-tion offered by some of the policies, while minimizing theamount of time spent with wide-area communication. Anapplication involved in less global communication can ben-efit better from a policy that uses co-allocation.

3.1.2 Balanced versus unbalanced local queues

Comparing the balanced and unbalanced cases for LS andLP (see Fig. 3) we notice that an unbalanced load for thelocal queues has a negative impact on performance. Sincefor both GS and SC all jobs go to the global queue, theircurves can be used as a reference when comparing the pairsof balanced-unbalanced graphs for LS and LP.

The worsening of the performance is more pronouncedfor LS, especially for larger job-component-size limits,when there is a higher percentage of local jobs. One causeis that the least loaded queues get empty sooner, which de-creases the backfilling window for the global jobs. The de-terioration is stronger when there is a high percentage oflocal jobs which indicates a second cause: the local jobsare restricted to their corresponding clusters, and a higherpercentage of (local) jobs in a queue means a higher loadfor the local cluster; besides, their cluster can be used withequal priority by global jobs from the other local queues. Asa result, the more loaded local queue saturates faster than inthe balanced case, decreasing this way the maximal utiliza-tion of the system. For a size limit of �� and unbalancedlocal queues, LS performs worse than GS and similarly toLP.

For LP the performance deterioration due to the unbal-ance of the local queues is small for all size limits. Allglobal jobs go to the global queue, and the local queueshave priority on their local clusters as long as none of themis empty. When there are few local jobs, the loads of alllocal queues are low even in the unbalanced case (for a sizelimit of �� the most loaded local queue receives �� ofall jobs); for a high percentage of local jobs the local queuesaccess their clusters with priority, which reduces the impactof the unbalance. Even when a local queue gets empty and

multi-component jobs from the global queue are started, theuse of WF insures that the least loaded clusters are chosen.

3.1.3 LP approaching saturation

For LP, we can expect that the performance differs betweenthe global and local queues. In Fig. 4 we depict for the LPpolicy, beside the total average response times, the averageresponse times for the local queues and for the global queue.Beside the gross utilization we display in each chart the cor-responding net utilization. The figure shows the same re-sults as Fig. 3, but for a small set of utilization values cho-sen so that at least one of the policies approaches saturation.This representation makes it easier to compare the responsetimes of the policies but has the limitation of providing lessinformation. At the chosen utilization values, LP, the policywith the worst performance, is approaching saturation—theglobal queue grows without bounds. As a consequence, wecan easily see for LP the differences in response times, forexample among the balanced and unbalanced cases, eventhough as the graphs in Fig. 3 show these differences aresmall. On the other hand, if we compare the response timesfor the ”better” policies like LS, the charts do not show thesignificant performance loss from the unbalanced case. Atthe utilizations from Fig. 4, LS is on a very low point onits response-time curve and its performance is very goodin both the balanced and the unbalanced cases. LP delaysthe large multi-component jobs much longer than LS andits performance bottleneck is the global queue and not thelocal ones. This is shown in Fig. 4 where for LP the av-erage response times for the global queue are much largerthan those for the local queues. The response times for theglobal queue are displayed on the corresponding bars in thecharts.

3.1.4 Conclusions

We conclude that for a system with a workload similar tothat of the DAS, and an overhead due to the wide-area com-munication covered by the �� extension factor, the LSpolicy is the best option. In all the cases the performance isvery poor: for all job-component-size limits, both in the bal-anced and the unbalanced cases, and even for total requests.The main cause seems to be the total job-size distribution,and not the job-component-size limit, the global communi-cation, the policies, or the extra fragmentation introducedby scheduling multi-component jobs in a multicluster sys-tem. We verify this assumption in the next section.

3.2 Limiting the total job size

In the job-size distribution derived from the DAS, only�� of the jobs require more than �� processors (out ofwhich �� need �� processors to run, which is the entire


1082-8907/03 $17.00 © 2003 IEEE

system). Our assumption is that eliminating from the dis-tribution this small percentage of very large jobs can bringimportant changes to our previous results.

Figure 5 compares the performance of the DAS-s-128and the DAS-s-64 distributions for the four policies dis-cussed before, a job-component-size-limit equal to �� andbalanced local queues. We choose this set of parameters be-cause they previously provided the most interesting results(LS better than SC).

0

2000

4000

6000

8000

10000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Ave

rage

Res

pons

e T

ime

Utilization

SC 64LS 64LP 64GS 64

LS 128SC 128GS 128LP 128

Figure 5. The response times for maximal to-tal job size 64 and 128 (job-component-sizelimit ��, balanced local queues)

With DAS-s-64, the improvements in performance arelarge for LS, and even more so for SC. When a job requir-ing �� (or a similarly large number of) processors is at thetop of the queue, SC waits for the entire system to becomeempty, which yields a very low utilization. LS on the otherhand can run jobs from the other queues and postpone thelarge job until either the other queues are empty, or they alsohave at the top large jobs that do not fit. For DAS-s-64 thelargest jobs in the system require only half of the proces-sors, which improves the utilization of SC and diminishesthe advantage of LS.

LP and GS also perform better for DAS-s-64. Since itgives the local queues priority, with LP the very large jobsin the global queue are much postponed for DAS-s-128. ForDAS-s-64 jobs are smaller, and LP outperforms GS becauseit can benefit from having more queues (also a form of back-filling but worse than in the case of LS since all global jobsgo to the global queue).

A very small percentage of very large jobs can signifi-cantly worsen the performance, and rather than designinga complicated policy to deal with such jobs, simply impos-ing a maximum size for the jobs submitted to the systembrings more important improvements. Of course, for theusers whose jobs are larger than the limit allowed, comply-ing to this restriction translates into reconfiguring their jobsto use fewer processors and accepting the consequence of

having longer service times. The extension of the servicetimes is highly dependent on the specific application.

3.3 Setting the job-component-size limit

In this section we evaluate the performance of GS, LSand LP depending on the maximum size admitted for thejob components. We compare three size limits: ��, �� and�� (see Fig. 6), and use the DAS-s-128 distribution.

Having smaller job components improves the system’sperformance, but having many components relative to thenumber of clusters deteriorates the performance. A smallersize limit also means more multi-component jobs, so morejobs with extended service times, which also worsens theperformance; for a size limit of �� there are �� moremulti-component jobs than for a size limit of ��. Adding upall these factors, for GS a size limit of �� provides slightlybetter results than ��.

For LS, due to the backfilling effect and to the fact thatmulti-component jobs can be spread across any of the clus-ters, while the single-component jobs are restricted to thelocal clusters, a smaller size limit becomes an important ad-vantage. For this policy a size limit of �� is a much betterchoice than a size limit of ��. Comparing the balanced andunbalanced cases for LS we notice that the performance de-teriorates more when the local queues are not balanced fora size limit of ��. The queue receiving extra load can placethe multi-component jobs also on other clusters, but it has tokeep local the single-component jobs, overloading the localcluster. It means that the more single-component jobs thereare, the more will the system be affected by the unbalance.

For LP, all multi-component jobs go to the global queueand only the single-component jobs are spread among thelocal queues. Smaller local jobs fit better, and fewer localjobs mean fewer jobs restricted to the local clusters. Fewerlocal jobs also mean that the local queues empty faster andthe global queue gets more often the chance to start jobs. Onthe other hand, the global queue has low priority when nolocal queue is empty, and the more global jobs, the higherthe average response time for those jobs. As Fig. 6 shows,LP performs better for a size limit of �� than for ��, but thedifference is small in both the balanced and the unbalancedcases.

For all the policies, the worst results are obtained for ajob-component size limit of ��. Since all the factors dis-cussed above would place a system with limit �� in betweenthe other two cases, the reason for this very large differ-ence in performance should be in the way jobs fit togetherin the system, depending on the size limit of their compo-nents. For a size limit of ��, the jobs split up differentlyare only those with sizes (integers) in the intervals �� and �� when compared to a size limit of ��, and the in-tervals �� , �� and �� when compared to ��.Since for GS and LP the performance is very similar for size


1082-8907/03 $17.00 © 2003 IEEE

0

2000

4000

6000

8000

10000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Ave

rage

Res

pons

e T

ime

Utilization

LS 16LS 32LS 24

0

2000

4000

6000

8000

10000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Ave

rage

Res

pons

e T

ime

Utilization

LP 16LP 32LP 24

0

2000

4000

6000

8000

10000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Ave

rage

Res

pons

e T

ime

Utilization

GS 32GS 16GS 24

0

2000

4000

6000

8000

10000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Ave

rage

Res

pons

e T

ime

Utilization

LS 16LS 32LS 24

0

2000

4000

6000

8000

10000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Ave

rage

Res

pons

e T

ime

Utilization

LP 16LP 32LP 24

Figure 6. The performance of LS, LP and GS (left-right) depending on the size limit of the job compo-nents. For LS and LP both the balanced (top) and unbalanced (bottom) cases are depicted

limits of �� and ��, it makes sense to look for the reasonof the much worse performance when the size limit is ��in the interval �� , which contains the jobs split differ-ently compared to both the other limits. Checking the log,we found that the most relevant size in that interval is ��:�� of the jobs in the log have this size. For a size limitof ��, the corresponding job request is �� , fora size limit of �� it is �� and for a limit of �� weobtain �� . Considering an empty system whichreceives a job of size ��, in the first two cases after the job isplaced there are many jobs, with different numbers of com-ponents and sizes up to �� that would still fit in the system.However, in the third case only single-component jobs withmaximum sizes of �� and �� can fit in three of the clusters,and single-component jobs with a maximum size of �� (dueto the size limit) in the fourth, empty cluster. A second jobwith a size of �� would also fit in the first two cases, but notin the third.

4 Gross versus net utilization

In Sect. 3 we have studied the average response time asa function of the gross utilization. In this section we dis-cuss the difference between gross and net utilization, andquantify this difference for the cases considered in Sect. 3.We have defined in Sect. 2.4 the net and the gross uti-lization based on the job service times in single clusterswith fast local communication, and on the extended servicetimes (at least, for multi-component jobs) to account for theslow wide-area communications, respectively. The differ-

ence between these utilizations is the capacity lost internallyin multi-component jobs due to slow wide-area links. Thisinternal capacity loss might be reduced by restructuring ap-plications or by having them use (collective-) communica-tion operations optimized for wide-area systems.

The performance of a multicluster policy may look goodwhen considering the response time as a function of thegross utilization, but, when there is much internal capacityloss, the performance as a function of the net utilization (orof the throughput) may be poor. This ”real” performanceof a multicluster policy would improve with more efficientapplications or with faster global communication (smallerextension factor). In the extreme case, if the global commu-nication was as fast as the local communication, LS wouldsometimes provide even better performance than SC (seeFig. 3).

In Fig. 7 we depict the average response time for ourthree policies, for the DAS-s-128 job-size distribution, andfor the three job-component-size limits as a function of boththe gross and the net utilization. For LS and LP the lo-cal queues are balanced; the results for unbalanced localqueues do not bring additional information. To assess thedifference between the two utilizations for a specific policyand job-component-size limit at a certain response time, oneshould compare the graphs in the horizontal direction. Ofcourse, for the same workload (defined by the arrival rate,and so, by the net utilization) and job-component-size limit,the difference between the gross and the net utilization is thesame for all scheduling policies, albeit at possibly different


1082-8907/03 $17.00 © 2003 IEEE

0

2000

4000

6000

8000

10000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Ave

rage

Res

pons

e T

ime

Utilization

LS 16 grossLS 16 net

0

2000

4000

6000

8000

10000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Ave

rage

Res

pons

e T

ime

Utilization


0

2000

4000

6000

8000

10000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Ave

rage

Res

pons

e T

ime

Utilization


0

2000

4000

6000

8000

10000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Ave

rage

Res

pons

e T

ime

Utilization

LP 16 grossLP 16 net

0

2000

4000

6000

8000

10000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Ave

rage

Res

pons

e T

ime

Utilization


0

2000

4000

6000

8000

10000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Ave

rage

Res

pons

e T

ime

Utilization


0

2000

4000

6000

8000

10000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Ave

rage

Res

pons

e T

ime

Utilization

GS 16 grossGS 16 net

0

2000

4000

6000

8000

10000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Ave

rage

Res

pons

e T

ime

Utilization


0

2000

4000

6000

8000

10000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1A

vera

ge R

espo

nse

Tim

eUtilization


Figure 7. The response time as a function of the gross and the net utilization for the LS, LP and GSpolicies and for the three job-component-size limits (balanced local queues for LS and LP)

response times.

In our model, job sizes and job service times are inde-pendent. This means that we can compute the ratio betweenthe gross and the net utilization, independent of the schedul-ing policy, as the quotient of the weighted average total jobsize with single-component jobs having weight � and multi-component jobs having weight �� (the extension factor)and the average total job size. For the DAS-s-128 job-sizedistributionwe find for the job-component-size limits of ��,��, and �� ratios of gross and net utilization of ��, ��,and ��, respectively.

The difference between the gross and the net utilizationgrows with a decrease of the job-component-size limit—the more multi-component jobs the higher the amountof global, slow communication—and with the number of(multi-component) jobs that fit on the system—when jobsfit better, the workload can be higher, entailing more globalcommunication. For these reasons, for all the policies a job-component-size limit of �� yields a larger distance between

the curves of the net and gross utilizations than �� and ��.For a size limit of �� there are ��more multi-componentjobs than for ��, but since jobs fit very poorly, the gross andnet utilizations are closer together when the size limit is ��.Comparing the policies, we notice that for a size limit of��, LS yields a much better utilization, and so also morewide-area communication, than the other policies, and as aconsequence, it has the largest difference between the grossand the net utilization. For component-size limits of �� and�� the amount of utilization spent in global communicationis rather similar for all three policies.

In [9], we have studied the maximal utilization (beyondwhich the system gets unstable) of co-allocation with an-alytic means (when the service times are exponential) andwith simulations. Both methods only work when there is asingle, global queue, so in the cases of GS and SC. In thesesimulations, we maintain a constant backlog and observethe time-average fraction of processors being busy, whichyields the maximal gross utilization. For GS, the simula-


1082-8907/03 $17.00 © 2003 IEEE

Table 3. The maximal gross and net utiliza-tions for different job-component-size limitsfor the GS policy

maximal utilizationjob-component-size limit gross net

��

��

��

tions produced the values for the maximal gross utilizationas in Table 3; the maximal net utilizations are then com-puted with the ratios between the two types of utilization.These values are in very good agreement with the graphsfor GS in Figure 7. For SC, our simulations gave us a max-imal utilization of �� (cf. the curve for SC in Fig. 3).

5 Conclusions

In this paper we have evaluated the performance ofseveral scheduling policies for co-allocating unordered re-quests in multiclusters with a workload derived from theDAS. The best policy proved to be LS, with a performancecomparable in some cases to using FCFS for total requestsin a single cluster.

Studying the job sizes, we have observed that a smallpercentage of jobs with total sizes close to the size of thesystem can strongly impact on performance. Although thechoice of policy does matter for the performance, whendealing with very large jobs requiring (almost) the entiresystem to run, we found that by far the largest improvementin performance can be obtained from simply limiting thetotal job size.

When choosing the job-component-size limit, careshould be taken that the jobs of the total sizes that occurmost often (in our log �� of the jobs are of size ��!) aresplit up in a way that makes them easy to fit in the system.When both the clusters’ sizes and the most popular total jobsizes are powers of two, a limit that is also a power of twomakes sense.

In multicluster systems we have to deal with a significantamount of processor time spent waiting for the global com-munication. However, co-allocation remains a viable optionwhile the duration of the global communication is coveredby an extension factor of ��.

For some policies, such as LS, a higher amount of co-allocation—going from component size limit of �� to ��—can significantly improve the performance, while for others(LP, GS) it does not. More co-allocation brings along moreflexibility in spreading the jobs over the system, but alsomore global communication. For a policy that can take ad-vantage of it, this flexibility can compensate the disadvan-

tage of slower communication.

References

[1] The Distributed ASCI Supercomputer (DAS).www.cs.vu.nl/das2.

[2] The Global Grid Forum. www.gridforum.org.[3] Globus. www.globus.org.[4] The Portable Batch System. www.openpbs.org.[5] G. Allen, T. Dramlitsch, I. Foster, N. Karonis, M. Ripeanu,

E. Seidel, and B. Toonen. Supporting Efficient Execution inHeterogeneous Distributed Computing Environments withCACTUS and Globus. In Proc. of Supercomputing, 2001.

[6] A. Bucur and D. Epema. The Influence of the Structureand Sizes of Jobs on the Performance of Co-Allocation. InD. Feitelson and L. Rudolph, editors, 6th Workshop on JobScheduling Strategies for Parallel Processing, volume 1911of LNCS, pages 154–173. Springer-Verlag, 2000.

[7] A. Bucur and D. Epema. The Influence of Communicationon the Performance of Co-Allocation. In D. Feitelson andL. Rudolph, editors, 7th Workshop on Job Scheduling Strate-gies for Parallel Processing, volume 2221 of LNCS, pages66–86. Springer-Verlag, 2001.

[8] A. Bucur and D. Epema. Local versus Global Queues withProcessor Co-Allocation in Multicluster Systems. In D. Fei-telson, L. Rudolph, and U. Schwiegelshohn, editors, 8thWorkshop on Job Scheduling Strategies for Parallel Pro-cessing, volume 2537 of LNCS, pages 184–204. Springer-Verlag, 2002.

[9] A. Bucur and D. Epema. The Maximal Utilization of Pro-cessor Co-Allocation in Multicluster Systems. In Proc. Int’lParallel and Distributed Processing Symp. (IPDPS), 2003(to appear; available from www.pds.its.tudelft.nl/ epema).

[10] S.-H. Chiang and M. Vernon. Characteristics of a LargeShared Memory Production Workload. In D. Feitelson andL. Rudolph, editors, 7th Workshop on Job Scheduling Strate-gies for Parallel Processing, volume 2221 of LNCS, pages159–187. Springer-Verlag, 2001.

[11] C. Ernemann, V. Hamscher, U. Schwiegelshohn,R. Yahyapour, and A. Streit. On Advantages of GridComputing for Parallel Job Scheduling. In 2nd IEEE/ACMInt’l Symposium on Cluster Computing and the GRID(CCGrid2002), pages 39–46, 2002.

[12] H.E. Bal et al. The Distributed ASCI SupercomputerProject. ACM Operating Systems Review, 34(4):76–96,2000.

[13] Mesquite Software, Inc. The CSIM18 Simulation Engine,User’s Guide.

[14] R. van Nieuwpoort, J. Maassen, H. Bal, T. Kielmann, andR. Veldema. Wide-Area Parallel Programming Using theRemote Method Invocation Method. Concurrency: Practiceand Experience, 12(8):643–666, 2000.


1082-8907/03 $17.00 © 2003 IEEE

[ieee comput. soc 12th ieee international symposium on high performance distributed computing -...

Documents