scheduling functionally heterogeneous systems with ... · dag scheduling of homogeneous systems...

Scheduling Functionally Heterogeneous Systems with Utilization Balancing

Yuxiong He, Jie Liu

Microsoft Researchyuxhe;[email protected]

Hongyang Sun

Nanyang Technological [email protected]

Abstract—Heterogeneous systems become popular inboth client and cloud. A parallel program can incuroperations on multiple processing resources such as CPU,GPU, and vector processor units. This paper investigatesscheduling problems on functionally heterogeneous systemswith the objective of minimizing the completion time ofparallel jobs.

We first present performance bounds of online schedul-ing and show that any online algorithm is at best around(K + 1)-competitive with respect to job completion time,where K is the total number of resource types. Thereexist “bad” jobs that prevent any online algorithms fromobtaining good interleaving of heterogeneous tasks. Thislower bound suggests that the relative performance ofonline algorithms versus an offline optimal could degradelinearly as types of heterogeneous resources increase.

The limitation of online scheduling motivates our studyof how additional offline or lookahead information canhelp improve scheduling performance. We propose a Multi-Queue Balancing algorithm (MQB) that effectively trans-forms the problem of minimizing completion time to oneof maximizing utilization of heterogeneous resources. Itpromotes interleaving of heterogeneous tasks through bal-ancing the task queues of different types. Our simulationresults suggest that MQB reduces the execution time ofonline greedy algorithms up to 40% over various workloadsand outperforms other offline schemes in most cases.Furthermore, MQB can use limited and approximatedoffline information to improve scheduling decisions.

Keywords-completion time; DAG; functional hetero-geneity; heterogeneous systems; offline; online; parallel;scheduling; utilization balancing;

I. IntroductionScheduling parallel jobs on multiprocessors, an importantarea of research in computer science, has been exten-sively explored on homogeneous resources [4], [6], [9],[17], [23], [27], [31]. However, heterogeneous systemsare becoming popular in both client and cloud withthe increasing popularity of various accelerators, suchas GPU, FPGA, and vector processing units, as wellas better programming support across heterogeneousplatforms [13], [18], [24], [25].

Heterogeneity in computer systems can be charac-terized in two ways. Performance heterogeneity existsin systems that contain general-purpose processors ofdifferent speeds. A task can execute on any processor,but it will execute faster on some than others. Executingparallel programs on processors with different speeds hasbeen studied intensely in scheduling theory [4], [8], [10],

[11], [31].Functional heterogeneity, on the other hand, exists in

systems that contain various types of functional units,and not all tasks of an application can be executed on allfunctional units. A system that mixes different physicalprocessors and co-processors such as CPU, GPU, FPGA,DSP, and vector processing units is functionally hetero-geneous. However, functional heterogeneity is not limitedto hardware resources. For example, security and privacyneeds may restrict some application computations thataccess sensitive data on local machines, while otherscan be performed in a data center. Software licenses canimpose constraints that differentiate the functionality ofmachines as well.

We motivate our problem further by a map-reducestyle parallel data analysis framework, known as Cos-mos [2], that runs on a cluster of tens of thousands ofcommodity servers in the back end of the Bing searchengine from Microsoft. User programs in Cosmos arewritten in Scope [7], a mash-up language that mixesSQL-like queries with native code. The Scope compilertransforms a program, called a job, into a workflow,which is a directed acyclic graph. Each node of the graphrepresents a set of tasks that runs in parallel to performa computation on different parts of the input stream (i.e.,data parallelism). A task, assigned to a server based ondata availability, will read its input over the network ifnot locally available. Task outputs are always written tothe local disk. Each edge that joins two nodes representsa data production and consumption relationship. A jobcan have about 20 nodes on average, and each node caninvolve thousands of servers. In a typical day, Cosmoshandles over a thousand jobs, with a total computationtime over 50 server-year.

We address the scheduling problem for functionallyheterogeneous systems (FHSs) like Cosmos. To minimizethe job completion time, schedulers should assign tasksto machines based on data locations, dependence patternsand cluster-wide resource availability. As a simplifiedabstraction, we assume that the servers are clustered intoclasses based on their data allocation and tasks are notassigned across classes. Thus, we treat server classes asheterogeneous functional units.

In an FHS, an application can perform operations onmultiple processing resources. We model such systemsand jobs as follows. Processors are categorized into K

2011 IEEE International Parallel & Distributed Processing Symposium

1530-2075/11 $26.00 © 2011 IEEE

DOI 10.1109/IPDPS.2011.113

1187


1530-2075/11 $26.00 © 2011 IEEE

DOI 10.1109/IPDPS.2011.113

1187


1530-2075/11 $26.00 © 2011 IEEE

DOI 10.1109/IPDPS.2011.113

1174


1530-2075/11 $26.00 © 2011 IEEE

DOI 10.1109/IPDPS.2011.113

1187


1530-2075/11 $26.00 © 2011 IEEE

DOI 10.1109/IPDPS.2011.113

1187


1530-2075/11 $26.00 © 2011 IEEE

DOI 10.1109/IPDPS.2011.113

1187


1530-2075/11 $26.00 © 2011 IEEE

DOI 10.1109/IPDPS.2011.113

1187


1530-2075/11 $26.00 © 2011 IEEE

DOI 10.1109/IPDPS.2011.113

1187

types. A parallel job can consist of different types oftasks, and each task type can be executed only on thematching type of resources. We model the executionof a parallel job as a directed acyclic graph (DAG).We extended the conventional DAG in prior work onhomogeneous resources [5], [6], [14], [22], [26] to aparallel job with heterogeneous tasks as a K-resourceDAG, or K-DAG in short. We strive to find a scheduleto complete the job as soon as possible.

Although functional heterogeneity is quite commonin both client and cloud, we found little prior workon this problem. Early work on FHSs uses job-shop[3], [16], [23], [31] or DAG-shop scheduling [31] tominimize the completion time of multiple jobs. Neitherof these approaches allows concurrent execution of tasksin the same job. Despite the many prior results onDAG scheduling of homogeneous systems [9], [17], [31],little quantitative research exists on how these schemesperform on functionally heterogeneous ones.

This paper presents scheduling algorithms for func-tionally heterogeneous systems to minimize job com-pletion time. To achieve this goal, key insight is thata scheduler must achieve good interleaving of differenttypes of tasks. In other words, an effective schedulerwants to balance resource utilization to keep all typesof resources as busy as possible. We elaborate theimportance and usage of utilization balancing in bothonline and offline algorithms.

This paper first describes performance and limitationsof online scheduling algorithms on FHS. An onlinescheduler lacks knowledge of future job information suchas remaining work and tasks. Let K represent the totalnumber of resource types and Pα represent the numberof type α resources where 1 ≤ α ≤ K. We show that anyonline algorithm is at best

K + 1−K∑

α=1

1

Pα + 1− 1

Pmax

competitive with respect to job completion time, wherePmax = maxα=1,...,K Pα. This lower bound suggeststhat the relative performance of online algorithms againstan offline optimal could degrade linearly with increasedtypes of heterogeneous resources: the lack of lookaheadinformation can prevent any online algorithm from ef-ficiently interleaving different types of tasks and makethem fail to balance the utilization of heterogeneousresources.

We present a simple online greedy algorithm —KGreedy — that provides the performance guarantee of(K + 1)-competitiveness. This greedy scheduler is closeto the best that any online algorithm can get. This carriestwo messages. On one side, it saves us effort in findingbetter online algorithms, because they can at most leadto marginal performance improvements. For less time-critical applications where (K + 1)-competitiveness ofcompletion time is acceptable, KGreedy offers a simple

approximate. On the other side, for time-critical appli-cations, the limitations of online scheduling motivateour use of offline/lookahead information to improvescheduling performance.

To achieve efficient offline scheduling, we propose aMulti-Queue Balancing algorithm (MQB) that minimizesjob completion time by balancing the task queues ofdifferent types of heterogeneous resources to maximizetask interleaving. It effectively transforms the problemof minimizing completion time to one of balancingand maximizing system utilization. If there are mul-tiple ready tasks, MQB gives priority to those whoseexecution can potentially activate more descendants thatcan use under-utilized types of resources to achievebetter balanced system utilization. We conduct extensivesimulations to evaluate the performance of the onlineKGreedy algorithm with MQB and four other commonoffline heuristics. Experimental results suggest that MQBreduces the execution time of online KGreedy by morethan 40% over various workloads and consistently out-performs other offline schemes. Even with limited andapproximated offline information of a job, MQB canmake use of them to improve scheduling decisions.

The remainder of this paper is organized as follows.Section II formulates the K-DAG scheduling problem,and Section III presents online results. We introduce ourMQB algorithm and describe four other offline algo-rithms in Section IV. Section V presents the experimentalresults of comparing online and offline algorithms forvarious workloads. Section VI explores related work, andwe offer concluding remarks in Section VII.

II. Problem FormulationOur scheduling problem maps a job that consists acollection of tasks with varied types and precedenceconstraints. The processors and tasks are categorized intoK types, and a task can be executed only on a processorwith the matching type. For simplicity, we define theprocessors belonging to category α as α-processors, andthe tasks running on α-processors as α-tasks. For eachcategory α ∈ 1, . . . ,K, there are Pα α-processors inthe system. This section formalizes the job model andpresent the optimization criteria for job completion time.

Job ModelWe model the execution of a parallel job J with hetero-geneous tasks as a K-DAG such that J = (V (J), E(J)),where V (J) and E(J) represent the sets of J’s tasks (ornodes) and edges respectively. A K-DAG can have up toK types of tasks; each α-task v has its work (executiontime), denoted as T1 (v, α), that can be executed onlyon an α-processor. The notation V (J, α) represents theset of α-tasks of the job. The value of α-work T1 (J, α)

indicates the total amount of work of the α-nodes in theK-DAG, i.e., T1 (J, α) =

∑v∈V (J,α) T1 (v, α). Each edge

e ∈ E(J) from nodes u to v represents a dependencybetween the two tasks, regardless of their types. A task

11881188117511881188118811881188

Figure 1. A job represented by a K-DAG with 3 different types oftasks drawn as circles, squares, and triangles respectively.

becomes ready to execute if all of its parents have beencompleted. The precedence relationship u ≺ v holdsif and only if there exists a path from task u to v inE(J). The span — or the critical path length T∞ (J)

— corresponds to the amount of work on the longestchain of the precedence dependencies. We consider bothonline and offline schedulers. An offline scheduler knowsin advance the entire K-DAG of a job. For onlinescheduling, the K-DAG is unfolded dynamically duringthe job’s execution; at any moment, an online schedulerknows only the tasks either in execution or completed,as well as the tasks that are ready. The work of anexecuting or a ready task, however, is unknown to theonline scheduler.

Figure 1 shows an example of a K-DAG J , with K =

3 represented by the shapes of the nodes. Each task hasunit-size work. Let the circular, square and the triangularnodes denote α1-tasks, α2-tasks and α3-tasks of the job,respectively. The work of each type is T1 (J, α1) = 7,T1 (J, α2) = 4 and T1 (J, α3) = 3. The span of the job isT∞ (J) = 7.

Objective Function

Our scheduler uses completion time as the performancemeasure. The completion time T (J) of job J under aschedule is the time at which the schedule completes J’sexecution.

We use competitive analysis to show online sched-uler’s performance relative to an optimal offline algo-rithm. Let T∗(J) denote the completion time produced byan optimal algorithm on job J , and let TA(J) denote thecompletion time produced by a deterministic algorithmA for the same job. Algorithm A is said to be c-competitive in terms of completion time if there existsa constant b such that TA(J) ≤ c · T∗(J) + b holds forany job. Similarly, let

T

R(

J

) denote the completion time

produced by a randomized algorithm R for the samejob. Algorithm R is said to be c-competitive in termsof completion time if there exists a constant b such thatE[TR(J)] ≤ c · T∗(J) + b holds for any job J .

III. Online SchedulingThis section presents lower and upper bounds for onlinescheduling on an FHS. In previous work [20], He, Sunand Hsu showed that any deterministic online algorithmcan be no better than (K + 1 − 1/Pmax)-competitivewith respect to completion time for K-DAG scheduling,where Pmax = maxα=1,...,K Pα. This section shows thatany randomized online algorithm is at best

K + 1−K∑

α=1

1

Pα + 1− 1

Pmax

competitive. In the worst case, these two lower boundssuggest that, the completion time produced by an onlinealgorithm degrades linearly with an increase in types ofheterogeneous resources. We also present in this sectionan online greedy algorithm that provides the performanceguarantee of (K + 1)-competitiveness and discuss itsimplications.

Performance Lower BoundThis section presents a lower bound of randomized onlinealgorithms on completion time. We start from a technicallemma that will be used in the lower bound analysis.

Lemma 1: There are n balls in a non-transparent box.Of the n balls, r are red and n−r are black. At each step,we can get one ball from the box without replacement,and each ball has the same probability of being chosen.The expected number of steps one needs to get all thered balls is r

r+1 (n+ 1).Proof: Given n balls and r red ones, there are

(nr

)possible combinations to arrange the r red balls at n totalpositions, and each combination is equally likely. Let Qdenote the total number of steps we need to get all thered balls for a trial. The expected value of Q is equal tothe average number of steps to get all red balls over allcombinations. The minimum value of Q is r because ittakes at least r steps to get r red balls. The probabilityPr Q = r is 1/

(nr

)because only one combination has all

red balls in the front among(nr

)combinations. Similarly,

we have Pr Q = r + i =(r+i−1

i

)/(nr

)because

(r+i−1i

)combinations give Q = r + i, where the last red ball isat the (r + i)-th position and i black balls are in any ofthe r + i− 1 positions in the front.

Therefore, the expected value of Q is

E [Q] =

n∑i=r

i× Pr Q = i

=

n−r∑i=0

(r + i)× Pr Q = r + i

=

(

r +

n−r

∑i=1

(r + i)×

(

r

+ i

−

1

i

))

/

(

n

r

)

=r

r + 1(n+ 1).

11891189117611891189118911891189

...

...

...

...

... ...

... ...

...

Figure 2. An example of a job instance used in the analysis ofTheorem 2.

We now prove Theorem 2.Theorem 2: The competitive ratio of any randomized

online algorithm cannot be better than

K + 1−K∑

α=1

1

Pα + 1− 1

Pmax

with respect to completion time for K-DAG scheduling,where Pmax = maxα=1,...,K Pα.

Proof: We use Yao’s technique [32] in the analysis.We show a lower bound on the performance ratio ofany deterministic algorithm on a job instance drawnfrom a probabilistic distribution. According to Yao’stheorem, the same lower bound holds for all randomizedalgorithms.

We first describe the probabilistic distribution of jobsthat we use in this proof. An example of such a job isillustrated in Figure 2. Without loss of generality, let usassume PK = Pmax. The job has PαPKm number ofα-tasks for each type α = 1, ...,K, where m is a positiveinteger constant. For α-tasks where α = 1, ...,K − 1, Pα

of them have outgoing edges to all (α+1)-tasks; we callthese active α-tasks. The remaining PαPKm − Pα haveno outgoing edges. The active α-tasks follow uniformrandom distribution on all α-tasks. For K-tasks, mPK−1

of them form a chain, where each task but the last hasone outgoing edge. These tasks are called chain tasks.Among the remaining PKPKm − mPK + 1 K-tasks,PK active ones have outgoing edges to the first of thechain tasks. The remaining K-tasks do not have outgoingedges. Active K-tasks are uniformly distributed amongall K-tasks that are not chain tasks. All the tasks haveunit-size work.

Given a job instance J from the distribution, for α =

2, ...,K, any α-tasks can get ready to run only after allthe active (α − 1)-tasks have completed. The first chaintasks get ready only after all the active K-tasks have

completed. We use Sα to denote the number of stepsfrom the time any α-tasks get ready to the time all activeα-tasks have completed. The job completion time is atleast the summation of Sα for all α = 1, ...,K plus thetime tc taken to complete the chain. So we have

T(J) ≥K∑

α=1

Sα + tc . (1)

To schedule J , an optimal offline scheduler S alwaysexecutes the active tasks as soon as possible. Since 1-tasks do not have any incoming edges, all of them areready at time step 0. An optimal scheduler runs all P1

active 1-tasks at the first time step. Then, all 2-tasks areready at time 1. So, we have S1 = 1. For α = 2, ...,K, allα-tasks get ready at time α−1. Once ready, the scheduleralways executes active tasks first in one step, i.e., Sα = 1.K-tasks, the last type of tasks, can be completed in mPK

time steps by executing one unit of K-task on the criticalpath at each step and executing the remaining K-taskson the PK − 1 other processors. Therefore, the optimalscheduler S produces a completion time of T∗(J ) =

K − 1 +mPK .

We now bound Sα for any online deterministic al-gorithm. We start from α = 1. Denote Q1 as the totalnumber of 1-tasks completed when the last active 1-task is done. Computing the expected value of Q1 isanalogous to computing the expected number of ballsin Lemma 1 with the collection of balls equivalent tothe collection of 1-tasks, i.e., n = P1PKm and withred balls corresponding to active tasks, i.e. r = P1.According to Lemma 1, the expected value of Q1 isE [Q1] = (P1/(P1 + 1))(P1PKm+ 1). Since at each timestep, at most P1 tasks can be processed, we have

E [S1] ≥ E [Q1] /P1 ≥ P1

P1 + 1PKm (2)

Similarly, we can bound S2, ..., SK−1 as E [Sα] ≥(Pα/(Pα + 1))PKm. For K-tasks, since there arePKPKm −mPK + 1 non-chain K-tasks with PK activetasks among them, we have E [SK ] ≥ (PK/(PK +

1))(PK − 1)m. The time tc to complete the chain tasksis at least mPK − 1 since they have a span of mPK − 1.

According to the linearity property of expectation andInequality (2), we can bound the expected time for anydeterministic algorithm to complete J as

E [T(J)]

≥K∑

α=1

E [Sα] + E [tc]

≥

(K + 1−

K∑α=1

1

Pα + 1

)mPK − PK

PK + 1m− 1 .

11901190117711901190119011901190

Thus, the competitive ratio is given by

E [T(J)]

T∗(J)

≥

(K + 1−

∑Kα=1

1Pα+1

)mPK − PK

PK+1m− 1

K − 1 +mPK

=K + 1−

∑Kα=1

1Pα+1 − 1

PK+1 − 1PKm

1 + K−1mPK

. (3)

Let m ≫ K, then K−1mPK

and 1PKm approach 0. Since

PK = Pmax, according to Inequality (3) we have

E [T(J)]

T∗(J)≥ K + 1−

K∑α=1

1

Pα + 1− 1

Pmax + 1(4)

According to Yao’s theorem, the above bound holds forany randomized algorithms.

Theorem 2 gives a performance lower bound for anyonline algorithms that use K-DAG scheduling. Accord-ing to the analysis, some “bad” jobs (such as those inFigure 2) prevent any online algorithms from obtainingoptimal interleaving and balancing different types oftasks. At any moment, a few types of resources arebusy while other types are idle. Since offline optimalalgorithms can balance the utilization of different typesof resources but online algorithms cannot, we see theΩ(K) times of performance difference as shown inInequality (4).

Performance Upper BoundIn homogeneous systems, list scheduling using a greedyalgorithm, one of the most well-known online algorithms,represents the first problem for which competitive anal-ysis was presented in 1966 (by Graham [17]). Given P

identical machines and a DAG job, a greedy schedulerworks as follows: at any time step, if there are morethan P ready tasks, the scheduler executes any P ofthem; if there are at most P ready tasks, it executesthem all. Graham showed that this greedy scheduler was(2− 1/P )−competitive for completion time.

In K-DAG scheduling, we consider a simple extensionto this greedy scheduler that we call KGreedy. KGreedyuses K number of greedy schedulers, where each worksfor one type of resource. For example, when K = 2,KGreedy has two greedy schedulers working for 1-tasksand 2-tasks. At any time step, if there are more than P1

ready 1-tasks, KGreedy executes any P1 of them; if thereare at most P1 ready tasks, it executes them all. KGreedyperforms the same greedy scheduling for type 2-tasks.

It is not hard to show that KGreedy is (K + 1)-competitive for completion time. An extension of Gra-ham’s competitive arguments [17] would serve the pur-pose; refer to Theorem 3 of [20] for a complete proof.This competitive result of KGreedy matches the lowerbound in Theorem 2 when there is a large number ofprocessors for each type of resources. Therefore, thegreedy scheduler performs close to the best that anyonline algorithm can get. It saves us effort in finding

better online algorithms, which can at most lead tomarginal performance improvements in the worst case.For non-time-critical applications that satisfy (K + 1)-competitiveness of completion time, KGreedy offers asimple solution. However, for time-critical ones, thelimitations of online scheduling motivated us to explorewhether additional offline/lookahead information on jobcharacteristics could help schedulers make better deci-sions.

IV. Offline AlgorithmsThis section presents offline scheduling algorithms onan FHS. Since computing optimal completion timefor homogeneous DAG jobs is already NP-hard [15],K-DAG scheduling for minimum completion time isNP-hard. Therefore, our study focuses on developingefficient heuristic algorithms. We now present a newoffline algorithm, called Multi-Queue Balancing (MQB),for scheduling K-DAG. MQB seeks to minimize jobcompletion time by balancing system utilization andimproving interleaving of different types of tasks. Toevaluate MQB performance, we will compare it to onlineKGreedy and four other offline algorithms. Some ofthese algorithms are from well-known heuristics in ho-mogeneous scheduling, while others drive from popularheuristics in job-shop scheduling.

A. Multi-Queue Balancing AlgorithmMQB maintains a set of K ready queues in the system,one for each type of resources. At any time, if an α-processor runs out of work, it selects a task to executefrom the corresponding ready queue, namely the α-queue. MQB gives priority to tasks whose executioncan potentially activate more descendants, using under-utilized resources to achieve better balanced systemutilization.

Two important concepts underpin the MQB algorithm.The first is the notion of balance for the set of readyqueues. Intuitively, we say that a snapshot A of readyqueues has better balance than a snapshot B if theshortest queue of A is larger than that of B: the short-est queue will likely be the bottleneck to maximizingsystem utilization. In the actual algorithm, MQB alsoconsiders the number of processors for each resource.We define the x-utilization metric for the α-queue tobe rα(A) = lα(A)/Pα, where lα(A) is the total work ofready tasks in α-queue at snapshot A, and Pα is the totalnumber of α-processors. MQB compares the queues withthe smallest x-utilization values to decide the balance.More precisely, let πA(·) denote a non-decreasing orderof all queues in terms of the x-utilization value, i.e.,rπA(1) ≤ rπA(2) ≤ · · · ≤ rπA(K). The balance RA ofthe K ready queues at snapshot A is then given by theordered set RA = rπA(1), rπA(2), · · · , rπA(K). We saythat the set of K ready queues at snapshot A has betterbalance than at snapshot B if we have RA > RB in the

11911191117811911191119111911191

lexicographical sense, that is, there exists a j ≤ K suchthat rπA(j) > rπB(j) and rπA(i) = rπB(i) for all 1 ≤ i < j.

The second involves a task’s descendant value. Foreach resource type α, a task in the K-DAG maintainsa descendant value dα approximating its descendantworkload of type α for each 1 ≤ α ≤ K. MQB calculatesthe descendant value recursively, as follows:

dα(v) = 0 if task v has no children, otherwise ,

dα(v) =∑

u∈children(v)(dα(u) + wα(u))/pr(u) ,

where pr(u) represents the number of parents of tasku, and wα(u) is equal to the work of u if u is an α-task and 0 otherwise. In other words, an α-task u withpr(u) parents contributes 1/pr(u) of its own descendantvalue to the descendant value of each of its parents forany resource type, plus an additional 1/pr(u) of its ownwork to dα of each parent.

MQB works as follows. For each type α, when thereare at most Pα ready α-tasks to run at any time, MQBruns them all. When there are more than Pα readyα-tasks, MQB gives priority to tasks whose executioncan potentially activate more descendants to achievebetter balanced utilization. Specifically, MQB assumesthat the work of each ready queue can be increased bythe corresponding descendant value of a ready task. Itchooses the task leading to the best balance when thattask’s descendant values are added to the existing readyqueues. MQB repeats this process until all processorshave been assigned.

B. Other Heuristic AlgorithmsThe four other offline heuristic algorithms, whose perfor-mances we compare to MQB and KGreedy in Section Vinclude. These four algorithms are

• Longest span first (LSpan)• Maximum descendants first (MaxDP)• Different type first (DType)• Shifting bottleneck (ShiftBT)

The first two heuristics, LSpan and MaxDP, well-known for scheduling homogeneous resource, can bedirectly applied to heterogeneous resource schedulingby favoring tasks with the longest span and the maxi-mum descendants, respectively. Specifically, when an α-processor runs out of tasks, it uses the following rules toselect a ready α-task.• LSpan picks an α-task with the longest remaining span.If a task does not have a child, its remaining span is itsremaining work. Otherwise, its span can be computedas the sum of its remaining work and the longest spanamong its children.

• MaxDP picks an α-task with the largest descendant. Atask without children has descendant value 0. A tasku with pr(u) parents contributes 1/pr(u) of its owndescendant and its own work to the descendant of eachof its parents. The descendant calculation for MaxDP

resembles that for MQB. However, MaxDP does notdifferentiate the descendant values of different types.

While LSpan and MaxDP are straightforward exten-sions of the homogeneous resource scheduling heuristicswithout considering the presence of different resourcetypes, the last two heuristics, DType and ShiftBT, doconsider the resource heterogeneity. Specifically, theyselect a ready α-tasks using these rules:

• DType picks an α-task with the smallest different-child distance, where a task’s different-child distance isthe shortest distance to any descendant with a differenttype. The DType thus prioritize tasks that are parentsor ancestors of tasks of other types.

• ShiftBT, an extension of the well-known shifting bot-tleneck heuristic [1] in job-shop scheduling, works asfollows. For each resource type α, assuming all othertypes of resources have infinite number of processors,ShiftBT finds a schedule that tries to minimize themaximum lateness of all tasks. Lateness of a taskmeasures the difference between its actual completiontime and due date; the due date, the latest time to starta task without delaying other tasks, is computed as thetotal span of the job minus the remaining span of thetask. The larger the lateness, the worse a task coulddelay the execution of other tasks. In order to find aschedule that minimizes the maximum lateness, ShiftBTuses a heuristic that always gives priority to tasks withearlier due date. For a resource type α, let’s use Lα

to represent the maximum lateness value of all tasksin the schedule produced by earliest due date heuristic.ShiftBT finds the resource type k that maximizes thelateness value, i.e., k = argmax Lα|α = 1..K. Thisresource, considered to be the biggest bottleneck re-source, gets priority to be scheduled first. ShiftBT thenrepeats this process with the remaining resource typesuntil all types have been considered.

All scheduling algorithms described in this paper canwork in either non-preemptive or preemptive mode. Anon-preemptive scheduler decides task allocation when aprocessor is idle, and the task runs through completion onits assigned processor without preemption. A preemptivescheduler makes decisions for each processor at thebeginning of every scheduling quantum, and a task can bepreempted at one processor and reallocated to another.

V. Experimental EvaluationWe conducted the following experiments to evaluate theperformance of our six scheduling algorithms (KGreedy,LSpan, DType, MaxDP, ShiftBT, and MQB):

• Algorithm performance experiments to comparethe performance of the six algorithms over differentworkloads

• Changing K experiments to investigate the impactof changing K values from 1 to 6, where K repre-sent the number of different resource types

11921192117911921192119211921192

• Skewed load experiments to study the impact ofbiased workloads where some resources are moreheavily loaded than others

• Preemptive scheduling experiments to compareperformance of non-preemptive algorithms to theirpreemptive counterparts

• Approximated information experiments to evaluatethe influence of partial and imprecise job informa-tion on scheduling

We first discuss our simulation setup (Section V-A)and workload (Section V-B) before presenting experi-mental results (from Section V-C to Section V-G).

A. Simulation SetupWe built a discrete-time simulator using C# to evaluatethe scheduling algorithms. Composed of resources, jobs,schedulers, our simulator simulates the interactions oftasks on the resources. For preemptive scheduling, weignore the overhead caused by processor reallocation.Our experiments assumes non-preemptive scheduling asdefault except for preemptive scheduling experiments.We use a default number of different resource typesK = 4 except for changing K experiments. In all figuresrepresenting experimental results, each per bar/point plotwas collected from 5000 instances of jobs.

Given a job J and an algorithm A, we are interested inthe completion time generated by algorithm A comparingto an offline optimal. Since obtaining an offline optimalis NP-hard, we compare to the lower bound instead. Thelower bound we use is

L(J) = max

(T∞ (J) , max

1≤α≤K(T1 (J, α) /Pα)

).

We call this performance metric the completion time ra-tio, which is calculated as the ratio of the job completiontime produced by algorithm A to the lower bound, i.e.,T (J)/L(J).

B. Workloads and Resource ConfigurationOur experiments use three types of workloads — em-barrassingly parallel jobs, trees, and iterative reductionjobs. Each represents a class of parallel programmingapplications.

Embarrassingly Parallel Workload : An embarrass-ingly parallel (EP) workload is one for which little or noeffort is required to separate the problem into a numberof parallel branches, and no dependency exists betweenthose parallel branches. Each branch is represented asa chain of tasks. Monte Carlo calculations and otherforms of statistical simulation offer the cleanest exampleof an EP job, but many applications in physics, imageprocessing, bioinformatics, etc., also fall into this gen-eral category, In a heterogeneous environment, differentphases of an EP branch can be executed on differentresource types.

Figure 3(a) shows an example of an EP DAG. Wefurther divided them into two categories: (1) layered EP

where each branch includes a fixed sequence of taskswith type from 1 to K, and (2) random EP where thetype of each task in any branch is decided uniformly atrandom. To obtain an EP workload, we varied the numberof branches, the number of tasks in each branch, and thework and type of each task.

Tree workload : A tree workload starts from a roottask and explores parallelism at each task by solvingthe task using parallel execution of subtasks. A divide-and-conquer parallel program with trivial conquer phasesrepresents the structure of this workload. Many usefulapplications — such as search, graph traversal, andapplications applying speculated parallelism — fall intothis category.

Figure 3(b) shows an example of a tree DAG. Wefurther divided the tree workload into two categories: (1)layered trees where all the nodes at each level of a treehave the same type, and (2) random trees where the typeof each task in a tree is decided uniformly at random. Atree workload involves the fanout number m and fanoutprobability p of any node, i.e., a node has probabilityp of having m direct children and probability 1 − p ofhaving no children. To obtain a tree workload, we varythe fanout number, fanout probability, and the work ofeach task.

Iterative Reduction Workload : An iterative reduction(IR) workload resembles MapReduce [12] workload withmultiple iterations of map and reduce operations. Themap phase takes inputs, divides them into smaller sub-problems and runs them in parallel. All map tasks in eachiteration are independent of one another. A reduce phasecombines the answers of all the sub-problems to get theoutput. There can be many reduce tasks with each onecomputing the final answers over a sub-range of finalresults; each reduce task can depend on results from aset of map tasks. Useful in both distributed and parallelsystems, MapReduce can be applied to a wide range ofapplications [28] such as distributed sort, inverted indexconstruction, document clustering, machine learning, etc.

Figure 3(c) shows an example of an IR DAG. Weconsider the case where a reduce task depends on asubset of all map tasks. Each MapReduce iteration hasmap tasks with different fanouts. Tasks with a highfanout have a higher probability of providing outputto each reduce task. Similarly, some reduce tasks havedifferent fanins. To obtain an IR workload, we variedthe probability values, the total number of tasks at eachphase, and the work of each task. We also divided thisworkload into two categories: (1) layered IR where allnodes at each iteration of the IR computation have thesame type, and (2) random IR where the type of eachtask is decided uniformly at random.

Resource Configuration : We test the workloads onsmall- and medium-size systems. Our small system had1 − 5 resources per type; if K = 4, the system wascomposed of 4 − 20 resources in total. Our mediumsystem had 10− 20 resources per type; if K = 4, it was

11931193118011931193119311931193

(a) EP Random (b) Layered Tree (c) Layered IR

Figure 3. Example K-DAG of EP, tree and IR workloads.

0

0.5

1

1.5

2

2.5

3

3.5

4

KGreedy

LSpan

DType

MaxDP

ShiftBT

MQB

Ave completion time ratio

(a) Small Random EP

0

0.5

1

1.5

2

2.5

3

3.5

4

KGreedy

LSpan

DType

MaxDP

ShiftBT

MQB


(b) Medium Random Tree

0

0.5

1

1.5

2

2.5

3

3.5

4

KGreedy

LSpan

DType

MaxDP

ShiftBT

MQB


(c) Medium Random IR

0

0.5

1

1.5

2

2.5

3

3.5

4

KGreedy

LSpan

DType

MaxDP

ShiftBT

MQB

Completion time ratio

(d) Small Layered EP

0

0.5

1

1.5

2

2.5

3

3.5

4

KGreedy

LSpan

DType

MaxDP

ShiftBT

MQB


(e) Medium Layered Tree

0

0.5

1

1.5

2

2.5

3

3.5

4

KGreedy

LSpan

DType

MaxDP

ShiftBT

MQB


(f) Medium Layered IR

Figure 4. Algorithms performance on various workload.

composed of 40−80 resources. Each workload we presentin this section combines application workload and ma-chine configurations. For example, “medium layered IR”represents a workload running layered IR K-DAG onmedium systems.

C. Algorithm Performance ExperimentsThese experiments compare the completion time ratioof the six algorithms on EP, Tree and IR workloads.Figure 4 shows the results.

Figures 4(a), 4(b) and 4(c) present algorithm perfor-mance on the three randomized workloads — randomEP, random tree, and random IR. The average completiontime ratios of the corresponding workloads are close to1. This suggests that KGreedy performs close to thelower bound, close to optimal, and comparably with otheroffline approaches. These results match our intuition:the lack of structural information makes it difficult fora scheduling algorithm to take advantage of a randommodel. Therefore, any “best-effort“ algorithm would

work just fine, and a simple greedy online algorithm issufficient for such workloads.

Looking at more structured workloads like layered EP,layered tree and layered IR (Figures 4(d), 4(e) and 4(f)),appropriate offline information offers considerable im-provements over KGreedy. Moreover, MQB outperformsKGreedy by reducing completion time ratio by at least40% in all layered workloads. Moreover, in practice,structured programs occur more often because differenttasks and stages could require or better suit differenttypes of resources.

The following discussion about structured workloadspresents the comparison results:• Some offline information helps greatly for simpleworkloads. For example, all five offline heuristics re-duce the average completion ratio of layered trees byhalf compared to online KGreedy. It is relatively easyin the tree workload to identify tasks whose executionleads to better balancing on heterogeneous resources.These tasks have more descendants, longer remaining

11941194118111941194119411941194

1

1.5

2

2.5

3

3.5

4

4.5

1 2 3 4 5 6


K

KGreedyLSpanDTypeMaxDP

ShiftBTMQB

(a) Small Layered EP

1

1.2

1.4

1.6

1.8

2

2.2

2.4

1 2 3 4 5 6


K


ShiftBTMQB

(b) Medium Layered Tree

1

1.5

2

2.5

3

3.5

4

4.5

5

5.5

6

1 2 3 4 5 6


K


ShiftBTMQB

(c) Medium Layered IR

Figure 5. Performance comparison when varying the total types of resources K from 1 to 6.

spans, and smaller distances to other type of resources.An algorithm capturing any of this information workswell.

• A good choice of offline heuristics can be applicationspecific. For example, MaxDP performs well in layeredtree and IR workloads, but performs poorly in layeredEP ones: for EP jobs, balancing resource utilizationdepends mostly on the type distribution of descendants,not their number. However, this information is not cap-tured by MaxDP. Similarly, LSpan works well for treeworkloads but not for others. An algorithm performingwell consistently would be highly desirable.

• A well-known and effective heuristic for job-shopscheduling ShiftBT’s performance varies in K-DAG. Inparticular, it works well in EP and trees but not in IR.This is related to the optimality of the subcomputationson minimizing the maximum lateness. With a chainor a tree of tasks, the earliest due date heuristic thatShiftBT uses to minimize the maximum lateness leadsto good solutions. However, there is no effective way tominimize lateness for more general K-DAG. When thequality of the subcomputation is compromised, ShiftBTcannot perform well consistently.

• MQB works well in most of our workloads. Comparedto KGreedy and other offline heuristics, it performsalways the best, or almost the best. Intuitively, ittries to balance the workload among all resources bybalancing the queue sizes. It minimizes completiontime by maximizing system utilization over differentresource types and effectively transforms a problemof minimizing completion time into one of balancingsystem utilization.

D. Changing K ExperimentsThese experiments investigate the impact of changing thenumber of resource types K from 1 to 6. Figure 5 showsthe results. As indicated by the worst-case performancebound in Theorem 2, increasing K linearly degradesthe competitiveness of any online algorithm comparedto an offline optimal. The competitive ratio of KGreedyshown in Figure 5, although obtained in average sense,nevertheless grows as K increases. The increase ofaverage competitiveness, however, is not always linearin K because Theorem 2 is a worst-case performance

bound.Offline information helps to reduce the performance

degradation of increased K, and yields results closerto optimal even when K is large. In layered tree (Fig-ure 5(b)), all offline algorithms perform very close to op-timal at any K value. For layered EP (Figure 5(a)), MQBperforms close to optimal. In layered IR (Figure 5(c)),although no algorithms perform close to the lower bound,MQB and MaxDP consistently reduce the execution timeof a program produced by KGreedy by around half forany K ≥ 2.

E. Skewed Load ExperimentsThese experiments study workloads where some re-sources are more heavily loaded than others. To quantifythe resource load, we use a measure called work-per-processor ratio. For a job’s type α resource, its α-work-per-processor ratio is computed as the α-work divided bythe number of α-processors assigned to the job. When ajob has similar work-per-processor ratios for each typeof resource, its load is considered to be well balanced.Otherwise, the larger the variance, the more skewed theload.

Figure 6 shows the influence of a skewed load. Usingthe same jobs as those tested in Figures 4(e) and 4(f), weincreased the skew by reducing the number of machinesfor type 1 resources to 1/5 of the original and keeping theother machines unchanged. From Figures 6(a) and 6(b),we can see that the difference among algorithms shrinks,and KGreedy performs closer to optimal. When a job’sload is skewed, one or a few resource types becomethe bottleneck. This situation resembles a homogeneouscase with the difference between algorithms becomingsmaller.

Although the scheduling issue is of less concern forskewed load, such loads incur unavoidable waste on re-sources with small work-per-processor ratios. Assigningproper numbers of resources to a job is an aspect ofscheduling decisions that goes beyond the scope of thiswork. However, on an FHS, achieving high resourceutilization depends upon having balanced load for eachjob.

11951195118211951195119511951195

0

0.2

0.4

0.6

0.8

1

1.2

1.4

KGreedy

LSpan

DType

MaxDP

ShiftBT

MQB


(a) Medium Layered Tree

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

KGreedy

LSpan

DType

MaxDP

ShiftBT

MQB


(b) Medium Layered IR

Figure 6. Impact of scheduling algorithms on jobs with skewed load.

0

0.5

1

1.5

2

2.5

3

3.5

4

KGreedy

LSpan

DType

MaxDp

ShiftBt

MQB

Ave completion time ratio Non-Preemptive

Preemptive


0

0.5

1

1.5

2

2.5

3

3.5

4

KGreedy

LSpan

DType

MaxDp

ShiftBt

MQB


Preemptive


0

0.5

1

1.5

2

2.5

3

3.5

4

KGreedy

LSpan

DType

MaxDp

ShiftBt

MQB


Preemptive


Figure 7. Comparison of non-preemptive and preemptive scheduling.

F. Preemptive Scheduling ExperimentsPrevious experiments used non-preemptive scheduling;we now evaluate the impact of preemptive scheduling.Figure 7 compares the performance of both versions ofthe algorithms. Results suggest that, in many cases, thepreemptive version performs comparably with or betterthan its corresponding non-preemptive one: preemptionsoffer earlier chances to correct bad scheduling decisions.However, preemption does not solve the performancedegradation of online scheduling on heterogeneous sys-tems since the average completion ratio of preemptiveKGreedy still greatly exceeds that of good offline algo-rithms.

G. Approximated Information ExperimentsOur previous experiments suggest that offline infor-mation helps shorten job completion time. Althoughcomplete information of a job’s K-DAG may not beavailable during online execution, a job’s previous ex-ecution history, statistics and internal characteristics canprovide us an approximation on its future behavior. Forexample, for some data-intensive tasks, such as thosein MapReduce jobs, task processing time scales linearlywith the amount of data to be processed. One can collectthe scale factor of a task from its previous executionand estimate the amount of data to be processed ina new run to predict task’s total processing time. De-tailed techniques to obtain job information through staticanalysis or dynamic prediction exceed scope of thiswork. We consider the case where estimates are availableand investigate how an offline algorithm performs withapproximated job information. Since MQB offers the

best overall performance, we now show how MQB withpartial and imprecise job information performs comparedto KGreedy.

We categorize the approximation of job informationinto two types: (1) partial information, where one canhave only limited amount of lookahead into future, and(2) imprecise information, where the approximation hasnoise and uncertainty. For partial information, we studya restricted case of MQB, called MQB+1Step, that has asingle step of lookahead. To schedule a task, MQB+1Stepconsiders only its immediate children. In comparison,the original MQB algorithm uses information of all taskdescendants, and we call it MQB+All to differentiate.

Imprecise information may come from differentsources and appear in various forms such as impreciseworkload calibration, user inputs, compiler outputs, etc.While it is not feasible to examine them all, we con-sider here two simple approximations: MQB+Exp withstochastic uncertainly and MQB+Noise with the interfer-ence of noise. In MQB+Exp, the descendant value of atask is a random value following exponential distributionwith a mean equal to the true value. In MQB+Noise, thedescendant value of a task is its true value interferedby a noise with a multiplicative and an additional term.The multiplicative term is a uniformly distributed randomvalue from 0.5 to 1.5, and the additional term is auniformly distributed random value from 0 to the averagework of the task.

Combining the two cases (All/1Step) of completeor partial information with the three cases (Pre-cise/Exp/Noise) of precise or imprecise information, we

11961196118311961196119611961196

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

KGreedy

MQB+All+Pre

MQB+All+EXP

MQB+All+Noise

MQB+1Step+Pre

MQB+1Step+EXP

MQB+1Step+Noise

Completion Time Ratio

averagemax


0

0.5

1

1.5

2

2.5

3

3.5

KGreedy

MQB+All+Pre

MQB+All+EXP

MQB+All+Noise

MQB+1Step+Pre

MQB+1Step+EXP

MQB+1Step+Noise


averagemax


0

0.5

1

1.5

2

2.5

3

3.5

4

KGreedy

MQB+All+Pre

MQB+All+EXP

MQB+All+Noise

MQB+1Step+Pre

MQB+1Step+EXP

MQB+1Step+Noise


averagemax


Figure 8. Comparison of KGreedy and MQB with approximated information.

have six cases of MQB. Figure 8 compares average andmaximum completion ratios of KGreedy to these sixcases of MQB. Our comparison yields these observa-tions:

• Partial information. For some workloads, a schedulerneeds only a small amount of lookahead informationto obtain substantial performance improvement. Theresults of MQB+All and MQB+1Step are similar forboth trees and IR, which suggests that one-step looka-head offers sufficient information for these two work-loads. This matches our intuition since in both cases agood scheduler should give priority to tasks with morechildren. However, the performance of MQB+1Step isworse than MQB+All at EP since a good schedule of EPcan be decided only with more global job information.

• Imprecise information. Even with imprecise pre-dicted or estimated information for a job’s future, afair estimate can still help improve scheduling. Forexample, in all workloads, both MQB+All+Exp andMQB+All+Noise offer performance benefits comparedto KGreedy. Even with one-step lookahead and whenapproximated information can be two times off the truevalue, MQB still outperforms KGreedy, with 20%-30%improvements in both tree and IR workloads.

VI. Related WorkWe start from classical results on scheduling homoge-neous resources. The famous greedy scheduling algo-rithm by Graham [17] guarantees that the completiontime of a DAG is no more than 2 − 1/P times that ofan optimal scheduler, where P is the total number ofprocessors. Shmoys, Wein and Williamson [31] showeda matching lower bound of 2 − 1/P on the competitiveratio of any deterministic online algorithm, and thisresult holds true even if preemption is allowed. Theyalso showed a lower bound of 2 − O(1/

√P ) for any

non-preemptive randomized online algorithm. For offlinescheduling, the longest span first (LSpan) algorithmhas been shown to have an approximation ratio of2 − 1/(P − 1) for P ≥ 3 [9]. In the special case wherethe DAG is an out-tree, Hu [21] showed that LSpanguarantees the optimal schedule. In K-DAG scheduling,however, one can find simple counter-examples to show

that LSpan is no longer optimal for out-trees.Many analytical results on functionally heterogeneous

systems have been obtained using the job-shop schedul-ing model [3], [16], [23], [31]. This model has multiplejobs, each consisting of a chain of heterogeneous tasks,and there is only one machine from each resource type.For the non-preemptive setting, the best known approxi-mation ratio is O(log2 µP/ log2 log µP ) [16], [31], whereµ is the maximum number of tasks in a job; the bestratio for the preemptive setting is O(logP/ log logP ) [3].Shmoys et al. [30] generalized job-shop scheduling toDAG-shop scheduling, where the precedence constraintsof each job are represented by a DAG instead of achain, and there are multiple processors for each resourcetype. However, no two tasks from the same job can beexecuted simultaneously. Similar results [16], [31] havebeen derived for this model. Neither job-shop nor DAG-shop scheduling allows concurrent execution of tasks inthe same job.

He, Sun and Hsu [20] studied the scheduling ofheterogeneous resources under the K-DAG model, whichgeneralizes DAG-shop scheduling by allowing concur-rent executions of tasks from the same type. They showedthat the completion time of a K-DAG scheduled by anonline greedy algorithm is (K+1−1/Pmax)-competitive,which matches a lower bound in the deterministic setting.Similar results have been observed in [29] under a dif-ferent job model. This paper showed that randomizationis of little help in improving the performances of onlinescheduling algorithms.

Besides theoretical results, some empirical work hasstudied different heuristics for scheduling functionallyheterogeneous resources [1], [19]. In particular, shiftingbottleneck, a popular heuristic [1], has been shown tohave good performance in practice for job-shop schedul-ing [27]. Hamidzadeh, Lilja and Atif [19] consideredboth computation and communication costs of assigninga processor to a task and developed a dynamic schedulingheuristic for heterogeneous computing platforms.

VII. Conclusion RemarksThis paper described some limitations of online schedul-ing on functionally heterogeneous systems and proposed

11971197118411971197119711971197

an efficient algorithm — MQB that uses additionaloffline information to achieve utilization balancing andminimize completion time of K-DAG jobs.

Scheduling of functionally heterogeneous systems canbe extended to a more general context. In a K-DAGmodel, each task can be executed only on its matchingtype of processors, which is similar to the case thata compiled binary of a task can only be executed onits matching architecture. Just-In-Time (JIT) compilationbrings an interesting new dimension to this problem.With the support of JIT, a task can be compiled todifferent binaries at run time and flexibly executed ondifferent types of resources. Here, a scheduler requiresadditional functionality and must choose appropriateresource types to compile the task for and execute it. Howto schedule this more flexible job model on functionallyheterogeneous systems remains an interesting open prob-lem.

AcknowledgementWe thank Jim Larus, Burton Smith, Dennis Crain, Xeno-fon D. Koutsoukos, Wen-Jing Hsu and Paul Waterson forhelpful discussions. We also thank anonymous reviewersfor their valuable comments.

References[1] J. Adams, E. Balas, and D. Zawack. The shifting bot-

tleneck procedure for job shop scheduling. ManagementScience, 34(3):391–401, 1988.

[2] G. Ananthanarayanan, S. Kandula, A. Greenberg, I. Sto-ica, Y. Lu, B. Saha, and E. Harris. Reining in the outliersin map-reduce clusters using mantri. In OSDI, October2010.

[3] N. Bansal, T. Kimbrel, and M. Sviridenko. Job shopscheduling with unit processing times. Mathematics ofOperations Research, 31(2):381–389, 2006.

[4] M. A. Bender and M. O. Rabin. Scheduling Cilk multi-threaded computations on processors of different speeds.In SPAA, pages 13–21, July 2000.

[5] G. E. Blelloch and J. Greiner. A provable time and spaceefficient implementation of NESL. In ICFP, pages 213–225, 1996.

[6] R. D. Blumofe and C. E. Leiserson. Space-efficientscheduling of multithreaded computations. SIAM Journalon Computing, 27(1):202–229, 1998.

[7] R. Chaiken, B. Jenkins, P. Larson, B. Ramsey, D. Shakib,S. Weaver, and J. Zhou. Scope: Easy and efficient parallelprocessing of massive datasets. In VLDB 2008, August2008.

[8] C. Chekuri and M. Bender. An efficient approximationalgorithm for minimizing makespan on uniformly relatedmachines. Journal of Algorithms, 41(2):212–224, 2001.

[9] N. F. Chen and C. L. Liu. On a class of schedulingalgorithms for multiprocessor computing systems. In InSCCPP, pages 1–16, 1975.

[10] F. A. Chudak and D. B. Shmoys. Approximation algo-rithms for precedence-constrained scheduling problemson parallel machines that run at different speeds. InSODA, pages 581–590, Philadelphia, PA, USA, 1997.

[11] E. Davis and J. M. Jaffe. Algorithms for scheduling taskson unrelated processors. Journal of ACM, 28(4):721–736,1981.

[12] J. Dean and S. Ghemawat. Mapreduce: simplified dataprocessing on large clusters. In OSDI, 2004.

[13] T. Endo and S. Matsuoka. Massive supercomputingcoping with heterogeneity of modern accelerators. InIPDPS, pages 1–10, 2008.

[14] Z. Fang, P. Tang, P.-C. Yew, and C.-Q. Zhu. Dynamic pro-cessor self-scheduling for general parallel nested loops.IEEE Transactions on Computers, 39(7):919–929, 1990.

[15] M. R. Garey and D. S. Johnson. Computers and In-tractability: A Guide to the Theory of NP-Completeness.W. H. Freeman & Co., 1979.

[16] Goldberg, Paterson, Srinivasan, and Sweedyk. Betterapproximation guarantees for job-shop scheduling. InSODA, 1997.

[17] R. L. Graham. Bounds on multiprocessing anomalies.SIAM Journal on Applied Mathematics, pages 17(2):416–429, 1969.

[18] M. Hall, Y. Gil, and R. Lucas. Self-configuring appli-cations for heterogeneous systems: Program compositionand optimization using cognitive techniques. Proceedingsof the IEEE, 96(5):849–862, 2008.

[19] B. Hamidzadeh, D. J. Lilja, and Y. Atif. Dynamicscheduling techniques for heterogeneous computing sys-tems. Concurrency: Practice and Experience, 7(7):633–652, 1995.

[20] Y. He, H. Sun, and W.-J. Hsu. Adaptive scheduling ofparallel jobs on functionally heterogeneous resources. InICPP, page 43, 2007.

[21] T. Hu. Parallel sequencing and assembly line problems.Operations Research, 9(6):841–848, 1961.

[22] S. F. Hummel and E. Schonberg. Low-overhead schedul-ing of nested parallelism. IBM Journal of Research andDevelopment, 35(5-6):743–765, 1991.

[23] E. L. Lawler, J. K. Lenstra, A. H. G. Rinnooy Kan, andD. B. Shmoys. Sequencing and scheduling: Algorithmsand complexity. Technical report, Centre for Mathematicsand Computer Science, 1989.

[24] M. D. Linderman, J. Balfour, T. H. Meng, and W. J.Dally. Embracing heterogeneity — parallel programmingfor changing hardware. In HotPar, pages 1–6, 2009.

[25] M. D. Linderman, J. D. Collins, H. Wang, and T. H.Meng. Merge: a programming model for heterogeneousmulti-core systems. SIGOPS Operating Systems Review,42(2):287–296, 2008.

[26] G. J. Narlikar and G. E. Blelloch. Space-efficientscheduling of nested parallelism. ACM Transactions onProgramming Languages and Systems, 21(1):138–173,1999.

[27] M. L. Pinedo. Scheduling: Theory, Algorithms, andSystems. Springer (Third edition), 2008.

[28] C. Ranger, R. Raghuraman, A. Penmetsa, G. Bradski, andC. Kozyrakis. Evaluating mapreduce for multi-core andmultiprocessor systems. In HPCA, pages 13–24, 2007.

[29] H. Shachnai and J. J. Turek. Multiresource malleabletask scheduling to minimize response time. InformationProcessing Letters, 70(1):47–52, 1999.

[30] Shmoys, Stein, and Wein. Improved approximation algo-rithms for shop scheduling problems. In SODA, 1991.

[31] D. B. Shmoys, J. Wein, and D. P. Williamson. Schedulingparallel machines online. In FOCS, pages 131–140, 1991.

[32] A. Yao. Probabilistic computations: Toward a unifiedmeasure of complexity. In FOCS, 1977.

11981198118511981198119811981198

scheduling functionally heterogeneous systems with ... · dag scheduling of homogeneous systems...

Documents