optimal scheduling of file transfers with divisible sizes on multiple disjoint paths

Optimal Scheduling of File Optimal Scheduling of File Transfers with Divisible Sizes on Transfers with Divisible Sizes on

Multiple Disjoint PathsMultiple Disjoint Paths

Mugurel Ionut AndreicaMugurel Ionut AndreicaPolytechnic University of BucharestPolytechnic University of Bucharest

Computer Science DepartmentComputer Science Department

Optimal Scheduling of File Transfers with Divisible Sizes on Multiple Disjoint PathsOptimal Scheduling of File Transfers with Divisible Sizes on Multiple Disjoint Paths 22

SummarySummary

Motivation – Scheduling File TransfersMotivation – Scheduling File Transfers Offline SchedulingOffline Scheduling

maximum profit scheduling (files with divisible maximum profit scheduling (files with divisible sizes) – heuristic algorithmsizes) – heuristic algorithm

minimum cost scheduling – optimal algorithmminimum cost scheduling – optimal algorithm

Online Resource ManagementOnline Resource Management algorithmic framework for the block partitioning algorithmic framework for the block partitioning

method (=data structure)method (=data structure)


MotivationMotivation

world-wide development and world-wide development and deployment of distributed systems, deployment of distributed systems, services and applicationsservices and applications

competition for bottleneck resources competition for bottleneck resources => poor performance=> poor performance

inefficient usage of available inefficient usage of available resources => poor performanceresources => poor performance

scheduling techniques => scheduling techniques => performance improvementsperformance improvements


Maximum Profit SchedulingMaximum Profit Scheduling

n files (divisible sizes)n files (divisible sizes) szszii – size of the i – size of the ithth file file szsz11≤sz≤sz22 ≤... ≤ sz ≤... ≤ sznn szszii=q=qii··szszi-1i-1, q, qii≥1 is an integer≥1 is an integer ppii=profit of the i=profit of the ithth file (transfer request) file (transfer request)

k pathsk paths path j – available in the interval [0,Tpath j – available in the interval [0,Tjj]] unit speedunit speed

schedule the requests non-preemptively + schedule the requests non-preemptively + at most one request per path per time unitat most one request per path per time unit

equivalent to multiple knapsack with equivalent to multiple knapsack with divisible item sizesdivisible item sizes


Multiple Knapsack with Multiple Knapsack with Divisible Item SizesDivisible Item Sizes

NP-hardNP-hard can be solved in can be solved in O(n·max{TO(n·max{Tjj}}kk)) (extension from (extension from

single knapsack problem)single knapsack problem) heuristicsheuristics

• Greedy1MultipleKnapsack(n,k)Greedy1MultipleKnapsack(n,k) pack the items optimally in the first knapsackpack the items optimally in the first knapsack remove the q packed items and the knapsackremove the q packed items and the knapsack call Greedy1MultipleKnapsack(n-q,k-1) with the call Greedy1MultipleKnapsack(n-q,k-1) with the

remaining items and knapsacksremaining items and knapsacks• Greedy2Greedy2

sort the items according to some criterion (e.g. sort the items according to some criterion (e.g. Profit/size)Profit/size)

insert the items into knapsack using the First Fit insert the items into knapsack using the First Fit heuristicheuristic

new heuristic algorithm – this papernew heuristic algorithm – this paper


New Heuristic Algorithm (1/2)New Heuristic Algorithm (1/2) (Multiple Knapsack with (Multiple Knapsack with

Divisible Item Sizes)Divisible Item Sizes) split the items into groupssplit the items into groups

all the items in the same group (i) have the same size all the items in the same group (i) have the same size (sg(sgii))

sgsg11>sg>sg22>...>sg>...>sgGG (G=total # of groups) (G=total # of groups) within a group, sort the items in decreasing order within a group, sort the items in decreasing order

of their profitsof their profits prpri,1i,1≥pr≥pri,2i,2≥…≥pr≥…≥pri,nii,ni (n (nii=the # of items in group i)=the # of items in group i)

insert the items into knapsacks using the insert the items into knapsacks using the First FitFirst Fit heuristicheuristic

traverse the items in inreasing order of group number; traverse the items in inreasing order of group number; within a group, in increasing order of item numberwithin a group, in increasing order of item number

choice of knapsack for an item (i,j) – not important => choice of knapsack for an item (i,j) – not important => because of divisible item sizes, we insert the same set because of divisible item sizes, we insert the same set of items of items


New Heuristic Algorithm (2/2)New Heuristic Algorithm (2/2) (Multiple Knapsack with (Multiple Knapsack with

Divisible Item Sizes)Divisible Item Sizes) repeatedly improve the initial solutionrepeatedly improve the initial solution

replace an item (i,j) in the knapsack with a replace an item (i,j) in the knapsack with a subset Q of items outside the knapsack, s.t. subset Q of items outside the knapsack, s.t. pr(i,j)<profit(Q) && profit(Q)-pr(i,j)=maximumpr(i,j)<profit(Q) && profit(Q)-pr(i,j)=maximum

ignore item (i,j) from now onignore item (i,j) from now on

time complexitytime complexity O(n·S·min{n,S·log(S)})O(n·S·min{n,S·log(S)}) S=maximum size of an itemS=maximum size of an item


Performance EvaluationPerformance Evaluation(Multiple Knapsack with (Multiple Knapsack with

Divisible Item Sizes)Divisible Item Sizes) tested algorithmstested algorithms

new heuristicnew heuristic Greedy1MultipleKnapsackGreedy1MultipleKnapsack Greedy2 (with multiple criteria)Greedy2 (with multiple criteria) direct solution (extension from single knapsack)direct solution (extension from single knapsack)

resultsresults most of the cases – solved optimally by the new most of the cases – solved optimally by the new

heuristicheuristic in terms of quality of solution and running time in terms of quality of solution and running time

=> the new heuristic = clear winner=> the new heuristic = clear winner• followed by Greedy1MultipleKnapsackfollowed by Greedy1MultipleKnapsack


Minimum Cost Scheduling Minimum Cost Scheduling (1/3)(1/3)

sequence of n identical filessequence of n identical files to be sent consecutively from the same source to be sent consecutively from the same source

to the same destinationto the same destination sending time per file=1 time unitsending time per file=1 time unit

k data transfer providersk data transfer providers provider jprovider j

• cost: Ccost: Cj j per time unitper time unit• lease at most ONE time interval of duration at most lease at most ONE time interval of duration at most

TTmax,imax,i which includes [T which includes [T1,j1,j, T, T2,j2,j]]

default network link: cost Ldefault network link: cost L ii for transferring for transferring the ithe ithth file file

minimize the total costminimize the total cost



O(nO(n··k) dynamic programmingk) dynamic programming sort the data transfer providers in increasing sort the data transfer providers in increasing

order of Torder of T2,j2,j: : TT2,12,1≤T≤T2,22,2≤…≤T≤…≤T2,n2,n Cmin[i,j]Cmin[i,j]=the minimum total cost for sending =the minimum total cost for sending

the first j files using a subset of the first i the first j files using a subset of the first i providers (in the sorted order)providers (in the sorted order)



compute an auxiliary array minpcompute an auxiliary array minpi i in O(n) timein O(n) time

for for j in [Tj in [T2,i2,i, T, T1,i1,i+T+Tmax,imax,i]:]: Cmin[i,j]=min{Cmin[i-1,j], Cmin[i,j-1]+LCmin[i,j]=min{Cmin[i-1,j], Cmin[i,j-1]+L jj, , (j-T(j-T1,i1,i)·C)·Cii+minp+minpii[j-T[j-Tmax,imax,i]}]}

Cmin[k,n] – the answerCmin[k,n] – the answer O(nO(n··k) overallk) overall


Online Resource ManagementOnline Resource Management(1/4)(1/4)

scenarioscenario a resource manager receives resource a resource manager receives resource

allocation and reservation requestsallocation and reservation requests request=amount of resource (bandwidth) + request=amount of resource (bandwidth) +

time constraints (fixed duration, earliest start time constraints (fixed duration, earliest start time, latest finish time)time, latest finish time)

modelmodel time – divided into equally-sized time slotstime – divided into equally-sized time slots many requests simultaneously => we need many requests simultaneously => we need

low response times => efficient data structurelow response times => efficient data structure



algorithmic frameworkalgorithmic framework block partitioning methodblock partitioning method

array of n cellsarray of n cells each cell – value veach cell – value vii divide the cells: n/k blocks of size kdivide the cells: n/k blocks of size k update and query functions: O(k+n/k)update and query functions: O(k+n/k) k=sqrt(n) => O(k+n/k)=O(sqrt(n))k=sqrt(n) => O(k+n/k)=O(sqrt(n))

point and range queriespoint and range queries vvii=?=? qFunc(vqFunc(vaa, v, va+1a+1, ..., v, ..., vbb)=?)=?

point and range updatespoint and range updates vvii=u=u vvii=uFunc(u, v=uFunc(u, vii), a≤i≤ b), a≤i≤ b



update functionsupdate functions BPpointUpdateBPpointUpdate BPrangeUpdateBPrangeUpdate BPrangeUpdatePointsBPrangeUpdatePoints BPrangeUpdatePartialBlockBPrangeUpdatePartialBlock BPrangeUpdateFullBlockBPrangeUpdateFullBlock

query functionsquery functions BPpointQueryBPpointQuery BPrangeQueryBPrangeQuery BPrangeQueryPointsBPrangeQueryPoints BPrangeQueryPartialBlockBPrangeQueryPartialBlock BPrangeQueryFullBlockBPrangeQueryFullBlock



solves efficiently many problemssolves efficiently many problems range addition update+range min queryrange addition update+range min query range set update+range maximum sum range set update+range maximum sum

segment querysegment query ... many more... many more

typical resource reservation requeststypical resource reservation requests range addition update (reserve bandwidth for range addition update (reserve bandwidth for

a full time interval)a full time interval) range minimum query (check to see if enough range minimum query (check to see if enough

bandwidth is available in every time slot)bandwidth is available in every time slot)


ConclusionsConclusions

offline scheduling problemsoffline scheduling problems maximum profit schedulingmaximum profit scheduling

• multiple knapsack with divisible item sizes – multiple knapsack with divisible item sizes – efficient heuristicefficient heuristic

minimum cost schedulingminimum cost scheduling• optimal dynamic programming algorithmoptimal dynamic programming algorithm

online resource managementonline resource management algorithmic framework (new) for the algorithmic framework (new) for the

block partitioning method (well known)block partitioning method (well known)


Thank You !Thank You !

optimal scheduling of file transfers with divisible sizes on multiple disjoint paths

Documents

remaining items

divisible item sizessplit

divisible item sizesrepeatedly

q packed items

subset q of items

divisible item sizesnphardcan

time unitequivalent

time unitlease