A Stochastic Approach for Virtual MachinePlacement in Volunteer Cloud Federations

Abdelmounaam RezguiDepartment of Computer Science & Engineering

New Mexico Tech801 Leroy Pl., Socorro, New Mexico, 87801, USA

Email: rezgui@cs.nmt.edu

Sami RezguiUniversite d’Alger III

2 Rue Ahmed Ouaked, Dely BrahimAlgiers, Algeria

Email: rezguisami@gmail.com

Abstract—Volunteer cloud federations (VCFs) are cloud fed-erations where clouds may join and leave a federation withoutrestrictions and may contribute resources to the federation with-out long term commitment. This makes it difficult to predict thelong term availability of resources. Also, in IaaS VCFs, volunteersmay collectively contribute a large number of heterogeneousvirtual machine instances. In this paper, we focus on the problemof efficiently allocating this dynamic, heterogeneous capacity toa flow of incoming VM instantiation requests. We propose anapproach, called stochastic least differential capacity (SLDC),that allows over-provisioning only when necessary. The approachuses historical information about recent instantiation requeststo derive stochastic predictions regarding future demand. Weimplemented VCFSim, a VCF simulator that uses the proposedresource allocation solution. The results of the experimentalevaluation show that the proposed approach is able to improvethe success rate of VM instantiation requests by up to 38%compared to an approach that uses exact matching with nodemand forecasting.

I. INTRODUCTION

A cloud federation is a loosely coupled, possibly interoper-able, collection of clouds that provides to its users the abstrac-tion of a single cloud. Through this abstraction, users see asingle pool of services aggregated from different providers.Cloud federations are viewed as a means to address theeconomic problems of vendor lock-in and provider integration.In addition, they are an alternative to reduce cost (e.g., throughpartial outsourcing to more cost-efficient regions) and improveperformance and disaster-recovery through methods such asco-location and geographic distribution [1].

As the idea of cloud federations matures, scientists areincreasingly recognizing that sharing resources from severalclouds could provide the computing/storage resources neededto solve many complex science problems. The owners of manyprivate clouds now realize that it is in their mutual benefitsto volunteer resources from their individual clouds to form alarger volunteer cloud federation (VCF). This is a more costeffective alternative (than deploying larger clouds) to enablethe execution of applications that may not be possible usingresources on a single cloud. Volunteer cloud federations enablevolunteer cloud computing (VCC), a computing model thatcombines underutilized computing and/or storage resources onseveral clouds to quickly solve compute- and/or data-intensiveproblems.

While volunteer cloud computing is a promising computing

model, substantial research is needed to explore architectures,systems, and techniques enabling this model. In this paper,we focus on a fundamental and complex challenge, namely,developing a virtual machine placement technique for VCFs.Traditionally, VM placement has been studied in the context ofa single cloud. In a single cloud, efficient VM placement tech-niques are important because they help achieve one or moreobjectives including increasing resource utilization, improvingload balancing, reducing the need for VM migration, reducingenergy consumption, improving fault tolerance, etc. In VCFs,the problem is to determine an efficient mapping between VMsand individual member clouds. Both inter- and intra-cloud VMplacement techniques are needed in cloud federations: once aVM placement algorithm for federated clouds determines inwhich member cloud a VM must be hosted, another algorithmmust determine which host of the selected cloud must run theVM.

Solutions to the problem of VM placement in VCFs musttake into account that, in VCFs, resources are volunteered andthat the owner of a volunteered resource may withdraw it fromthe federation at any time. Also, users access resources on aVCF with no monetary obligations. VM placement in this casemust attempt to maximize the utilization of these resources (notprofit) while, possibly, enforcing additional constraints such asfairness or proportional service, i.e., users who also volunteerresources must be able to get a service from the federationthat is proportional to the resources that they contribute to thecloud federation.

Compared to the case of single clouds, the problem ofVM placement in VCFs remains largely unexplored. In thispaper, we focus on the problem of maximizing the number ofvirtual machines that may be placed on a VCF. Specifically, wepropose a new approach for VM placement in VCFs based onthe idea of demand forecasting. In this context, the demandis the aggregate number of VM instantiation requests. Theproposed approach uses historical information about recentinstantiation requests to derive stochastic predictions regardingfuture demand. Unlike existing approaches for traditional cloudfederations, our approach takes into account the unique charac-teristics of VCFs. In particular, since resources are volunteered(i.e., donated for free), the optimization goal in VCFs is tomaximize resource utilization, i.e., maximize the number ofvirtual machines admitted to the federation. The proposedsolution uses a model of VCFs where member clouds exposetheir capacity in terms of number and size of computing

instances (i.e., VMs) that they are willing to contribute to thecloud federation.

The remainder of this paper is organized as follows. Inthe next section, we present our model of cloud federations.In Sections III and IV, we present the theoretical detailsof the proposed approach for VM placement in VCFs. InSection V, we present an experimental evaluation of theproposed solution. Finally, in Section VI, we give concludingremarks and some future research directions.

II. CLOUD FEDERATION MODEL

Fig. 1: Interactions in a Cloud Federation

We consider a cloud federation (Figure 1) where eachcloud owner contributes a certain amount of resources. Whenjoining the federation, a cloud owner contributes resources tothe cloud federation in the form of a set of VM instances ofdifferent sizes. When a new set of instances is made availableby a contributor, the cloud federation controller updates itslist of available instances accordingly. In practice, privateclouds may be built using different software packages. As aresult, different federation members may define heterogeneousVM instances, i.e., instances that differ in terms of resourcessuch as CPU speed, disk or memory size. Let R be the listof m resources over which instances are defined. An instancetype ITi is defined as a tuple (ri1, r

i2, .., r

im) where rik is the

amount of resource k offered in instances of type ITi. Forexample, let R be the list:(Number of CPU cores, CPU frequency,memory size, disk size, bandwidth).

Define instance types: IT1 and IT2 as follows:

IT1 ≡ (2, 2Ghz, 2GB, 150GB, 100MB/s)IT2 ≡ (4, 2.5Ghz, 4GB, 100GB, 500MB/s)

An instance of type IT1 would have 2 CPU cores of 2 Ghzeach, 2 GB of memory, 150 GB of disk, and a network band-width of 100 MB/s. A volunteer may, for example, contributeten instances of type IT1 and five instances of type IT2. Wedefine a contribution as a set of pairs {(IT1, c1), .., (ITn, cn)}

Fig. 2: LDC-based VM placement

Fig. 3: SLDC-based VM placement

where ci is the number of instances of type ITi volunteeredby a given cloud owner.

Definition 1:

Let D be the set of member clouds in the federation. Atany given time, the current capacity of the federation is theunion of all contributions made by all its member clouds andnot currently allocated to any user.

Definition 2:

Let ITi ≡ (ri1, ri2, .., r

im) and ITj ≡ (rj1, r

j2, .., r

jm) be two

instance types. ITi is said to dominate ITj if and only if:

∀k, 1 ≤ k ≤ m : rik ≥ rjk

We note this by: ITi ∈ Dom(ITj). Note that the relationshipdominate is a partial order over all instance types of the cloudfederation.

Users of the federation submit their requests in terms ofVM instantiation requests (Figure 1). Each request specifiesan instance type ITi. The federation controller then attemptsto determine a cloud member that still has capacity to createsuch an instance. Two cases may occur:

Case 1: if a member cloud is found that has capacity for,at least, one instance of the requested type, the federationmanager forwards the user’s request to that member cloud.In general, if the requested instance type is available onseveral member clouds, the federation controller chooses oneof the member clouds based on some policy such as: cloudgeographically closest to the user, round-robin, cloud withhighest remaining capacity first, etc.

Case 2: if no member cloud has available instances ofthe requested type, the federation controller may either rejectthe request or create an instance larger than (i.e., dominating)

the one requested (if, of course, one can be found). If thelatter approach is adopted, a large number of options (i.e.,combinations of member clouds and their hosted instances)may satisfy the submitted request. A significant challenge is todetermine an efficient mapping that determines which membercloud to host the new VM and which instance in this membercloud is the most suitable for the new VM. In this context, theterm “efficient” means “in a way that leaves as much capacityas possible available for future VM instantiation requests.”

III. LEAST DIFFERENTIAL CAPACITY APPROACH

To create a virtual machine on the cloud federation, a userfirst selects an instance type ITi that is appropriate to his/herapplication(s) and submits a request to the federation controllerto create a VM of the desired type. The federation controllerthen uses a VM placement approach to determine the bestmember cloud to host the new VM and which instance inthis member cloud is most suitable for the new VM. If anexact match is not available, the federation controller mustselect a VM instance among a large number of dominatingVM instances that may be available on several member clouds.An easy alternative in this case is to randomly allocate oneof those larger-than-requested VMs to the user’s request. ThisVM over-provisioning, however, can quickly exhaust resourceson the member clouds. In this section, we introduce a basicscheme, called the Least Differential Capacity (LDC) approachthat attempts to create VMs while minimizing the “wasted”capacity, i.e., capacity in excess to what the user is requesting.

It is important to note that the problem addressed in thispaper cannot be addressed by simply adjusting the size of VMinstances available on the volunteered clouds to match the sizeof the VM requested by users. With current virtualization tech-nology, the overhead of VM resizing has become negligible.However, in the context of VCFs, cloud owners generally donot allow external entities (e.g., a cloud federation’s controller)to make changes that may impact their clouds, e.g., changingthe volunteered capacity. We first present the intuition throughan example.

Consider a scenario (Figure 2) where m, the number ofresources is 2: disk and memory, and instance types are definedas tuples (r1, r2) where r1 and r2 are the disk and memorysizes of the given instance type. Assume that, at a giventime, the cloud federation’s available capacity consists of fourinstances of types IT1, IT2, IT3, and IT4. Figure 2 is a graphicrepresentation of the cloud federation’s instance types in a 2Dplane. First, a VM creation request is received for type IT1.The federation has capacity to create one VM of type IT1.It creates a VM of type IT1. Assume that the next requestis also the creation of a VM of type IT1. Since there is noavailable instance of type IT1, the LDC approach will selectinstance type IT2 to satisfy the current request since IT2is the closest in terms of capacity to IT1 compared to IT3and IT4. The “wasted” capacity corresponds to the rectanglelabeled 1 in Figure 2. After this operation, the cloud federationhas no more available instances of type IT2. Assume thatthe following VM creation request is of type IT2. Sincethis request cannot be satisfied as requested, the federationcontroller will create a larger instance of type IT3 to satisfy thecurrent request. Here again, there will be a “wasted” capacitythat corresponds to the rectangle labeled 2 in Figure 2. Notice

that the instance IT4 cannot be used to satisfy this third requestbecause IT4 /∈ Dom(IT2).

Now, consider a scenario where the cloud federation isable to probabilistically predict the sequence of future VMcreation requests. In the previous example, the sequence ofrequested instance types was: IT1, IT1, and IT2. In this newscenario, the federation controller will process the first requestas it did before, i.e., create a VM of type IT1. When it receivesthe second request, the federation controller may exploit theprobabilistic knowledge that a request of type IT2 is imminentand, as a result, spare the remaining available instance of typeIT2 for that likely future request and create a VM of typeIT4. Notice that this is possible since IT4 ∈ Dom(IT1).When the third request of type IT2 is received, the federationcontroller will be able to create a VM of the exact requestedsize without over-provisioning. In this scenario, the “wasted”capacity corresponds to the rectangle labeled 3 in Figure 3. Asit is shown in Figures 2 and 3, the total “wasted” capacity inthe second scenario (rectangle 3) is much smaller than the total“wasted” capacity in the first scenario (sum of rectangles 1 and2). The basic idea here is to exploit probabilistic informationabout the likelihood that the instantiation of a VM of a certainsize will be requested in the near future. The example showsthat the basic LDC approach does improve the success ratewhen processing VM instantiation requests. However, it alsoshows that it may yield sub-optimal VM allocations resultingin substantial capacity loss. We now elaborate on the proposedstochastic least differential capacity approach.

IV. STOCHASTIC LEAST DIFFERENTIAL CAPACITYAPPROACH

In the previous section, we showed the limitation of theLDC approach. To address this limitation, we introduce thestochastic least differential capacity (SLDC) approach whichimproves on the basic LDC approach by exploiting two ele-ments, namely: (i) the current VM supply, i.e., current availablecapacity for each instance type and (ii) a stochastic informationabout future demand, i.e., a probabilistic knowledge (derivedby extrapolating recent history) about the types of future VMinstantiation requests.

We now associate a pair (ci, µi) with each instance typeITi. ci is the current available capacity for instances of typeITi, i.e., the total count of instances of type ITi available onall member clouds. µi is the probability that, in the near future,the federation receives a VM instantiation request of type ITi.Several approaches may be used to evaluate µi. We presentour approach to evaluate µi in the next section.

Consider a cloud federation that offers n instance typesof VMs. We represent VM instance types as points in anm-dimensional space where each dimension corresponds toa resource (see Figure 2). In this space, VM instance type ITicorresponds to the point whose coordinates are (ri1, r

i2, .., r

im).

We introduce a matrix ∆ of size n × n, called the pairwisecapacity distance matrix. Intuitively, for two instance typesITi and ITj , ∆(i, j) is a measure of the difference in capacitybetween ITi and ITj . Formally, ∆ is defined as follows:

∆(i, j) =

{‖−−−−→ITiITj ‖ if ITj ∈ Dom(ITi)

+∞ otherwise(1)

Initializationh = 0;/* h is the index in the history Hq */

When a request to create a VM of instance type ITi is receivedif an instance I of type ITi is available

allocate the instance I to the requestelse

if necessaryUpdate DIC(i, ∗) the vector of differential instantiation costs relevant to instance type ITi;/* In practice, the total number of instance types is small. The cost of this update is thereforein the order of O(1) */

endif

Find a VM instance type ITj such that:ITj ∈ Dom(ITi) and DIC(i, j) = minkDIC(i, k)

/* Again, as the total number of instance types is small in practice, the cost of this operation is in the order of O(1) */if found

Create a VM instance using instance type ITj ;endif

Hq[h] := ih = (h + 1) % q;ci := ci − 1;

endif

Fig. 4: Stochastic Least Differential Capacity (SLDC) Algorithm

Where ‖−−−−→ITiITj ‖ is the length of the vector

−−−−→ITiITj , i.e.,

‖−−−−→ITiITj ‖=

√∑mk=1(rjk − rik)2. This definition assumes that

all resources have equal importance. In practice, to better cap-ture the distance between VM instance types, cloud providersmay associate different weights to different resources, If αk isthe weight of resource rk, ∆(i, j) may be given as:

∆(i, j) =

{√∑mk=1 αk(rjk − rik)2 if ITj ∈ Dom(ITi)

+∞ otherwise(2)

Normally, the matrix ∆ may be computed entirely offline.It must be updated only when new instance types are madeavailable to the federation or when the federation ceases tooffer a given VM instance type. For example, when a newmember joins the federation and volunteers new VM instancetypes, n (∆’s dimension) must be increased to accommodatethe newly introduced VM instance types. Also, when the sizeof an existing VM instance type changes, ∆ must be updatedto reflect this change.

Given a request to create a VM of type ITi, if an exactmatch is not found, the SLDC algorithm uses over-provisioningto satisfy this request. Let ITj be the VM instance typeallocated to the user’s request instead of the requested instancetype ITi. To measure the capacity “waste” resulting fromthis over-provisioning, we introduce the concept of differential

instantiation cost (DIC) defined as follows for a given pair ofinstance types ITi and ITj where: ITi is the requested instancetype and ITj is an instance type such that: ITj ∈ Dom(ITi):

DIC(i, j) =µj

cj∆(i, j) (3)

Intuitively, DIC(i, j) is the cost of creating a largerinstance of type ITj when the the actual request specifiesa smaller instance of type ITi. The formula captures threeelements:

µj : the instantiation probability of type ITj . The cost isobviously higher if it is more likely that, in the future, morerequests of type ITj will be received.

ci: the current count of instances of type ITj . The cost isclearly inversely proportional to the “abundance” of instancesof type ITj . Allocating an instance of type ITj to the currentrequest makes instances of type ITj scarcer which, in turn,results in an increase in the cost of creating instances of thistype.

∆(i, j): this value captures how much excess capacity weare wasting by allocating an instance of type ITj instead ofthe smaller requested VM of type ITi.

We now turn to the issue of forecasting future instantiationprobabilities (i.e., µj in Equation 3). Several approaches may

be adopted to predict VM instantiation probability. We adopta history-based approach where the probability of an instancetype to be requested in the future is derived from the historyof past VM creation requests that the cloud federation hasreceived up to the current time. The federation controllermaintains an instantiation history, noted H, of VM creationrequests. H is a list where the ith entry H[i] is the ith

instance type requested. Taking into account the entire VMinstantiation history may not be accurate in practice. Forexample, many users may have created VM instances far inthe past, e.g., several months ago. Including these old requestswhen evaluating VM instantiation probabilities may distort theresult. To improve accuracy, we limit the computation to onlythe most recent history, i.e., the most recent q requests. Thissubset of H is noted Hq . Let vi be the number of times thecloud federation receives a VM creation of type ITi in thepartial history Hq . A simple formula to determine µi at thecurrent time may be: µi = vi

‖Hq‖ = viq . This formula, however,

gives equal weights to old and recent requests. We argue thatrecent history tends to repeat itself and that recent history ismore relevant than older history. In practice, this tends to betrue. For example, for a few weeks, the first author and otherco-authors were working on these two scientific problems: [3]and [2]. During that period, they frequently used a number ofsimilar Amazon EC2 instances. If a cloud federation were usedinstead, the federation would have been processing similarrequests for a certain period of time. To exploit the informationabout recent history, we define a weight function w over VMinstantiation events in the history Hq . w is a function thatincreases linearly from 1/q, which is the weight associatedwith the oldest VM instantiation event (i.e., the first in thecurrent moving window) to q/q = 1, which is the weightassociated with the most recent VM instantiation event in thepartial history Hq (i.e., the qth in the current moving window).We note this by: wj = j

q ,∀j, 1 ≤ j ≤ q.

To normalize (since the objective is to determine prob-abilities), we first compute W , the sum of weights for allinstantiation events: W =

∑qj=1 wj =

∑qj=1

jq = q+1

2 .

µi can now be computed as:

µi =

∑qj,Hq [j]=i wj

W=

2

q(q + 1)

q∑j,Hq [j]=i

j (4)

In terms of complexity, computing µi as given in theprevious equation is clearly in the order of Θ(‖ Hq ‖) = Θ(q).In practice, however, it is easy to use data structures that lowerthis complexity to O(q).

Figure 4 gives the details of the SLDC algorithm.

V. EXPERIMENTAL EVALUATION

We implemented VCFSim, a simulator of VCFs based onour federation model presented in Section II. VCFSim uses theSLDC approach for VM instantiation. To determine the typesof the volunteered VM instances, we adopt the characteristicsof Amazon EC2 instances1 shown in Table I. The simulatorfirst builds the federation by having a number of member

1http://aws.amazon.com/ec2/instance-types

TABLE II: Configurations Used in the Experiments

Config 1 Config 2Number of Member Clouds 20 40Number of VM Types 40 100Max Number of Volunteered VMs 10 100History Size 100 100

Fig. 5: Effect of SLDC on Capacity (Config 1)

Fig. 6: Effect of SLDC on Capacity (Config 2)

clouds join the federation. We focused on a scenario of a staticcloud federation, i.e., members do not leave the federation. Toadd a cloud to the federation, VCFSim generates a (different)random volunteered capacity for each member cloud. VCFSimalso randomly distributes this capacity, i.e., it randomly selectshow many instances of each type the new member cloud iswilling to contribute.

To evaluate the SLDC algorithm, we focused on twoaspects: (i) the absolute impact of the SLDC algorithm onincreasing the federation’s usable capacity and (ii) the relativeimpact of the SLDC algorithm compared to scenarios thatuse random over-provisioning. Figure 5 shows how the SLDCalgorithm contributes to improving the success rate of VMinstantiation requests. The figure shows the results of 20simulations. Each simulation first forms a VCF having theconfiguration Config 1 of Table II. In Figure 5, the x-axiscorresponds to the (randomly generated) federation’s capacity.The solid line corresponds to the number of successful VMallocations without applying the SLDC algorithm. The dottedline corresponds to the additional VM allocations made pos-sible by applying the SLDC technique. The experiment showsthat, in this scenario, the proposed technique improves thesuccess rate by a factor that ranges from 12% to 38%. Theaverage improvement was 23%.

TABLE I: Characteristics of VCFSim’s VM Instances (Similar to Amazon EC2 Instances)

Processor Architecture (bits) 32,64vCPU 1,2,4,8,16,32ECU 1,2,4,5, 6.5, 8, 13, 20, 26, 33.5, 35, 88

Memory (GiB) 0.615, 1.7, 3.75, 7, 7.5, 15, 17.1, 22.5, 30, 34.2, 60.5, 68.4, 117, 244Instance Storage (GB) 160, 240, 350, 410, 420, 840, 1680, 2048, 49152Network Performance 1 (Very Low), 2 (Low), 3 (Moderate), 4 (High), 5 (10 Gigabit)

Fig. 7: Effect of Random Over-provisioning on Capacity(Config 1)

We ran the same 20 simulations with a larger federation(configuration Config 2 in Table II). Figure 6 shows theeffect of the SLDC technique in this larger configuration. Theexperiment shows that, in this case, the proposed techniqueimproves the success rate of VM instantiation requests by afactor that ranges from 10% to 17%.

In the second set of experiments, we studied the relativeimpact of the SLDC algorithm compared to scenarios that userandom over-provisioning. Figure 7 shows the results of over20 simulations. As in the case using SLDC, each simulationfirst forms a VCF having the configuration Config 1 of Table II.The experiment shows that, in this configuration, random over-provisioning improves capacity utilization by a factor thatranges from 9% to 35%. The average improvement was 20%.Compared with the SLDC scenario (where the average is 23%),the difference in this small configuration is not consequential.We also ran the same set of experiments (random over-provisioning) using Config 2 of Table II. The results (Figure 8)show that, in this case, random over-provisioning improvesthe success rate of VM instantiation requests by a factor thatranges only from 5% to 11%.

The experiments discussed in this section show two results.First, over-provisioning (whether random or SLDC-based) canimprove the success rate of VM instantiation. The improve-ment is much more pronounced in the case of SLDC-basedover-provisioning. Second, in both types of over-provisioning,the improvement tends to be more substantial in small/mediumcloud federations than in larger federations. This finding canbe explained by the fact that, as a cloud federation gets larger,capacity also increases. As a result, VM instantiation requestswill then increasingly be satisfied without the frequent needfor over-provisioning.

VI. CONCLUSION

We presented a probabilistic approach to the problemof virtual machine placement in volunteer cloud federations.

Fig. 8: Effect of Random Over-provisioning on Capacity(Config 2)

The solution, called the stochastic least differential capacity(SLDC) approach, is based on three elements: (i) evaluatingthe probability of future demand in terms of virtual machineinstantiation requests using recent history, (ii) taking intoaccount the current supply in terms of VM availability whenmaking VM allocation decisions, and (iii) minimizing thecapacity waste when exact virtual machine matching is notpossible. To evaluate the proposed technique, we implementedVCFSim, a simulator for VCFs. Our experiments showedthat the SLDC approach can potentially increase the successrate of VM placement by up to 38% compared to traditionalapproaches.

The solution presented in this paper forecasts future VMdemand but does not forecast future VM supply (it only usescurrent information about VM supply). We plan on improvingthis solution by forecasting both supply and demand. We alsoplan on extending the current approach to dynamic VCFswhere member clouds may join and leave the federation andmay also change their resource contribution to the federation.

ACKNOWLEDGMENT

This research is supported in part by a grant from the NMNASA EPSCoR.

REFERENCES

[1] David Bermbach, Tobias Kurze, and Stefan Tai. Cloud Federation:Effects of Federated Compute Resources on Quality of Service and Cost.In IC2E, pages 31–37, 2013.

[2] Abdelmounaam Rezgui, Zaki Malik, Jizhe Xia, Kai Liu, and ChaoweiYang. Data-intensive Spatial Indexing on the Clouds. In Vassil N.Alexandrov, Michael Lees, Valeria V. Krzhizhanovskaya, Jack Dongarra,and Peter M. A. Sloot, editors, ICCS, volume 18 of Procedia ComputerScience, pages 2615–2618. Elsevier, 2013.

[3] Abdelmounaam Rezgui, Zaki Malik, and Chaowei Yang. High-resolutionSpatial Interpolation on Cloud Platforms. In Sung Y. Shin and Jose CarlosMaldonado, editors, SAC, pages 377–382. ACM, 2013.