Distributed VNF Scaling in Large-scale Datacenters:An ADMM-based Approach

Farzad Tashtarian?, Amir Varasteh†, Ahmadreza Montazerolghaem§, and Wolfgang Kellerer†

? Department of Computer Engineering, Mashhad Branch, Islamic Azad University, Mashhad, IranEmail: f.tashtarian@mshdiau.ac.ir

† Chair of Communication Networks, Department of Electrical and Computer Engineering,Technical University of Munich, Munich, GermanyEmail: amir.varasteh, wolfgang.kellerer@tum.de

§ Department of Computer Engineering, Ferdowsi University, Mashhad, IranEmail: ahmadreza.montazerolghaem@stu.um.ac.ir

c© 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, includingreprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, orreuse of any copyrighted component of this work in other works.

Abstract—Network Functions Virtualization (NFV) is apromising network architecture where network functions arevirtualized and decoupled from proprietary hardware. In moderndatacenters, user network traffic requires a set of Virtual NetworkFunctions (VNFs) as a service chain to process traffic demands.Traffic fluctuations in Large-scale DataCenters (LDCs) couldresult in overload and underload phenomena in service chains.In this paper, we propose a distributed approach based onAlternating Direction Method of Multipliers (ADMM) to jointlyload balance the traffic and horizontally scale up and downVNFs in LDCs with minimum deployment and forwarding costs.Initially we formulate the targeted optimization problem as aMixed Integer Linear Programming (MILP) model, which is NP-complete. Secondly, we relax it into two Linear Programming(LP) models to cope with over and underloaded service chains. Inthe case of small or medium size datacenters, LP models could berun in a central fashion with a low time complexity. However, inLDCs, increasing the number of LP variables results in additionaltime consumption in the central algorithm. To mitigate this, ourstudy proposes a distributed approach based on ADMM. Theeffectiveness of the proposed mechanism is validated in differentscenarios.

Keywords—Network function virtualization, datacenters, VNFscaling, service chaining, distributed optimization, alternating di-rection method of multipliers (ADMM).

I. INTRODUCTION

Today, different technologies such as Software DefinedNetworking (SDN) and Network Function Virtualization(NFV) are being integrated to facilitate the modern datacenterservice providers [1]. In these datacenters, the network servicesare frequently deployed as Virtual Network Function (VNF)chains to process the incoming traffic [2] (see Fig. 1). The NFVarchitecture relies on off-the-shelf Physical Machines (PMs)which provide computational resources to several Virtual Ma-chines (VMs), each VM implementing a network function withspecial software programs. In addition, SDN technology couldmanage the communication network between these VNFs inthe datacenter.

Due to traffic fluctuations in datacenters, resource overand underload can occur in service chains. The impact ofan overload could be the degradation of overall performanceand SLA violations, while underload could lead to significantresource wastage. One of the most popular methods to ensure

Internet

Different VNF types (VMs)

Core Layer

Aggregation Layer

Access Layer

Physical Machines

ToR switch

Fig. 1: A service chain with three VNFs in a conventional datacenter

efficient resource management, is the appropriate placement ofVNFs on PMs, which is called the VNF placement problem[3]. This problem has been addressed in literature regardingvarious parameters, such as network delay and end-to-endlatency [4], [5], and operational costs of VNFs (deploymentcost, forwarding/processing cost, and energy consumption) [6],[7], and [8]. Although the majority of studies focus on theinitial placement of the service chain, network-aware resourcemanagement of the service chains during their operational timehas emerged as a critical issue. To overcome this issue, thetraffic could be load balanced between VNF instances, or theover and underloaded VNF(s) could be scaled up and down. Inthis paper, we jointly load balance the traffic and horizontallyscale up and down VNFs with minimum deployment andforwarding costs. Notably, we use thresholds to automaticallydetect over and underloaded VM(s), respectively.

Based on the above considerations, the problem is modeledusing Mixed Integer Linear Programming (MILP) which isNP-complete [9]. Therefore, we relax our MILP model intotwo LP formulations. These models perform well in small andmedium size datacenters. However, in Large-scale DataCenters(LDCs), the number of variables and constraints of the LPmodels escalate significantly resulting in an increase in thetime complexity of the algorithm, which requires the useof a distributed approach. Hence, we extend our approachto distributed optimization, using a technique known as Al-ternating Direction Method of Multipliers (ADMM) [10]. Inaddition to simplicity and its powerfulness, this method hasseveral advantages like fast convergence, low computationalcomplexity, and low overhead of message passing [10]. Using

arX

iv:1

709.

0131

3v1

[cs

.NI]

5 S

ep 2

017

this method, the convex optimization problem is decomposedinto different subproblems, each of which is easier to handle.These subproblems are solved iteratively and in parallel usingdefined agents [10].

Although the conventional ADMM is guaranteed to con-verge for two-block optimization problems, for n-block vari-able problems (i.e. multi-block variables) with n > 2, updatingdifferent variables sequentially has no convergence guarantee[11]. Therefore, in this paper, because our problem is in forma of multi-block ADMM, we use a specific ADMM extensioncalled Randomly Permuted ADMM (RP-ADMM) [12] to solvethe problem in a distributed way.

Afterall, considering the aforementioned explanations, themain contributions of our proposed work are described asfollows:

1) Introducing a joint traffic load balancing and VNFscaling mechanism to mitigate both over and under-loaded situations in datacenters,

2) Formulating the above problem as a MILP opti-mization model, and proposing two LP-relaxations toaddress time complexity,

3) Extending our approach to a distributed method basedon the RP-ADMM technique,

4) Evaluation of the proposed work in different scenar-ios via simulation.

The remainder of this paper is organized as follows, westart by presenting the related work in the literature (sectionII). Then, in sections III and IV we illustrate the system modeland problem formulation, respectively. Next, the proposed ap-proaches are described in section V. We validate our approachin section VI, and finally, section VII concludes the wholepaper.

II. RELATED WORK

VM placement has been widely studied in the cloud com-puting environment [13], [14], [15], and [3]. However, specificcharacteristics and constraints of the NFV paradigm such asservice chaining requirements, deployment and license costs,etc., has made the VNF placement problem more complex.Several studies exist in the literature that tackle this problem.For instance, StEERING [16] focused on VNF chain placementand traffic steering and uses a simple heuristic algorithm tosolve the problem. Bari et. al. [6] proposed a model that canbe used to determine the optimal number of VNFs and to placethem at optimal locations (PMs). In [7], authors modeled thisproblem to minimize client-server communication distance andoperational costs. Also, [17] presented a joint traffic steeringand VNF placement using an Integer Linear Programming(ILP) model which sought to minimize the total delay ofthe traffic. While these approaches largely focus on initialplacement of VNFs, they do not monitor VNF chains duringtheir operational time. Hence, because of traffic fluctuations,a resource over or underload could happen in VNF chains,which is a significant challenge to address.

The work most similar to our approach is Stratos [18]. Ituses a four-step solution attempting to solve the VNF chainbottlenecks during its operational time. When an overloadhappens, it initially uses a flow distribution method. When

this method fails to solve the bottleneck, Stratos attempts tomigrate the VNFs. If the problem persists, Stratos scales up thebottlenecked VNFs. Finally, when all these procedures failed,it scales up all the VNFs in the chain with a fixed number ofinstances. The primary drawback of Stratos is that it works in atrial manner, which creates resource wastage and inefficienciesto solve the resource bottlenecks. Another close work to us is[19] which attempts to tackle dynamic provisioning of one ormultiple service chains in cloud datacenters in a centralizedand time-slotted manner. However, [19] does not address theforwarding cost in their model. This could bring additionalgrowth of east-west traffic in the datacenter which increasesthe wastage of datacenter computation resources [20].

In contrast to afore-mentioned works, we introduce a jointtraffic load balancing and VNF scaling mechanism to dealwith over and underload phenomena during service chainoperational time, while effectively considering forwarding anddeployment costs in form of a mathematical optimizationmodel. To the best of our knowledge, there is no literaturethat is applicable in LDCs. Therefore, we extend the model toa distributed form of optimization, using RP-ADMM method,to be able to use this approach in LDCs.

III. SYSTEM MODEL

We consider a datacenter as a graph G = V,L, where thevertices set V consists of U physical machines Pi ,i = 1 . . . Uand S switches Si ,i = 1 . . . S, and L represents edges of Gwhere lij = 1 if a direct communication link exists betweentwo entities i, j ∈ V. For the ease of problem formulation,we assume a conventional network topology for the datacenterwhich consists of three layers: access layer, aggregation layer,and core layer (see Fig. 1). The total available resources of aPM is considered as Pri where r ∈ R and R is the set of limitedtypes of PM resources such as CPU, memory, bandwidth, etc.Let C be a set of service chains that each of them is constitutedby an ordered sequence of VNFs fk, where k ∈ K and K isthe set of various network functions such as Firewall, IDS,NAT, etc.

We assume that each VM is providing only one networkfunction. Also, we assume all configured VMs providingnetwork function k, are using an identical computationalresource. Each VM i that belongs to chain c ∈ C and providesfk, denoted by vi,ck , is configured with initial resource ui,ck,r(r ∈ R).

We use the notation gck = vi,zj | ∀j = k, z = cindicating a set of VMs that provide fk on chain c ∈ C.In addition, ingress and egress points of gck are defined asgck−1 and gck+1, where k + 1 and k − 1 refer to the previousand next service function of fk in chain c, respectively. Weconsider two main phases for each chain: setup phase andoperational phase. In setup phase, the requested service chainis deployed in datacenter network with respect to SLA andthe available resources of the datacenter. In the second phase,the chain begins to process the incoming traffic, which ratemay vary from time to time. Therefore, by having an initialVNF placement for service chains, in this paper, we focus onthe second phase of chain lifetime and address the problem ofmanaging the over and underloaded chain with the minimumdeployment and forwarding costs.

1cg 1

ckg

ceg

ckg 1

ckg

,1i c

,i ck

j,ck

,1

i ck

,i ce

( )a

1ckg

ckg 1

ckg

,1

i ck

,1

i ck

,1

i ck

,i ck

j,ck

'j ,ck

' ,i ck

( )b

1ckg

ckg 1

ckg

,1

i ck

,1

i ck

,i ck

( )c Fig. 2: VNF service chain states in a datacenter network, (a) Normal state: an operational VNF chain in a network; (b) Overload occurrence:VMs i′ and j′ are launched to manage the overload situation; and (c) Underload occurrence: VM j is turned off to optimize resource usageefficiency in the VNF chain.

Our proposed system architecture includes four main com-ponents: Monitoring, SDN Controller, VNF Placement andScaling Module, and REsource Management (REM). Since theproposed approach should respond efficiently and quickly totraffic fluctuations, we use monitoring component to contin-uously collect the amount of CPU, memory, and bandwidthusage of VMs. In addition to the reported data by monitor-ing component, the SDN controller collaborates with REMthrough sharing the routing paths and configuration of networkswitches. It also updates and configures the routing policyusing the REM output. We define V and V ∗ as the set ofrunning VMs in gck (online VMs), and the set of configuredVMs to join gck (offline VM pool), respectively. Also, let ui,ck,r bethe utilization of resource type r of vi,ck in τ units of time thatis measured by monitoring module. Further, we define ui,ck,r,ui,ck,r, and ui,ck,r as hot, warm, and cold thresholds on resourcetype r for vi,ck to enable us to specify the state of a chain,respectively, as follows:(1) overload state:

∃k : gck ∈ c | ∃i : vi,ck ∈ gck and ui,ck,r ≥ u

i,ck,r

(2) underload state:

∃k : gck ∈ c |1

| V |∑

i:vi,ck ∈gck

ui,ck,r ≤ ui,ck,r

and maxui,ck,r | ∀i : vi,ck ∈ gck ≤ u

i,ck,r

(3) normal state: If chain c is neither in overload nor underloadstate, it is considered being in the normal state.For illustration, suppose chain c consists of three functions:f1, f2, and f3, where v1,c

1 , v1,c2 , v2,c

2 , and v1,c3 are serving

these functions, respectively. We assume 50%, 65% as CPUutilization of v1,c

1 and v1,c3 , respectively. In this example, we

focus on f2 to show how the state of the chain c can be de-termined. Therefore, we consider the following two scenariosfor different CPU utilization of the VMs serving f2. Also, weassume hot, warm, and cold thresholds as 90%, 80% and 30%,respectively. Scenario (a) u1,c

2,[cpu] = 95%, u2,c2,[cpu] = 75%:

Since the CPU utilization of v1,c2 violates the hot threshold,

chain c falls into the overload state. We note that the presenceof only one violated VM is sufficient to put a chain in theoverload state. Scenario (b) u1,c

2,[cpu] = 40%, u2,c2,[cpu] = 15%:

In this scenario, it can be seen that the average of CPUutilizations is less than the cold threshold. Consequently, thisscenario makes chain c to be considered as underload. Notably,f1 and f3 cannot be in underload state, since there is only oneVM assigned for each of these functions.

Remarkably, the values of hot, warm thresholds should beopted in such a way that the gap between their values providesa safety margin to cope with the temporary network traffic

fluctuations. Additionally, different types of resources can havedifferent thresholds. For example, the hot thresholds for CPUand memory resources can be set to 90% and 80%, respectively[21]. In Fig. 2, we illustrate the three operational states for asample chain.

IV. PROBLEM FORMULATION

In this section, we address the problem of overload andunderload events that occur in network service chain. First, wepresent a central optimization model, and then extend our studyby proposing a distributed model using the ADMM technique.Table I shows the main notations.

To deal with the over/underloaded chain c, a number ofconstraints must be satisfied. Suppose gck is identified as anover/underloaded set in chain c. Let bij be a binary decisionvariable, where if bij = 1, then VM j ∈ V ∪ V ∗must be running on Pi. We denote αij as the percentage ofcollaboration of VM j ∈ V ∪ V ∗ on Pi, with existing VMs∈ V to process the total traffic coming from gck−1. The valueof αij must be restricted as follows:

εbij ≤ αij ≤ ϕkbij , ∀j ∈ V ∪ V ∗,∀i ∈ P (1)where ε shows the minimum amount of collaboration thatcauses a VM to be launched on a PM. Additionally, ϕk showsthe maximum percentage of service capacity of VMs runningnetwork function k in such a way that ∀j ∈ V ∪V ∗, ∀r ∈ R,ϕkω

rkT < uj,ck,r, where ωrk is defined as the service impact

factor of network function k on resource r ∈ R. Let ndij andmdij , (i, j ∈ S ∪ P and d ∈ V ∪ V ∗), be the amount of

unprocessed traffic rates destined to vd,ck , and processed trafficby vd,ck transmitting from i to j, respectively. The followingconstraints state that all online VMs must cover the total trafficflow originated by ingress point gck−1:∑

j∈V ∪V ∗

∑i∈P

αij = 1 (2)∑s∈S∪P

lsinjsi = Tαij , ∀j ∈ V ∪ V ∗,∀i ∈ P (3)

where T is the total traffic coming from gck−1 (i.e. ingresspoint) to the over/underloaded gck. The following constraintindicates that the total required resources for deploying VM(s)on Pi must not be more than its available resources, Pri :∑

j∈V ∪V ∗biju

j,ck,r ≤ Pri , ∀i ∈ P, r ∈ R (4)

Additionally, ∀i ∈ S and ∀d ∈ V ∪ V ∗, the next constraintis presented as follows:∑

j∈S∪Plij(n

dij − ndji) = 0 (5)∑

j∈S∪Plij(m

dij −md

ji) = 0 (6)

These constraints state that the total incoming (processed andunprocessed) traffic to Si must be equal to the total outgoingtraffic from it. A similar constraint ∀i ∈ P and ∀d ∈ V ∪V ∗could be written as follows:∑

j∈S∪Plij(m

dij − γkndji) = 0 (7)

where γk > 0 is data changing factor of fk serviced by gck [22].Although a connection between any pairs of PMs in the data-center must be established via a switch, the data flow betweentwo different types of VMs, both hosted by an identical PM, isapplicable. The following constraints guarantee that the totaltraffic Tγk, processed by gck, is received by gck+1 (i.e. egresspoint):∑

d∈V ∪V ∗

∑i:Pi∈gck−1

∑j∈S∪P

lijndij = T (8)∑

d∈V ∪V ∗

∑i:Pi∈gck+1

∑j∈S∪P

ljimdji = Tγk (9)

Moreover, ∀i, j ∈ S ∪ P a bandwidth constraint could beformulated as follows:∑

d:vd,ck ∈gck

lij(ndij +md

ij) ≤ wij (10)

where wij is the total available bandwidth of link(i, j). Toevaluate the performance of any solutions that satisfies theabove constraints, we now propose the following two costfunctions:Deployment Cost :∑

i∈P

∑j∈V ∗

bijX−∑

i∈P

∑j∈V

bijX (11)Forwarding Cost :∑

d∈V ∪V ∗

∑i∈S∪P

∑j∈S∪P

F (i, j)(ndij +mdij)

(12)

Also, X and X are considered as two penalty values in thedeployment cost formulation. We use the deployment cost tocontrol the power states, migration, and placement of VMs. Infact, it can be utilized for both over and underload situationsby adjusting the proper values X and X (see Proposition. 1 forproof). Further, we use the forwarding cost to force the modelto deliver traffic via efficient routes. To achieve this, we shouldprevent the traffic from being transferred through higher layersof network in the topology (i.e. aggregation and core layers,see Fig. 1). Hence, we defined F (i, j) as the forwarding costwhich applies on transmitting composite bit-stream per unit oftime, and it is proportional to layers of datacenter topologyin which i and j are located. Finally, the following MILP isproposed to jointly balance the traffic load and scale VNFswith the minimum costs:

minimize ℵ1Eq.(11) + ℵ2Eq.(12) (13)s.t.Constraints (1)− (10)vars. bij ∈ 0, 1, αij , ndij ,md

ij ≥ 0

where ℵ1 and ℵ2 are two constant weights for deployment andforwarding costs.

Proposition 1. The proposed MILP model (13) could beused to mitigate both over and underloaded service chainswith adjusting the appropriate penalty values of X and X.

Proof: Suppose gck is an over/underloaded set whichserves VNF type k in chain c. In Eq. (11), we presentedthe deployment cost that consists of two terms: the first termhandles the online VMs, vi,ck ∈ gck, and the second term

TABLE I: Notations

Notation DescriptionSi The ith switch in datacenter networkPi The ith physical machine in datacenter networkR The set of resource types (CPU, Memory, etc.)K The set of network function typesfk The network function type k ∈ Klij lij = 1 if there is a link between two entities

i, j ∈ S ∪ PPri The total available amount of resource r ∈ R in Pi

vi,ck VM i that belongs to chain c and hosts fkgck The set of VMs that provide fk on chain c

gck−1, gck+1 The ingress/egress points of gckui,ck,r The utilization of rth resource of vi,ck

ui,ck,r , ui,c

k,r , ui,ck,r Hot, warm, and cold thresholds for rth resource of vi,ck

ui,ck,r The initial resource type r for vi,ck

ndij The amount of unprocessed traffic destined to vd,ck

transmitting from i to j, where i, j ∈ S ∪ Pmd

ij The amount of processed traffic by vd,ck transmittingfrom i to j where i, j ∈ S ∪ P

T The total traffic received by gckγk Traffic changing factor of fkαij The percentage of collaboration of vj,ck , hosted on

Pi, with other VMs providing identical networkfunction and belong to gck

ϕk The maximum percentage of service capacity ofVMs running network function k

ωrk The service impact factor of network function k on

resource r ∈ Rbij bij = 1 if VM j is hosted by Pi

wij Total available bandwidth of link(i, j), i, j ∈ S ∪ Pv∗ The number of required VNFs to process the received

traffic by gckV , V ∗ Online VMs belongs to gck, offline VM pool for gckΨ, Ψ∗ The set of PMs hosting V and candidate PMs for

hosting V ∗

eij The amount of interest of each PM ∈ Ψ ∪ Ψ∗ inprocessing the incoming traffic

focuses on offline VM instances of k which are ready to betransferred to a PM and start servicing in gck. In the presenceof an overload, there is a solution that jointly balances thetraffic rates among the online VMs, and launches one ormore offline VMs. Considering overall cost, by choosing twopositive values for X and X, where X X, the MILPovercomes overload using flow distribution on online VMs,horizontal scaling, and migration, respectively. On the otherhand, when an underload is detected, the best solution is to turnoff one or more online VMs and load balance the traffic ratesamong the remained online VMs. To preserve the resourceefficiency, there is no need to turn on new VM instances, unlessmigrating a VM dramatically decreases the forwarding cost.Hence, the value of X must be set to a large negative numberwhile X should remain as a very large positive number.

Considering the aforementioned observations, by detectingover/underloaded chain, the REM adjusts the proper values forX and X and triggers the optimization algorithm.

V. THE PROPOSED APPROACH

Since the proposed MILP model (13) is NP-completeand suffers from high time complexity [9], we consider twolinear relaxations of the MILP model to address over and

underloaded VNFs. These LP models could be centrally run insmall or even medium size datacenters. However, using thesemodels in LDCs is not applicable due to increasing the numberof variables and constraints. Thus, we extend our study byproposing a distributed mechanism based on the RP-ADMMtechnique.

A. The Central Design

In case of an overloaded gck, suppose that the number ofrequired VNFs fk, which is denoted by v∗, to process thereceived traffic equals to MaxdTωrk/u

j,ck,re | ∀r ∈ R. Also,

we define Ψ and Ψ∗ sets, which are the set of PMs that arehosting V , and the set of candidate PMs that has adequateresources to host at least one VM ∈ V ∗, respectively. Therelaxed overload model of MILP (13) is presented as follows:

The overload LP model:minimize Eq.(12) (14)s.t.Constraints (5)− (9) (I)-(V)αij − αqj = 0, i = |V |+ 1 : v∗,∀j, q ∈ Ψ ∪Ψ∗ (VI)∑

q∈Ψ∪Ψ∗eqj − αij = 0, i = 1 : v∗,∀j ∈ Ψ ∪Ψ∗(VII)∑

s∈S∪Plsin

jsi − Teij = 0,

i = |V |+ 1 : v∗,∀j ∈ Ψ ∪Ψ∗ (VIII)∑s∈S∪P

lsinjsi − Tαij = 0, i = 1 : |V |,∀j ∈ Ψ (IX)

vars. ndij ,mdij ≥ 0, 0 ≤ αij ≤ ϕk, 0 ≤ eij ≤ 1

Since the number of desired VMs is known, the deploymentcost in the objective function could be omitted. In fact, havingv∗, the relaxed overload model (14) specifically focuses on theforwarding cost. In this model, the routing constraints could beused as they were before, except the bandwidth constraint (Eq.(10)). We ignored the bandwidth constraint in the proposedrelaxed models. However, we will relax this assumption inour future work. Additionally, to relax the binary variablebij in the MILP model (13), we define eij ∈ R+ as a newvariable in which j = 1 : v∗, and i ∈ Ψ∪Ψ∗. This variableidentifies the amount of interest of each PM in processing theincoming traffic. For each required instance j, the PM withthe highest value of eij is selected to run the jth VM instanceand handle Tαij of traffic (constraints (VI)-(VIII)). Finally,constraint (IX) determines the amount of received traffic thatmust be processed by each PM i ∈ Ψ.

On the other hand, when an underload occurs, thefollowing LP-relaxation model, which can be obtained easilyfrom the above model, runs on the first v∗ online VMs toappropriately assign them to PMs ∈ Ψ and distribute receivedload from gck−1 on gck:

The underload LP model:minimize Eq.(12) (15)s.t.Constraints (5)− (9) (I)-(V)αij − αqj = 0, j = 1 : v∗,∀i, q ∈ Ψ (VI)∑

q∈Ψeqj − αij = 0, j = 1 : v∗,∀i ∈ Ψ (VII)∑

s∈S∪Plsin

jsi − Teij = 0, i = 1 : v∗,∀j ∈ Ψ (VIII)

vars. ndij ,mdij ≥ 0, 0 ≤ αij ≤ ϕk, 0 ≤ eij ≤ 1

For clarification on details of the proposed models, anexample for overloaded and underloaded chains is shownin Fig. 3. In Fig. 3(a), the overloaded gck has two VMs(1, 2) ∈ V , which are running on (P1,P2) ∈ Ψ, respectively.By assuming overloaded VM 2 and setting v∗ = 4, two extraVMs (3, 4) ∈ V ∗ with the same VNF type must be added togck regarding (P2,P3,P4) ∈ Ψ∗. Obtained αij and eij , VMs 3and 4 are selected to be launched on P3 and P2, respectively.Additionally, in Fig. 3(b), a final decision has been made basedon the calculated values of eij and αij for the underloadedchain. In this case, the relaxed underload LP model turns offVM 3 running on P2.

B. Introducing ADMM

After many years without too much attention, the ADMMtechnique has recently witnessed a renaissance in many appli-cation areas [11]. Generally, there are two types of distributedmethods: Sub-gradient and ADMM. The best known rate ofconvergence for sub-gradient methods is O( 1√

`), while the

ADMM algorithm converges at the rate O( 1` ) for the general

case, where ` is the number of iterations [23]. Therefore,since both approaches are iteration-based and the speed ofconvergence in the ADMM is more than the sub-gradientmethod, we develop our distributed approach based on theADMM algorithm.

We first introduce the background of the ADMM andthen apply it to design our distributed algorithm. Considerthe following convex minimization problem with a separableobjective function and linear constraints:

minimize f1(x1) + f2(x2) + ...+ fn(xn), (16)s.t. A1x1 +A2x2 + ...+AnXn = b,

vars. xi ∈ Xi, i = 1, ..., n,

where Ai ∈ RN×di , b ∈ RN×1, Xi ⊆ Rdi is a closed convexset, and fi : Rdi → R is a closed convex function, i = 1, ..., n[12]. In case of n = 2 (i.e. two blocks of variables), theaugmented Lagrangian function for model (16) can be writtenas follows:Lβ(x1, x2, µ) =f1(x1) + f2(x2) + µᵀ(A1x1 +A2x2 − b)

+β

2‖ A1x1 +A2x2 − b ‖22 (17)

where µ is the Lagrangian multiplier and β is the positivepenalty scalar. The two-block ADMM updates the primalvariables x1 and x2, followed by a dual variable update µin an iterative manner. By having (x0

1, x02, µ

0) as an initialvector, the updated variables at iteration ` > 0 are computedin an alternating fashion as follows:

x`1 = argminx1

Lβ(x1, x`−12 , µ`−1) (18)

x`2 = argminx2

Lβ(x`1, x2, µ`−1) (19)

µ` =µ`−1 − β(A1x`1 +A2x

`2 − b) (20)

Separable structure of convex model (16) allows ADMM todecompose it over its primal variables and achieve the optimalsolution in a distributed way. To formulate the ADMM into ascaled form that is more convenient to be used in designingalgorithm, we define u = ( 1

β )µ as the scaled dual variable.Thus, we have:

1

2 2

3

4

11

2 2

3

4

3

4

11

2 2

11

2 23

1

11 %30 22 %35

33 %25

34 %10

43 %25

44 %10

24 %7e 34 %2e 44 %1e

23 %5e 33 %15e 23 %25

24 %10 43 %5e

11 %25

12 %10

21 %25

22 %10

12 %2e 22 %8e

11 %20e 21 %5e

( )a ( )b

Applying the proposed model (14) Applying the proposed model (15)

22 %35e 11 %30e

1 2 3 42

Online VMs 1 and 2

is a candidate to host two offline VMs 4Pare two candidates to host one offline VM 3and P 2P

Selected Offline VMs 3 and 4

1 2

Fig. 3: An example of over and underloaded chains and their solution, (a) The overload case (b) The underload case

µᵀ(A1x1 +A2x2 − b) +β

2‖ A1x1 +A2x2 − b ‖22

=β

2‖ A1x1 +A2x2 − b+ u ‖22 −

β

2‖ u ‖22 (21)

By disregarding independent terms of the minimization vari-ables, the ADMM method could be expressed as:

x`1 = argminx1

(f1(x1) + (β

2) ‖ A1x1 +A2x

`−12

− b+ u`−1 ‖22) (22)

x`2 = argminx2

(f2(x2) + (β

2) ‖ A1x

`1 +A2x2

− b+ u`−1 ‖22) (23)

u` = u`−1 +A1x`1 +A2x

`2 − b (24)

The optimality and convergence of the two-block ADMMfor model (16) is proved in [10] under mild technical assump-tions.

On the other hand, many applications need more thantwo blocks of variables to be considered (i.e. n > 2). Bydirect extending of ADMM to multi-block case, the conver-gence cannot be established well [11]. The authors in [12]proposed Randomly Permuted ADMM (RP-ADMM), as asimple modification in updating process of primal variablesto make the convergence possible in a desirable duration. Infact, RP-ADMM draw a permutation Ω of 1, ..., n uniformlyat random and update the primal variables in the order of thepermutation, followed by updating the dual variables in a usualway [12]. Since our proposed approach has more than twoblocks of variables, we apply RP-ADMM technique to solvethe problem in a distributed way.

C. Applying RP-ADMM

In this section, we develop our distributed algorithm basedon the proposed model (14). We firstly reformulate the pro-posed model (14) to an applicable model for applying RP-ADMM, and then introduce our RP-ADMM-based distributedalgorithm.

Since ∀j ∈ S ∪ Ψ ∪ Ψ∗, each node i ∈ S ∪ Ψ ∪ Ψ∗is responsible to determine its local flow variables ndij andmdij in a distributed manner, we have to reformulate flow

conservation constraints (I)-(V) in (14). So, by introducingauxiliary variables Adij , B

dij , and Cdij , we reform constraints

(I)-(III), j ∈ S ∪ Ψ ∪ Ψ∗, d = 1 : v∗, where lij = 1, asfollows:

ndij − ndji −Adij = 0, ∀i ∈ S (25)∑j∈S∪Ψ∪Ψ∗

Adij = 0 (26)

mdij −md

ji −Bdij = 0, ∀i ∈ S (27)∑j∈S∪Ψ∪Ψ∗

Bdij = 0 (28)

mdij − γkndji − Cdij = 0, ∀i ∈ P (29)∑j∈S∪Ψ∪Ψ∗

Cdij = 0 (30)

Without loss of generality, assume that ingress and egresspoints of gck have one PM indexed by 0 and 1, respectively. Toreform constraints (IV) and (V) in (14), we similarly definetwo more auxiliary variables Dd and Ed, (d = 1 : v∗):∑

j∈S∪P0l0jn

d0j −Dd = 0 (31)∑

d=1:v∗Dd − T = 0 (32)∑

j∈S∪P1lj1m

dj1 − Ed = 0 (33)∑

d=1:v∗Ed − Tγk = 0 (34)

Now, the RP-ADMM can be easily applied on the followingmodel:

minimize Eq.(12) (35)s.t. Eq.(25) - Eq.(34),

constraints (VI) - (IX) in model (14)vars. ndij ,m

dij , A

dij , B

dij , C

dij , Dd, Ed ≥ 0,

0 ≤ αij ≤ ϕk, 0 ≤ eij ≤ 1

The augmented Lagrangian function for the model (35) canbe presented as follows:Lβ(n,m,A,B,C,D,E, α, e; (36)δ, δ, η, η, ζ, ζ, λ0, λ0, λ1, λ1, µ, ξ, σ, ρ) = Eq.(12)

+ δᵀEq.(25) +β

2‖ Eq.(25) ‖22 +δᵀEq.(26) +

β

2‖ Eq.(26) ‖22

+ ηᵀEq.(27) +β

2‖ Eq.(27) ‖22 +ηᵀEq.(28) +

β

2‖ Eq.(28) ‖22

+ ζᵀEq.(29) +β

2‖ Eq.(29) ‖22 +ζᵀEq.(30) +

β

2‖ Eq.(30) ‖22

+ λᵀ0Eq.(31) +β

2‖ Eq.(31) ‖22 +λᵀ0Eq.(32) +

β

2‖ Eq.(32) ‖22

+ λᵀ1Eq.(33) +β

2‖ Eq.(33) ‖22 +λᵀ1Eq.(34) +

β

2‖ Eq.(34) ‖22

+ µᵀCons. (VI, model (14)) +β

2‖ Cons. (VI model (14)) ‖22

+ ξᵀCons. (VII, model (14)) +β

2‖ Cons. (VII model (14)) ‖22

+ σᵀCons. (VIII, model (14)) +β

2‖ Cons. (VIII model (14)) ‖22

+ ρᵀCons. (IX, model (14)) +β

2‖ Cons. (IX model (14)) ‖22

where δ, δ, η, η, ζ, ζ, λ0, λ0, λ1, λ1, µ, ξ, σ, and ρ are the La-grangian multiplier and β is the positivie penalty scalar.

Considering the above explanations, we are able to utilizethe n-block RP-ADMM algorithm to distributedly solve themodel (35) (see Algorithm 1). As illustrated in Algorithm 1,it starts by initializing primal and dual variables in the firstline. In line 2, we defined an array to store the last values ofthe variables (primal and dual). The iterative process beginsin line 3 where ` indicates the round number. In each round,RP-ADMM algorithm picks a uniformly random permutationΩ to specify the order of updating primal variables. In fact,the inner for loop (lines 6 to 10), selects the Ω(i)th variableto update its value in `th round by considering the last valueof variables stored in the LastValueArray. Then, in line 9 thevalue of Ω(i)th primal variable is overwrited. For illustration,suppose that Ω = 3, 4, 2, 1 (n = 4) and ` = 12. Therefore,according to the order of Ω, the third primal variable (i.e.Ω(1)th variable) must be updated first, followed by fourth,second, and first primal variable. Also, in this round, forexample, the second variable updating process uses the currentvalues (computed in 12th round) of third and forth variables,and also the obtained value of the first variable in previousround (i.e. 11th round). Using the scaled form of ADMMalgorithm, the primal variables are updated as follows:

Updating Switches Primal Variables: If the selected vari-able belongs to the set of network switches, each switchi, updates its traffic flow rate variable to its neighbor j ∈S ∪Ψ ∪Ψ∗ using the following equations:nd(`)ij = argmin

ndij

(Eq.(36))

= argminndij

(Eq.(12) +β

2‖ ndij − n

d(`∗)ji −Ad(`∗)

ij

+1

βδd(`∗)ij ‖22 +

β

2‖ nd(`∗)

ji − ndij −Ad(`∗)ji +

1

βδd(`∗)ji ‖22

+β

2‖ md(`∗)

ji − γkndij − Cd(`∗)ji +

1

βζd(`∗)ji ‖22 +

β

2‖ ndij

+∑

s∈S∪Ψ∪Ψ∗,s6=ilsjn

d(`∗)sj − Te`

∗

jd +1

βσd(`∗)j ‖22

+ Θj∈Ψ

β

2‖ ndij +

∑s∈S∪Ψ∪Ψ∗,s6=i

lsjnd(`∗)sj − Tα(`∗)

jd

+1

βρd(`∗)j ‖22) (37)

where ΘX

plays as an if statement; i.e. it returns 1 if condition

”X” is true. Also, ad(`∗) or a(`∗) shows the most recent valueof the primal variable a, stored in LastValueArray. Similarly,md(`)ij , Ad(`)

ij , and Bd(`)ij are obtained from the following

equations:

md(`)ij = argmin

mdij

(Eq.(12) +β

2‖ md

ij −md(`∗)ji −Bd(`∗)

ij

+1

βηd(`∗)ij ‖22 +

β

2‖ md(`∗)

ji −mdij −B

d(`∗)ij +

1

βηd(`∗)ji ‖22

+ Θj=1

β

2‖ md

ij +∑

s∈S∪P1,s6=ilsjm

d(`∗)sj − E(`∗)

d

+1

βλd(`∗)j ‖22) (38)

Ad(`)ij = argmin

Adij

(β

2‖ nd(`∗)

ij − nd(`∗)ji −Adij +

1

βδd(`∗)ij ‖22

+β

2‖ Adij +

∑s∈S∪Ψ∪Ψ∗,s6=j

Ad(`∗)is +

1

βδd(`∗)i ‖22) (39)

Algorithm 1: n-block RP-ADMM

1 Initialize: nd(0)ij , md(0)

ij , αd(0)ij , ed(0)

ij , Ad(0)ij , Bd(0)

ij ,Cd(0)ij , D(0)

d , E(0)d and dual variables

2 LastValueArray[] = initialized variables;3 for ` = 1, 2, ... do4 n=length(primal variables);5 Pick a permutation Ω of 1, ..., n uniformly at

random;6 for i = 1 : n do7 //Updating primal variables:8 Update the Ω(i)th primal variable;9 Update LastValueArray[Ω(i)];

10 end11 Update dual variables;12 end

Bd(`)ij = argmin

Bdij

(β

2‖ md(`∗)

ij −md(`∗)ji −Bdij +

1

βηd(`∗)ij ‖22

+β

2‖ Bdij +

∑s∈S∪Ψ∪Ψ∗,s 6=j

Bd(`∗)is +

1

βηd(`∗)i ‖22)

(40)

Updating PMs Primal Variables: If the selected variablebelongs to the set of PMs, each PM i updates the primal dualvariables ndij , m

dij , αid, eid, Cd(`)

ij , D(`)d , and E`d using the

following equations:

nd(`)ij = argmin

ndij

(Eq.(12) +β

2‖ nd(`∗)

ji − ndij −Ad(`∗)ji

+1

βδd(`∗)ji ‖22 + Θ

i=0

β

2‖ ndij +

∑s∈S∪P0,s6=j

lisnd(`∗)is

−D(`∗)d +

1

βλd(`∗)0 ‖22 +

β

2‖ ndij

+∑

s∈S∪Ψ∪Ψ∗,s 6=ilsjn

d(`∗)sj − Ted(`∗)

jd +1

βσd(`∗)j ‖22

+β

2‖ ndij +

∑s∈S∪Ψ∪Ψ∗,s 6=i

lsjnd(`∗)sj − Tαd(`∗)

jd

+1

βρd(`∗)j ‖22) (41)

md(`)ij = argmin

mdij

(Eq.(12) +β

2‖ md(`∗)

ji −mdij −B

d(`∗)ji

+1

βηd(`∗)ji ‖22 +

β

2‖ md

ij − γknd(`∗)ij − Cd(`∗)

ij +1

βζd(`∗)ij ‖22

+ Θi=1

β

2‖ md

ij +∑

s∈S∪P1,s6=ilsjm

d(`∗)sj − E(`∗)

d

+1

βλd(`∗)i ‖22) (42)

α(`)id = argmin

αid

( Θd>|V |

β

2

∑q∈Ψ∪Ψ∗,q 6=i

‖ αid − α(`∗)qd

+1

βµd(`∗)iq ‖22 +

β

2‖∑

q∈Ψ∪Ψ∗e

(`∗)qd − αid +

1

βξd(`∗)i ‖22

+ Θi∈Ψ

β

2‖∑

s∈S∪Ψ∪Ψ∗lsin

d(`∗)si − Tαid +

1

βρd(`∗)i ‖22)

(43)

e(`)id = argmin

eid

(β

2‖ eid +

∑q∈Ψ∪Ψ∗,q 6=i

e(`∗)id − α

(`∗)qd

+1

βξd(`∗)i ‖22 + Θ

d>|V

β

2‖∑

s∈S∪Ψ∪Ψ∗lsin

d(`∗)si − Teid

+1

βσd(`∗)i ‖22) (44)

Cd(`)ij = argmin

Cdij

(β

2‖ md(`∗)

ij − γknd(`∗)ji − Cdij +

1

βζd(`∗)ij ‖22

+β

2‖ Cdij +

∑s∈S∪Ψ∪Ψ∗,s6=j

Cd(`∗)is +

1

βζd(`∗)i ‖22)

(45)

In case of i ∈ ingress, we should update D(`)d as below:

D(`)d = argmin

Dd

(β

2(‖

∑s∈S∪P0

lisnd(`∗)is −Dd

+1

βλd(`∗)i ‖22 + ‖ Dd +

∑s=1:v∗,s6=d

D(`∗)s − T

+1

βλ

(`∗)i ‖22)) (46)

However, if i ∈ egress, E(`)d should be updated as follows:

E(`)d = argmin

Ed

(β

2(‖

∑s∈S∪P1

lsimd(`∗)si − Ed

+1

βλd(`∗)i ‖22 + ‖ Ed +

∑s=1:v∗,s6=d

E(`∗)s − T

+1

βλ

(`∗)i ‖22)) (47)

Finally, in line 11 of Algorithm 1, the dual variables can beupdated as follows:

• ∀i ∈ S and ∀j ∈ S ∪Ψ ∪Ψ∗ :

δd(`)ij = δ

d(`−1)ij + β(n

d(`)ij − n

d(`)ji −A

d(`)ij ) (48)

δid(`)

= δid(`−1)

+ β(∑

s∈S∪PAd(`)is ) (49)

ηd(`)ij = η

d(`−1)ij + β(m

d(`)ij −m

d(`)ji −B

d(`)ij ) (50)

ηid(`) = ηi

d(`−1) + β(∑

s∈S∪PBd(`)is ) (51)

• ∀i ∈ Ψ ∪Ψ∗ and ∀j ∈ S ∪Ψ ∪Ψ∗ :

ζd(`)ij = ζ

d(`−1)ij + β(m

d(`)ij − γkn

d(`)ji − C

d(`)ij ) (52)

ζid(`)

= ζid(`−1)

+ β(∑

s∈S∪PCd(`)il ) (53)

σd(`)i = σ

d(`−1)i + β(

∑s∈S∪P

lsind(`)si − Te

(`)id ) (54)

µd(`)ij = µ

d(`−1)ij + β(α

(`)id − α

(`)jd ) (55)

ξd(`)i = ξ

d(`−1)i + β(

∑l∈Ψ∪Ψ∗

ed(`)ld − α(`)

id ) (56)

• If (i ∈ ingress) :

λd(`)0 = λ

d(`−1)0 + β(

∑s∈S∪P0

l0snd(`)0s −D

(`)d ) (57)

λ(`)0 = λ

(`−1)0 + β(

∑d=1:v∗

D(`)d ) (58)

• If (i ∈ egress) :

λd(`)1 = λ

d(`−1)1 + β(

∑s∈S∪P1

ls1md(`)s1 − E

(`)d ) (59)

λ(`)1 = λ1

(`−1)+ β(

∑d=1:v∗

E(`)d ) (60)

Fig. 4: 4-fat-tree topology

• If (i ∈ Ψ) :

ρd(`)i = ρ

d(`−1)i + β(

∑s∈S∪P

lsind(`)si − Tα

(`)id ) (61)

Clearly, the primal values of the RP-ADMM model can beupdated according to the selected permutation. In case of over-load, when vi,ck violates the defined threshold, it determinesv∗ and then triggers PMs ∈ Ψ ∪ Ψ∗ and some adjacentswitches via sending a simple packet. Then, the RP-ADMMupdate process can be launched considering the selected PMsand switches.

VI. PERFORMANCE EVALUATION

To assess the performance of the proposed approach, wesimulate the proposed MILP, LP, and ADMM models utilizingMATLAB software. We conducted our simulation on a hard-ware composing of an INTEL Core i7 1.73GHz CPU, 8GBof RAM, and Windows 10 x64 OS. We considered k-fat-treetopologies as the datacenter network structure. In general, ak − fat − tree has k/2 ToR and aggregate switches in eachpod, and k2/4 core switches [24]. We mainly use 4-ary fat-treetopology for performance evaluation (see Fig. 4). However, werun some of our experiments based on different topology sizes(i.e. 8, 16, 32, and 64-ary fat-trees). Also τ is considered as 5seconds for all runs.

In our first experiment, we investigate the performances ofMILP (13), LP (14), and (15) models in terms of the numberof variables, constraints, and execution time on various k-fat-trees. We consider gkc with two VMs (i.e. |V | = 2), |Ψ| = 2,|V ∗| = 1, and each rack equipped with two PMs. As it canbe seen in Table II, number of switches, PMs, variables, andconstraints are proportional to the size of datacenter (i.e. k-fat-tree). Obviously, for larger datacenters, the execution timeof the proposed models also increased specifically for MILPmodel. Table II clearly implies that in LDCs, we need to applyan efficient distributed algorithm to achieve a solution in areasonable time.

As another experiment, we consider a 4-fat-tree topologyto evaluate the performance of the proposed LP models forover and underloaded chains in different scenarios. Each ofthese scenarios is designed based on the number of PMs ingck, ingress and egress points, Ψ∗ (i.e. the candidate PMs thatare able to launch new VMs), normalized (0 ∼ 1) forwardingcosts (N. FwdCost), and v∗ (the number of required VNFs toprocess the received traffic by gck) for overload and underloadcases (see column 1-4 in Table III). We also consideredthe forwarding costs between different layers of datacen-ter as: PM-ToR=10, ToR-Aggregation=20, and Aggregation-Core=40. Further, to simulate overload and underload cases,we increased the traffic rate by 50% (for each extra VM) anddecreased it by 50%, respectively.

TABLE II: Comparing time complexities of proposed models in different topologies

MILP (13) LP (Overload) (14) LP (Underload) (15)

k-fat-tree No. of Switches No. of PMs No. of Variables No. of Constraints Run time (s) No. of Constraints Run time (s) No. of Constraints Run Time (s)2 5 4 120 25 0.555 68 0.075 62 0.0554 20 16 672 315 5.064 254 0.614 188 0.3188 80 64 4224 1251 323.523 998 14.977 692 6.63516 320 256 29184 4995 N/A 3974 974.235 2708 456.67432 1280 1024 129024 19971 N/A 15878 N/A 10772 N/A64 5120 4096 466944 79875 N/A 63494 N/A 43028 N/A

Fig. 5: The normalized violation of the constraints in model (14)

The normalized forwarding cost and its increase/decreaserates are shown in Table III. We also configured the initialparameters in such a way that LP is forced to turn on/off VMsto mitigate the over or underload gck. Overall, Table III showshow LP models react to over/underloaded chains by turningon/off offline/online VMs. In case of overload, the proposedLP model (14) prefers to launch a VM on ingress and egressPMs in order to minimize the traffic forwarding cost. In fact,the traffic between two different VNFs on an identical PM(i.e. ingress or egress) is negligible due to an internal traffichandling. For instance, in the first scenario in Table III (P1 →(P2,P5) → P4), when we need to turn on two more VMsfor gck (i.e. v∗ = 4), the LP model (14) selects P1 and P4

(i.e. ingress and egress points, respectively) to launch the newVMs (P1 → (P1P1P1,P2,P5,P4P4P4)→ P4). Also, it can be seen thatby using this new configuration, the forwarding cost increasesby 20%, which is caused by the increased amount of networktraffic. In addition, it can be seen that in the third scenario(P1 → (P1,P2)→ P2), the maximum increase in forwardingcost is resulted among all scenarios (116%). This is because,the utilized PMs before chain overload (i.e. P1 and P2) were ona same rack, while after running LP model (14), two more PMs(i.e. P3 and P4) are use to handle the traffic, which are storedon another rack. It is noticeable that we assumed that onlyP3−P16 PMs can afford enough resources to launch new VMs(i.e. Ψ∗ = P3−P16). On the other hand, when an underloadoccurs, the LP model (15) prefers to turn off a VM that has themost forwarding cost. For example, in the first scenario, P5 isturned off. Because P5 is in pod 2, so the traffic should passthe higher layers of network (i.e. aggregate and core layers)to reach the next hop in the chain (i.e. P4). Also, as it canbe seen in the last column of Table III, the forwarding costdecreases by 14%. Additionally, it can be seen that maximumforwarding cost reduction took place in the last scenario. Thereason is that the LP model (15) turns off the VMs on P3 andP4, which are stored in another rack. Therefore, the trafficis not passing the higher network layers and as a result, theforwarding cost in the chain reduces significantly.

One of the well-known features of ADMM is its fast con-vergence which is clearly illustrated in Fig. 6. For declaration,

Iterations

0 5 10 15 20 25 30 35 40 45

50

100

150

200

250

LP (overload) model (14)

Proposed ADMM algorithm

X: 15

Y: 192

Fig. 6: The convergence rate of the proposed distributed RP-ADMMalgorithm

we focused on the last scenario in Table (III) and compared theoptimal cost value obtained by LP model (14) with the solutionof distributed ADMM algorithm (we set β = 5). According toFig. 6, the distributed ADMM converges to the optimal valuein 15 iterations. This low number of iterations indicates thatthe proposed distributed ADMM technique can perform wellwith low overhead (e.g. message passing) in a LDC. Notably,we set the stop criterion to 25 iterations. In addition, in Fig.5 we measure the normalized violation of each constraint inmodel (14) to show how fast violations decrease.

VII. CONCLUSION

Today, the integration of NFV and SDN introduces anew distinguished, yet challenging networking architecture.One of these challenges is VNF over and underloads in thenetwork during their operational time. This problem couldincrease resource wastage, SLA violations, and also decreaseefficiency. In this paper, we conducted a theoretical study of theVNF scaling problem with the aim of minimizing deploymentand forwarding costs in a datacenter. In fact, we addressedthe aforementioned issue by presenting a jointly traffic loadbalancing and VNF scaling mechanism to overcome both overand underload phenomena in datacenters. In this regard, weformulated the problem as a MILP optimization model. Also,to cope with the time complexity of MILP, we relaxed it intotwo LP models. However, in LDCs, there are huge numberof network switches and servers, that increase the number ofconstraints and variables in LP models significantly. Hence, theproposed central LP model is practically useless for LDCs.Therefore, we extend our approach to a distributed methodbased on the ADMM technique. However, the conventionalADMM is guaranteed to converge for two-block variablesproblem, while for multi-block variable problems, updatingdifferent variables sequentially has no convergence guarantee.Hence, because our problem is formulated in a multi-blockvariable form, we used an extension of the ADMM technique,called Randomly Permuted ADMM (RP-ADMM) to solvethe problem in a distributed manner. To the best of ourknowledge, our work is the first one to tackle this problem

TABLE III: Comparing the results of LP models in different scenarios

gck−1 → (gck)→ gck+1 N. FwdCost LP overload model (14) LP underload model (15)Ψ∗ v∗ New Configuration FwdCost v∗ New Configuration FwdCost

P1 → (P2,P5)→ P4 0.68 ALL PMs 3 P1 → (P2,P5,P4P4P4)→ P4 +14% 1 P1 → (P2)→ P4 -14%4 P1 → (P1P1P1,P2,P5,P4P4P4)→ P4 +20%

P1 → (P1,P2,P3)→ P16 1 P3 − P164 P1 → (P1,P2,P3,P16P16P16)→ P16 +11% 2 P1 → (P1,P2)→ P4 -6%5 P1 → (P1,P2,P3,P3P3P3,P16P16P16)→ P16 +17% 1 P1 → (P1)→ P4 -16%

P1 → (P1,P2)→ P2 0.23 ALL PMs 3 P1 → (P1,P1P1P1,P2)→ P2 +25% 1 P1 → (P1)→ P2 -33%P3 − P16 4 P1 → (P1,P1,P3P3P3,P4P4P4)→ P2 +116%

P1 → (P3,P5,P6)→ P2 0.77 P2 − P16 4 P1 → (P2P2P2,P3,P5,P6)→ P2 +13% 2 P1 → (P3,P5)→ P2 -9%P3 − P16 5 P1 → (P3,P3P3P3,P4P4P4,P5,P6)→ P2 +30% 1 P1 → (P3)→ P2 -36%

using a distributed optimization approach. We evaluated theperformance of the presented models numerically in differentscenarios based on k-fat-tree topology. The results validatedthe performance of our proposed algorithms. The most im-portant part of our future work is to implement the introduceddistributed approach in real-world test beds. On the other hand,considering a workload prediction module in our architecture isworth to be explored in future. In this way, we would be able topredict the traffic load and adjust VNF placement in advance.Additionally, the performance (e.g., end-to-end delay) is alsoa critical factor in forwarding path selection of service chains.In fact, it would influence the selection of candidate PMs andVMs. Therefore, it could be considered as an extension to themathematical formulation as another future direction.

VIII. ACKNOWLEDGEMENT

We are grateful to Islamic Azad University, Mashhadbranch authorities, for their useful collaboration. This workhas been partly supported by the German Federal Ministryof Education and Research (BMBF) under the project SEcureNetworking for a DATa center cloud in Europe (SENDATE)(Project ID 16KIS0261).

REFERENCES

[1] Rashid Mijumbi, Joan Serrat, Juan-Luis Gorricho, Niels Bouten, FilipDe Turck, and Raouf Boutaba. Network function virtualization: State-of-the-art and research challenges. IEEE Communications Surveys &Tutorials, 18(1):236–262, 2015.

[2] Dilip A Joseph, Arsalan Tavakoli, and Ion Stoica. A policy-awareswitching layer for data centers. In ACM SIGCOMM ComputerCommunication Review, volume 38, pages 51–62. ACM, 2008.

[3] Arsany Basta, Wolfgang Kellerer, Marco Hoffmann, Hans JochenMorper, and Klaus Hoffmann. Applying nfv and sdn to lte mobilecore gateways, the functions placement problem. In Proceedings ofthe 4th workshop on All things cellular: operations, applications, &challenges, pages 33–38. ACM, 2014.

[4] Dilip Krishnaswamy, Ravi Kothari, and Vijay Gabale. Latency andpolicy aware hierarchical partitioning for nfv systems. In NetworkFunction Virtualization and Software Defined Network (NFV-SDN),2015 IEEE Conference on, pages 205–211. IEEE, 2015.

[5] Minh-Tuan Thai, Ying-Dar Lin, and Yuan-Cheng Lai. A joint networkand server load balancing algorithm for chaining virtualized networkfunctions. In 2016 IEEE International Conference on Communications(ICC), pages 1–6. IEEE, 2016.

[6] Md Bari, Shihabur Rahman Chowdhury, Reaz Ahmed, Raouf Boutaba,et al. On orchestrating virtual network functions in nfv. IEEETransactions on Network and Service Management, 2015.

[7] Rami Cohen, Liane Lewin-Eytan, Joseph Seffi Naor, and Danny Raz.Near optimal placement of virtual network functions. In 2015 IEEEConference on Computer Communications (INFOCOM), pages 1346–1354. IEEE, 2015.

[8] Milad Ghaznavi, Aimal Khan, Nashid Shahriar, Khalid Alsubhi, ReazAhmed, and Raouf Boutaba. Elastic virtual network function place-ment. In Cloud Networking (CloudNet), 2015 IEEE 4th InternationalConference on, pages 255–260. IEEE, 2015.

[9] Stephen A Vavasis. Nonlinear optimization: complexity issues. OxfordUniversity Press, Inc., 1991.

[10] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and JonathanEckstein. Distributed optimization and statistical learning via thealternating direction method of multipliers. Foundations and Trendsin Machine Learning, 3(1):1–122, 2011.

[11] Caihua Chen, Bingsheng He, Yinyu Ye, and Xiaoming Yuan. The directextension of admm for multi-block convex minimization problems is notnecessarily convergent. Mathematical Programming, 155(1-2):57–79,2016.

[12] Ruoyu Sun, Zhi-Quan Luo, and Yinyu Ye. On the expected convergenceof randomly permuted admm. arXiv preprint arXiv:1503.06387, 2015.

[13] Joe Wenjie Jiang, Tian Lan, Sangtae Ha, Minghua Chen, and MungChiang. Joint vm placement and routing for data center trafficengineering. In INFOCOM, 2012 Proceedings IEEE, pages 2876–2880.IEEE, 2012.

[14] Xiaoqiao Meng, Vasileios Pappas, and Li Zhang. Improving thescalability of data center networks with traffic-aware virtual machineplacement. In INFOCOM, 2010 Proceedings IEEE, pages 1–9. IEEE,2010.

[15] Amir Varasteh and Maziar Goudarzi. Server consolidation techniquesin virtualized data centers: A survey. IEEE Systems Journal, 2015.

[16] Ying Zhang, Neda Beheshti, Ludovic Beliveau, Geoffrey Lefebvre,Ravi Manghirmalani, Ramesh Mishra, Ritun Patneyt, Meral Shirazipour,Ramesh Subrahmaniam, Catherine Truchan, et al. Steering: A software-defined networking for inline service chaining. In 2013 21st IEEEInternational Conference on Network Protocols (ICNP), pages 1–10.IEEE, 2013.

[17] Marcelo Caggiani Luizelli, Leonardo Richter Bays, Luciana SaleteBuriol, Marinho Pilla Barcellos, and Luciano Paschoal Gaspary. Piecingtogether the nfv provisioning puzzle: Efficient placement and chainingof virtual network functions. In 2015 IFIP/IEEE International Sympo-sium on Integrated Network Management (IM), pages 98–106. IEEE,2015.

[18] Aaron Gember, Anand Krishnamurthy, Saul St John, Robert Grandl,Xiaoyang Gao, Ashok Anand, Theophilus Benson, Aditya Akella,and Vyas Sekar. Stratos: A network-aware orchestration layer formiddleboxes in the cloud. Technical report, Technical Report, 2013.

[19] Xiaoke Wang, Chuan Wu, Franck Le, Alex Liu, Zongpeng Li, andFrancis Lau. Online vnf scaling in datacenters. arXiv preprintarXiv:1604.01136, 2016.

[20] Po-Wen Chi, Yu-Cheng Huang, and Chin-Laung Lei. Efficient nfvdeployment in data center networks. In 2015 IEEE InternationalConference on Communications (ICC), pages 5290–5295. IEEE, 2015.

[21] Zhen Xiao, Weijia Song, and Qi Chen. Dynamic resource allocationusing virtual machines for cloud computing environment. IEEE Trans-actions on parallel and distributed systems, 24(6):1107–1117, 2013.

[22] Wenrui Ma, Carlos Medina, and Deng Pan. Traffic-aware placement ofnfv middleboxes. In 2015 IEEE Global Communications Conference(GLOBECOM), pages 1–6. IEEE, 2015.

[23] Ermin Wei and Asuman Ozdaglar. Distributed alternating directionmethod of multipliers. In 2012 IEEE 51st IEEE Conference on Decisionand Control (CDC), pages 5445–5450. IEEE, 2012.

[24] Praveen Tammana, Rachit Agarwal, and Myungjin Lee. Cherrypick:Tracing packet trajectory in software-defined datacenter networks. InProceedings of the 1st ACM SIGCOMM Symposium on SoftwareDefined Networking Research, page 23. ACM, 2015.