LUND UNIVERSITY

PO Box 117221 00 Lund+46 46-222 00 00

Telco Clouds: Modelling and Simulation

Krzywda, Jakub; Tärneberg, William; Östberg, Per-Olov; Kihl, Maria; Elmroth, Erik

2015

Link to publication

Citation for published version (APA):Krzywda, J., Tärneberg, W., Östberg, P-O., Kihl, M., & Elmroth, E. (2015). Telco Clouds: Modelling andSimulation. Paper presented at Closer 2015, .

General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authorsand/or other copyright owners and it is a condition of accessing publications that users recognise and abide by thelegal requirements associated with these rights.

• Users may download and print one copy of any publication from the public portal for the purpose of private studyor research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portalTake down policyIf you believe that this document breaches copyright please contact us providing details, and we will removeaccess to the work immediately and investigate your claim.

Telco Clouds: Modelling and Simulation

Jakub Krzywda1, William Tarneberg2, Per-Olov Ostberg1, Maria Kihl2, and Erik Elmroth1

1Dept. of Computing Science, Umea University, SE-901 87 Umea, Sweden2Dept. of Electrical and Information Technology, Lund University, SE-223 63 Lund, Sweden

{jakub, p-o, elmroth}@cs.umu.se, {william.tarneberg, maria.kihl}@eit.lth.se

Keywords: Mobile cloud computing, Telecommunication infrastructure, Cloud infrastructure, Modelling, Simulation

Abstract: In this paper, we propose a telco cloud meta-model that can be used to simulate different infrastructure con-figurations and explore their consequences on the system performance and costs. To achieve this, we analysecurrent telecommunication and data centre infrastructure paradigms, describe the architecture of the telcocloud and detail the benefits of merging both infrastructures in a unified system. Next, we detail the dynamicsof the telco cloud and identify the components that are the most relevant from the perspective of modellingperformance and cost. A number of well established simulation technologies exist for most of the telco cloudcomponents, we thus proceed with surveying existing models in an attempt to construct a suitable compositemeta-model. Finally, we present a showcase scenario to demonstrate the scope of our telco cloud simulator.

1 INTRODUCTION

Recent technological developments have enabled aunion of telecommunication and cloud computing.Joint management of telecommunication infrastruc-ture, such as Radio Base Stations (RBS), and DataCentres (DC) may help to achieve better performanceof hosted applications and reduce the operation costs.In this paper we refer to this paradigm as telco cloudcomputing (Bosch et al., 2011).

Despite the interest in this paradigm, there are nosimulation models capable of simultaneously mod-elling the dynamics of Mobile Devices (MD), place-ment and capacity of DCs, and network infrastructure.Understanding of these relations is important for telcocloud stakeholders, e.g., Infrastructure Providers (IP)can use that knowledge to reduce infrastructure costs,while still delivering competitive performance.

We propose a meta-model of the telco cloud whichfacilitates experimentation and evaluation of possi-ble configurations, such as placement and capacityof DCs in a joint telecommunication-cloud infrastruc-ture. The meta-model uses existing, well establishedsimulation models, e.g., for Radio Access Networks(RANs) or DCs, for modelling of the aforementionedindividual parts of the infrastructure behaviour.

The contributions of this paper are: describing thedynamics of the telco cloud, including Quality of Ser-vice (QoS) and the associated costs of this paradigm(Section 4); surveying existing models of telco cloud

building blocks (Section 5); and establishing a meta-model that captures the described dynamics using ex-isting and composite models (Section 6).

This paper is structured as follows. Section 2outlines the architecture of the proposed telco cloud.Next, the simulation motivations, challenges, and re-quirements that the meta-model has to fulfill are pre-sented in Section 3. Section 4 describes the telcocloud dynamics, followed by a survey of existingmodels in Section 5. Section 6 introduces the telcocloud meta-model. Section 7 presents a showcasesimulation scenario, using a prototype implementa-tion of the introduced meta-model. In Section 8 welist a few relevant research topics that can be exploredusing the proposed model and conclude the paper.

2 THE TELCO CLOUDARCHITECTURE

In this section we present issues with current telecom-munication and cloud infrastructures, an overview ofthe proposed telco cloud topology, and how the telcocloud paradigm can help to remedy these issues.

2.1 Current Infrastructure

Currently, telecommunication and cloud computinginfrastructures are separated and managed indepen-dently. The telecommunication infrastructure is

placed in close proximity to end users and is built us-ing specialised hardware. The cloud computing in-frastructure consists of remote DCs that are signifi-cantly geographically separated from end users, con-sists of commodity hardware, and is connected withthe telecommunication infrastructure via the Internet.

We identify several issues of the current infras-tructure. Performance (especially latency) of cloudservices is not predictable, which makes computationoffloading difficult. All data processed in the cloudneed to be sent over the Internet to DCs, which addcommunication latency. For the Internet of Things,with millions of sensors generating huge amounts ofdata, the volume of traffic can cause network conges-tion. Specialised telecommunication hardware is ex-pensive and hard to upgrade.

As a consequence of the performance bottlenecks,particularly latency sensitive applications such as in-dustrial process control and Augmented Reality con-text recognition applications have mostly not yet beencloudified. The low latency, jitter free, and highthroughput connections required by such applicationscannot be provided by the existing telecommunica-tion and cloud infrastructures (Barker and Shenoy,2010). Moreover, compute and battery resources inMDs are limited and coupled. An approach to aug-menting MD’s capabilities is to offload the executionof applications to a cloud infrastructure. Such per-formance augmentation can only be seamless if theincurred communication latency is low enough andservice availability is high.

Devices are at an increasing rate gaining accessto the Internet (Atzori et al., 2010). Anything fromsmall sensors to petrol pumps, flowerpots, helmets,tumblers, windows, and spark plugs is being con-nected to the Internet to communicate and monitorits quantified performance metrics. Most of the con-nected devices are generating correlated contextualinformation, incurring large amounts of Wide AreaNetwork’s (WAN) traffic. The traffic typically con-verges to a handful of DCs for analysis and processingwhich introduces congestion in the capacity-sparseand increasingly congested RANs, core networks, andWANs. Additionally, a portion of the transported in-formation is only locally relevant and will add no orvery little entropy to the service in the Remote DC.

Wireless access network virtualisation and cloudi-fication of Telecom equipment and services proposedby (Wang et al., 2013), requires careful placement ofcompute capacity as not to introduce significant prop-agation delay. The placement of the compute nodeshas to reflect the demand for Telecom- and cloud-services in the geographic area which the RAN cov-ers. Current Telecom signalling standards and remote

Mobile Device

Radio Base Station

Radio Base Station

Controller Radio Catchment Border

Network

Proximal Data Centers

Remote Data Centre

Figure 1: Overview of telco cloud.

DCs do not interoperate well and are often not able tomeet Telecom latency requirements.

2.2 Introducing the Telco Cloud

A telco cloud is an infrastructure consisting of MDs(known also as User Equipment), stationary devices(sensors), access networks, intermediate WANs (con-necting the access networks to the backbone net-works), backbone (Internet) networks, and DCs. Wehere include two main types of DCs: Remote DataCentres (large DCs located far from the access net-works) and Proximal Data Centres (smaller DCs lo-cated close to the access networks).

The telco cloud topology paradigm proposesa closer integration between access networks anda cloud infrastructure than the current topologicalmodel where the Telecom and cloud infrastructuresare unaware of each other. When coexisting withcloud infrastructure, telecommunication functionalitycan be virtualised and augmented to the adjacent DCs.When virtualising RAN functions large portions of anRBS and the RAN control functions can be executedin a DC (Baroncelli et al., 2010).

Cloud capacity will reside in geographically dis-tributed DCs. The telco cloud topology is com-posed of multiple DCs that are dispersed in a mesh-structure, ranging from complete adjacency with theTelecom infrastructure, Proximal DCs, to more tradi-tional, Remote DCs, as depicted in Figure 1.

A group of telco cloud DCs can be provisionedand load balanced as one resource or act as indepen-dent DCs. In the former case, a control plane willneed to coordinate services and the shared resourcesby for example optimising locality and proximity toall entities by geographically placing services accord-ingly. Proximal DCs will thus have to be managed in

a distributed and coordinated manner.The cloud services hosted in Proximal DCs are ac-

cessed through the RAN. The MDs are assumed topossess varying degrees of mobility, with an equiv-alent likelihood of passing between a particular setof macro/micro-RBS1 over time. In an effort to beable to enforce DC management constraints and toglobally avoid unnecessary migration, an MD is notstrictly associated with the closest DC or the DC thatits current RBS cell is associated with.

The telco cloud infrastructure topology will needto reflect the capacity and latency objectives of thevirtualised RBSs, and to give access to the telcocloud-hosted services given the networks local capa-bility and diversity at a sufficient service level. Theprevailing 4G/LTE topology is centred around hierar-chical macro- and micro-cells which very much re-sembles that of its predecessors. The telco cloudtopology will evolve with mobile access technologygenerations, shifting and/or dispersing compute ca-pacity at various levels of mobile, Metropolitan AreaNetwork (MAN), and WAN networks to best suit theprevailing services and throughput channels.

2.3 Benefits of the Telco Cloud

We identified four main benefits of the telco cloud.First, it provides cloud applications with better andmore predictable performance. Second, it supportscomputation off-loading for resource-bounded MDs.Third, the telco cloud reduces network utilization byprocessing part of data closer to its producer or con-sumer. Fourth, it enhances cost-efficiency and flexi-bility of telecommunication infrastructure.

Thanks to a geographically distributed cloud in-frastructure, application developers and telecommu-nication operators can take advantage of the signif-icantly lower round-trip times. Moreover, we ex-pect that the average network throughput will increasewhen communicating with Proximal DCs in compar-ison to Remote DCs. Additionally, users offloadingMD applications will benefit from a low latency com-munication with a DC, where the code is executed.

The large amount of information generated bysensors can be filtered through the intermediate hier-archical cascade of DCs to prevent congestion in theintermediate WAN. Data that is only locally relevantcan be kept and consumed locally, while redundantand highly covariant information can be more easilyidentified in a local context and discarded.

Through the telco cloud traditional proprietaryhardware-bound telecom services can be virtualised,

1What is considered a traditional rural/urban cell, con-stituted by a high power RBS, mounted on a tower.

migrated, and executed in Proximal DCs. MultipleRBSs can be consolidated to increase the aggregateutilisation of the infrastructure. Executing RBSs oncloud infrastructure will allow for greater use of cost-effective commodity hardware and generic software.With the availability of the telco cloud, we expectthat an RBS in future mobile infrastructure genera-tions will only consist of a radio interface. The man-agement of the RAN, individual radio channels, sig-nalling, services, and signal processing, will be all vir-tualised and executed in a Proximal DC.

3 SIMULATION CHALLENGES

Simulation of a telco cloud is motivated by severalfactors, e.g., lack of existing infrastructure and appro-priate control plane standards. The desired simulatorhas to fulfil many requirements, such as, to be able tosimulate hundreds of thousands of various entities atfine grained time granularity (milliseconds) for longperiods of time (hours). We here motivate and de-scribe identified requirements and challenges in sim-ulation of the telco cloud. Addressing the challengesand fulfilling the requirements is crucial while design-ing a meta-model and implementing a simulator.

Telco cloud stakeholders will benefit from beingable to investigate the consequences of possible in-frastructure configurations. For example, IPs respon-sible for building and maintaining the infrastructuremay use the simulator for planning placement and ca-pacity for new DCs, as well as modifying capacityand connectivity of existing DCs. Service providersthat use infrastructures to host services are interestedin comparing different strategies for placement of ap-plication components. Moreover, developers that im-plement mobile applications utilising a telco cloud,need to determine what application components couldbenefit from offloading. To answer these and simi-lar questions, tests with various infrastructure config-urations need to be performed and results compared.There are two options for performing these tests: bysimulation or by running them in real test beds.

Currently, there is no existing infrastructure thatcan be used for testing the telco cloud. Creatinga physical test bed for large-scale testing of a telcocloud in different configurations is economically in-feasible and small-scale test beds will not be able tocapture phenomena occurring in reality, e.g., user mo-bility patterns or latency issues.

For these reasons, we believe that simulation isthe most feasible option to evaluate the telco cloud.However, we have identified several requirements thatmake simulation of telco clouds challenging. First of

Workload- Application- Mobility

Objectives- QoS- Cost

Setup- Topology- Capacity- Service placement

Figure 2: Dependencies between elements of a telco cloud.

all, the scale of simulation is large in terms of numberand types of entities. The simulation of telco cloudhas to concurrently cover hundreds of thousands ofMDs moving around a simulated area, each gener-ating requests; hundreds of RBSs providing an ac-cess to the core network; and tens of DCs, runningservices that process requests. Another challenge isthe ratio between time precision and length of simu-lation. We are interested in a very fine-grained latencysimulation, that requires precision of at least millisec-onds. However, to capture the daily patterns of MDmovement (e.g., moving between home, work, andshops) caused by migrations of users carrying them,the whole system needs to simulate several run-timehours. Moreover, a simulation of application stateful-ness, and of transferring those states when MDs aremoving, have not yet been described in the literatureand requires new models to be developed.

4 TELCO CLOUD DYNAMICS

Before constructing a meta-model of the telco cloudwe first need to understand telco cloud fundamen-tal dynamics, construed as the relations between sys-tem’s input, configuration and output. Later, we willuse the knowledge of these dynamics to build the in-tended simulation meta-model.

Figure 2 visualizes the dynamics inside the telcocloud. The workload is an input to the system that IPsdo not have influence over. It includes: applications,with a request generation model (rate and size), a re-source requirements model (the amount of resourcesneeded to process a request), and application stateful-ness (the overhead of transferring user’s state betweenDC); as well as, the mobility of users carrying MDs.

Next, objectives describe required output charac-teristics of the system. We have identified two funda-mental objectives. Firstly, QoS, which imposes per-formance requirements, e.g., latency or throughputthrough Service Level Objectives (SLO). Secondly,the monetary cost for IP associated with energy con-sumed for computation and maintenance of an infras-tructure. The objectives can be used when construct-ing an optimisation problem with QoS as conditionsand cost as the function that should be minimised.

Finally, setup is that part of the system that canbe adjusted by designers or operators to achieve de-sired objectives when processing existing workloads.Setup includes topology, location, capacity of DCsand the network that interconnects MDs and RBSswith DCs, as well as resource management policiesthat control placement and migration of services.

The above mentioned elements are all dependenton each other. First, the setup of a telco cloud is re-lated to the existing workload. The capacity of a DC isdefined by the resource demands of the services, e.g.how memory-, CPU-, network-, and disk-intensivethe services are. The locations and topology of DCsare defined by the geographic and demographic scopeof the services, the number of MDs that reside in thatdomain, and the capacity of the associated telecom-munication infrastructure.

Secondly, workload influences objectives. E.g.,user mobility is inducing delay during service migra-tion and potentially causes QoS penalties. Moreover,application statefulness introduces additional costs ofstoring data in a DC and transmitting data betweenProximal DCs. It may also increase latency due to anadditional time necessary to send the state data beforeprocessing of the request can begin.

Finally, objectives depend on the setup: QoS isproportional to the proximity and capacity of DC –a smaller DC catchment (the geographic area the DCserves) translates to greater locality and reduced prop-agation latency, while higher capacity allows host-ing of more services. Moreover, the capacity andcatchment of DCs determine telco cloud costs. Costsare proportional to dispersion of computing capacity.Firstly, there is an overhead of each DC, irregardlessof its capacity, e.g., building, cooling infrastructure,and connection to energy or network. Secondly, dis-persion increases costs of maintenance, e.g., techni-cians have to travel between locations. Therefore,costs are proportionally higher in smaller, dispersedDCs because of high initial costs and proportionallylower in huge, centralized DCs due to the economy ofscale (Armbrust et al., 2010).

5 EXISTING MODELS

To support creation of a meta-model that incorporatesworkload, setup, and objectives of the telco cloud de-scribed in the previous section, we here survey exist-ing models in the following categories: application re-quest generation and resource requirements, MD mo-bility, networks, DCs, and infrastructure costs.

Most of the models and simulators are assignedto only one of the above mentioned categories, how-

ever capabilities of four surveyed simulators extend tomany categories, so we summarise them in Table 1.

Table 1: Overview of surveyed simulators.

Framework RG RR M N DCNS-3 X X XOMNeT++ X X XCloudSim X XGreenCloud X X

RG – Request Generation, RR – Resource Require-ments, M – Mobility, N – Network, DC – Data Centre.

5.1 Workload Models

Applications running in the telco cloud consist of amobile client and a server processing offloaded com-putations. Therefore, they should be modelled fromtwo perspectives: request generation that describeshow requests are created and sent to the DCs; and re-source requirements that describes how much compu-tational resources are needed to process the requests.

Request Generation

Request generation models capture the user’s be-haviour by primarily representing interaction times orthe timing clicks through a stochastic process, oftenPoissonian in nature. A user behavioural model canbe further refined by introducing a stochastic modelfor how long time a user consumes a certain type ofcontent. Additionally, the transition between types ofcontent is often modelled as a Markov process.

Furthermore, the request generation characteris-tics are commonly modelled with multiple stochas-tic processes, encompassing the number of packetsin a session, and the size of each packet. Requestgeneration models are either closed or open looped.In an open loop model, the generation of each newsession is typically a Poisson process independent ofthe resulting DC action. Conversely, in a closed loopmodel, the generation of new sessions is dependenton timing of the response from the DC and thus theproperties of the previously generated session.

In the packet-level event driven network simula-tors, NS-3 (Riley and Henderson, 2010) and OM-NeT++ (Varga et al., 2001), a node can act as eithera client or server, by the mechanism of either send-ing packets provided by a stochastic model, at a givenrate, within a certain time period, and at a certain in-terval, or processing received packets from a buffer,at a given rate. Both server and client models can beaugmented with a more complex system of queues tosuch an extent that they can represent an abstract DCthat hosts multiple applications.

Resource Requirements

CloudSim (Calheiros et al., 2011), which is a simu-lator of cloud infrastructure, provides an applicationmodel that describes computational requirements –the amount of resources that needs to be available(e.g. number of cores, memory and storage); andcommunicational requirements – the amount of datathat needs to be transferred. GreenCloud (Kliazovichet al., 2012), which is a packet level simulator basedon NS-2, apart from computational and communica-tional requirements, describes also QoS requirements,expressed by an execution deadline. The applicationmodel may also include the size of the code that hasto be offloaded and dependencies on other services,e.g., in terms of amount of data that has to be sent orreceived (Kovachev, 2012).

Mobility

The NS-3 and OMNeT++ nodes described above canbe set into motion given a certain stochastic mobil-ity model. They can for example traverse the spaceas pedestrians, or auto mobiles, with correspond-ing velocity and rate of change. The spatial rela-tionship between nodes and RBS affects the prevail-ing channel properties and RBS-to-node associations.Node mobility will also result in handover betweenRBSs, which in turn will alter the paths of the node-generated workload in the network.

5.2 Setup Models

Below, we describe existing models and simulators ofnetworks and DCs, which can be used to configure thesetup of the telco cloud meta-model.

Network

There are several well-established event-drivenframeworks that are capable of modelling computernetworks, mobile networks, applications, packet-levelnetwork traffic, infrastructure, and independent users.The two primary examples are, mentioned in the pre-vious section, NS-3 and OMNeT++. These two arecommonly deployed in academic network researchand provide detailed results on network utilisation,throughput, congestion, and latency.

Both NS-3 and OMNeT++ are comprehensivepacket-level network simulation frameworks that in-clude wired and wireless standards, and are able tosimulate communication channel conditions. Fur-thermore, both frameworks have detailed models forchannel definition, such as propagation delay, inter-ference, data rate, and medium access schemes. In

addition, to a varying degree, NS-3 and OMNeT++,by default or through extension, support control planesignalling for a number of wireless standards andcomplex network topologies.

Both frameworks have support for modelling dif-ferent types of network nodes, ranging from com-puters to routers and switches. Each edge and nodepair has a defined communication and medium accessstandard, such as TCP/IP and Ethernet. Each packetthat is sent over the network is treated in accordancewith the prevailing network and transport protocolsand routing standard. In both, the event of arrival anddeparture of packets drives the simulation clock.

Furthermore, they require detailed configurationof all communication modes and node behaviour,making it very time-consuming to implement and ver-ify systems with different levels of abstractions, andare thus cumbersome to model abstract systems.

The telco cloud topology is yet to be definedwith unspecified control planes, it would thus becounter-intuitive and time consuming to implementtelco cloud topologies in either NS-3 or OMNeT++.In some instances, some modules would have to becompletely redesigned, and others would have to bespecified to a much greater detail than the telco cloudcan offer at this stage.

Data Centre

The purpose of this section is to survey the DC mod-els that are the most suitable for inclusion in thetelco cloud meta-model. An extensive list of mathe-matical models, simulation approaches, and test bedscan be found in (Sakellari and Loukas, 2013), while(Ahmed and Sabyasachi, 2014) provides a survey oftwelve cloud simulators. After careful examination,we chose the ones that best suit our goals.

We compare DC models and simulators based ondescriptions provided by the authors of the simulators.For each model we describe: Resource Provisioning– what resources are included and how they are mod-elled; QoS – what performance indicators are mea-sured; Costs of computation in the DC; Performanceof simulator – an estimation of the time needed to per-form a simulation.

CloudSim is an event-based simulator imple-mented in Java, for simulation of cloud computingsystem and application provisioning environments.Resource Provisioning. The CloudSim simulationlayer offers dedicated management interfaces forCPU, memory, storage and bandwidth allocation, aswell as, defining policies in allocating hosts to Vir-tual Machines (VM) – VM provisioning. Hosts aredescribed by processing capabilities (in MIPS) anda core provisioning policy, together with an amount

of available memory and storage. A model sup-ports time-sharing and space-sharing core provision-ing policies on both host and VM levels.Latency (QoS). The latency model is based on con-ceptual networking abstraction, where the communi-cation delays between each pair of entity type (e.g.host, storage, end-user) are described in a latency ma-trix as a constant value expressed in simulation timeunits (e.g. milliseconds).Costs. CloudSim provides a two-layered cost model,where the first layer relates to Infrastructure as a Ser-vice (IaaS), with costs per unit of resources, while thesecond one relates to Software as a Service (SaaS),with costs per task units (application requests). Thismodel allows calculation of the costs of using thecloud from the end-user perspective or the revenuefrom the IP perspective.Performance. CloudSim is able to perform large-scalesimulations, e.g., it can instantiate an experiment with1 million hosts in 12 seconds. Moreover, memoryusage grows linearly with the host number and evenwith 1 million hosts it does not exceed 320 MB.

CloudAnalyst (Wickremasinghe et al., 2010) isa simulator of geographically distributed large-scalecloud applications, developed with Java and thatutilises CloudSim and SimJava.Resource Provisioning. Cloud Analyst uses the sameresource provisioning model as CloudSim.Latency (QoS). A latency model allows configurationof network delays, available bandwidth between re-gions, and current traffic levels. CloudAnalyst facil-itates experiments with latency by producing follow-ing statistical metrics: average, minimum, and maxi-mum response time of all user requests; and responsetime grouped by time of the day, location, and DC.Costs. CloudAnalyst supports calculation of costs forusing cloud resources, such as cost per VM per hourand cost per Gigabit of data transfer.Performance. To improve performance of simulationentities are grouped at three levels: clusters of users,cluster of requests generated by users, and clusters ofrequests processed by VM.

GreenCloud is a packet level simulator based onNS-2, for simulation of energy-aware clouds.Resource Provisioning. Servers are modelled as a sin-gle core node with defined processing power limit (inMIPS or FLOPS), size of memory and storage, andimplementing different task scheduling mechanisms.Latency (QoS). Full support for the TCP/IP protocolreference model is provided and thanks to that thesimulator is able to calculate communication latencywith high accuracy.Costs. GreenCloud allows detailed modelling of en-ergy consumption by implementing energy models

for every DC element.Performance. Given that GreenCloud has to simulatethe full stack of Internet protocols, each simulationonly takes in the order of tens of minutes for a DCwith a few thousands of nodes.

5.3 Costs Models

The above mentioned DC models focus mostly on thecosts of running applications in DCs from the end-user perspective. Since we want to model costs ofDCs from the IP point of view, additional models areneeded for capital expenditures (CAPEX) and operat-ing expenditures (OPEX).CAPEX includes costs of infrastructure that needs tobe built and servers that have to be bought.Infrastructure Costs. Costs of building, power distri-bution, (and cooling can be estimated using a follow-ing equation: $200M · (1+ cm)/ai, where cm is thecost of money2, and ai is the time of infrastructureamortisation [in years] (Greenberg et al., 2008).Server Costs. Costs of servers can be modelled asns · ps ·(1+ cm)/as, where ns is the number of servers,ps is the price of one server [in $], cm is the cost ofmoney, and as is the time of server amortisation [inyears] (Greenberg et al., 2008).OPEX consists of power and personnel costs.Power Costs. To estimate costs of power, the follow-ing equation can be used, ns · pcs/1000 ·PUE · pKWH ·24 · 365, where ns is the number of servers, pcs isthe power consumption of one server [in W], PUEis Power Usage Efficiency, and pKWH is the price ofelectricity [in $ per KWH] (Greenberg et al., 2008).Personnel Costs. Costs of personnel can be calcu-lated using M1 ·C1 +M2 ·C2 +M3 ·C3, where M1 isthe number of IT personnel per rack, M2 is the num-ber of facility personnel per rack, M3 is the numberof administrative personnel per rack, and C1, C2, C3are the average costs per person for each of the abovementioned categories (Patel and Shah, 2005).

6 TELCO CLOUD META-MODEL

We here detail how we have composed the above sur-veyed models into a telco cloud meta-model. Figure 3depicts the visualisation of the meta-model. MDs,such as cell phones or laptops, are carried by end-users, who are in motion. The MDs generate requestswhich are sent over the network to a DC. It is alsopossible that requests are generated by sensors that

2Cost of money is the rate of interest or dividend pay-ment on borrowed capital.

Radio Base Station

Data Centre

User StateApplication Queue

Request

Figure 3: Visualisation of telco cloud meta-model.

may be static (e.g. traffic cameras) or mobile (e.g.trains). The requests are processed in the DC and theresponse is sent back to the MD or sensor if applica-ble. Processing requests, in case of statefull applica-tions, generates a user state that has to be migratedwith the end-user if he moves to the catchment of an-other DC.

The primary objective of the meta-model is to cap-ture the interactions between application workload,MD mobility, network topology, and DC character-istics, and their influence on QoS and costs of telcocloud. The parameters that define the meta-model arepresented in Table 2 and described in detail below.

6.1 Workload Model

The first group of parameters in Table 2 describes themobility of end-users carrying MD and the character-istics of requests generated by these MDs.

Request Generation

Many services may run in the telco cloud at the sametime and their number is defined by Nser. We modela service application as a stateful web service. Eachsession is separated in time with a Poisson processλses (Reyes-Lecuona et al., 1999). Each session pro-duces Nreq requests, sampled from an inverse Gaus-sian distribution, where each request is separated intime by Log-Normal distributed delay Dreq in sec-onds. The size of each request is given by Sreq KBand is drawn from a Pareto distribution.

Resource Requirements

To model application resource requirements we pro-pose a linear model specifying the needed amount

Table 2: Fundamental meta-model parameters.

Type Parameters Unit DescriptionWORKLOAD

RequestGeneration

Nser Total number of servicesλi

ses, where i = 1,2, . . . ,Nser s Session arrival rate to DCNi

req, where i = 1,2, . . . ,Nser Number of requests per sessionSi

req, where i = 1,2, . . . ,Nser KB Size of requestsDi

req, where i = 1,2, . . . ,Nser s Inter-request time

ResourceRequirements

CPU iidle, CPU i

req, where i = 1,2, . . . ,Nser MI CPU cycles used by servicememi

idle, memireq, where i = 1,2, . . . ,Nser MB Size of memory used by service

diskiidle, diski

req, where i = 1,2, . . . ,Nser MB Size of storage used by servicestatei, where i = 1,2, . . . ,Nser MB Size of user’s state per request

Mobility NMD Number of Mobile Devicessi

t , ait , θi

t , ωit , where i = 1,2, . . . ,NMD Movements of Mobile Devices

SETUP

NetworkNRBS Number of Radio Base StationsdRBS m Dimensions of an RBS cellDnet s Cumulative network delay

Data Centre

NDC Number of Data CentresNi

S, where i = 1,2, . . . ,NDC Number of servers in Data CentreN j

CPU , where j = 1,2, . . . ,NiS Number of CPUs per server

s jCPU , where j = 1,2, . . . ,Ni

S MIPS CPU’s speedmemory j, where j = 1,2, . . . ,Ni

S MB Amount of memory per serverstorage j, where j = 1,2, . . . ,Ni

S GB Amount of storage per servernetworki

bw, where i = 1,2, . . . ,NDC Mb/s Network bandwidthtinit , tidle, tterm s Times of VM transitions

Service Placement placement ={every, n-closests, . . .} Service placement policyOBJECTIVES

Qualityof Service

RT i, where i = 1,2, . . . ,Nser s Application response timeT Pi, where i = 1,2, . . . ,Nser req/s Application throughput

Costs Cost $ Total costs of infrastructure

of resources, both for an idle service and per pro-cessing each request. An idle service uses CPUidleCPU operations, memidle amount of memory, anddiskidle amount of storage. Additionally for each pro-cessed request, the service uses CPUreq CPU opera-tions, memreq amount of memory, and diskreq amountof storage. The amount of user’s state data createdby each request is defined by state and expressed inabsolute value or percentage of request size Sreq.

Mobility

The network is populated by NMD MDs, each sub-scribing to a subset of the Nser available services. The2-dimensional, multi modal, mobility model detailedin (Bettstetter, 2001) provides us with an on-averageuniform distribution of users, with movement propor-tional to the duration of a session and the scale of themobile network. The aforementioned model definesthe properties of an MD’s movement. A MD’s mo-

mentary movement is defined by its velocity consti-tuted by the current speed s and current direction θ.Changes in mobility are defined by multiple stochas-tic processes that describe the duration of its state. Anentity’s speed s is independent of direction θ and ismaintained for Ts seconds, after which acceleration ais applied between amin and amax for time Ta, until itreaches smin or smax. Furthermore, direction θ is main-tained for time Tθ until the next change-event wherethe direction θ is altered for Tω seconds with at therate of ω radians per second. Ts, Ta, Tθ, and Tω, de-scribing the timing of each change-event, are set foreach mobility mode, and are each defined by a proba-bility distribution bounded by a maxima and minima.

6.2 Setup Model

The second group of parameters in Table 2 charac-terises the network and DCs.

Network

In our model, the core network introduces a cumu-lative propagation, switching, and routing delay andit is modelled with a Weibull delay Dnet in multiplesof the number of network nodes between the sourceand the destination (Papagiannaki et al., 2003). Thenetwork distance between RBSs is equal to the cell di-mension dRBS. The associated RBSs are equidistant totheir common DC, and are for the sake of simplicityassumed to be separated by one network edge.

Furthermore, forthcoming cell planing practicesaim to increase area energy efficiency by favouringsmaller cells in urban areas (Shahab et al., 2013;Fehske et al., 2009). Our model employs a smallhomogeneous mobile network composed of NRBSequidistantly distributed RBSs.

In the absence of a specific mobile generationalstandard, an MD is handed over between RBSs at thegeographic point where they cross the cell boundarydistinguishing two independent RBSs defined by thewidth of the rectangular cells dRBS.

Data Centre

The DC model captures the influence that its capac-ity has on performance and costs of computation, asdescribed in Section 4.

To capture the influence on performance, quan-tity and quality of each DC resource is described.DC consists of NS servers, that can differ in spec-ification. Server contains NCPU CPUs capable ofexecuting sCPU operations in every second. Valuesof memory and storage specify the total amount ofavailable memory and storage, respectively. The net-work bandwidth is specified with networkbw. The DCmodel includes also a provisioning model, that de-scribes how available resources are shared among sev-eral applications, e.g., time-sharing or space-sharing.

A DC hosts services in VMs. A service can bedistributed over multiple VMs. Incoming workloadis load-balanced by either a method of round-robin,random selection, or placed in the VM with the lowestload. However, a user requests are always forwardedto the VM that served his first request. A service canspecify a minimum and maximum number of VMs itrequires. The DC scales the application within thesebounds based on the load-balancing outcome.

To emulate the life-cycle of a VM we have definedsix VM states described in Table 3. The transitionsbetween the states are presented in Figure 4. At thebeginning all VMs are in INACTIVE state. A VMis initiated when the first request arrives to a DC. Ittakes tinit seconds before VM is ready to start process-ing requests or receiving migrated requests and user

Table 3: States of Virtual Machine.

Name DescriptionINACTIVE VM is turned off.

INITIATING VM is booting up.PROCESSING VM is serving requests.

IDLE VM is waiting for requests.MIGRATING VM is transmitting data.

TERMINATING VM is shutting down.

Inactive Initiating Idle

Processing

Migrating

Terminating

Figure 4: Transitions between Virtual Machine states.

state from other DC. We assume that a VM is not ableto process requests and handle migrations at the sametime, so it changes state between PROCESSING andMIGRATION over the time. Moreover, we give mi-grations a higher priority than processing, so process-ing is paused if there are any migrations to perform.When there are no requests to process and no migra-tions to handle a VM goes into IDLE state. A VMis terminated if IDLE state lasts for longer than tidleseconds, and the VM termination takes tterm seconds.

Service Placement

Service placement policies define in what DC(s) a ser-vice should be hosted, what number of replicas shouldbe running, and when a service should be migratedbetween DCs. These decisions depend on the mo-bility of users, the size of users’ state that has to bemigrated, and QoS requirements. For example, a ser-vice can be hosted in n Proximal DCs closest to themajority of its users (n-closests), or in the case of la-tency sensitive services in every Proximal DC that isneeded to provide acceptable QoS (every).

6.3 Objectives Model

The third group of parameters in Table 2 describesQoS and costs of the telco cloud.

Quality of Service

Combining the resource requirements model, whichdescribes the amount of resources an application

needs, with a DC model, allows to simulate how col-location of different services in a DC influences theirresponse times RT i and throughputs T Pi.

Costs

In our opinion the cost models available in the liter-ature and described in Section 5.3 are very ”countrydependent”, because of the inclusion of variable pa-rameters such as salaries, costs of energy or costs ofproperty. They are also not taking into account pa-rameters important from the perspective of the telcocloud, such as the size of DC. Therefore, we modelthe costs of the telco cloud using a basic heuristicbased on observation that dispersion of infrastructurecauses additional costs, e.g.: increase of administratortravel time between locations, and higher unit costsof computation in proximal DCs because of smallerscale and high initial costs.

Cost ∝NDC

∑DC NS

(1)

As shown in Equation 1, the total cost of a telcocloud is directly proportional to the number of DCsand inversely proportional to the total number ofservers in all DCs. It means that distributing the samenumber of servers among many DCs is more expen-sive than placing them in one DC.

6.4 Limitations

The proposed meta-model has several limitations.The application model assumes that all requests gen-erated by one application are homogeneous and eachof them consumes the same amount of resources. Themobile access network model does not take into ac-count the physical layer, channel provisioning, andcell load balancing. Additionally, the radio accessnetwork functions as a mechanism to associate MDswith DCs propagation and system processing delaysare thus not modelled.

7 SIMULATION SHOWCASE

We have implemented a coarse grained simulator us-ing SimJava (Howell and McNab, 1998) as the under-lying event-driven simulation framework. All mod-ules are implemented from scratch but are based onthe meta-model presented in Section 6. The simula-tor fully implements the proposed request generationand network models, but has implemented more ab-stract mobility, resource requirements, DC, and ser-vice placement models.

To demonstrate the scope of the telco cloud meta-model and the simulator we introduce an elementaryshowcase scenario below. The scenario is designedto reveal the basic relationship between workload –MD mobility, setup – Proximal DC catchment, andobjectives – the aggregate utilisation of a telco cloud.

7.1 Experiments

For the sake of clarity we present a simplified sce-nario. Only one service in considered and the size ofsimulation is reduced when compared to the desiredscale. The VM scalability and placement models arejust a proof-of-concept. The goal is to obtain clearconclusions about the relation between MD mobilityand DC catchment, and avoid the interference of otherelements. The scenario is described in detail below.

The telecommunication infrastructure is com-posed of 16 RBSs, in a 4x4 layout. The cells aretangent but not overlapping and are dimensioned as atypical LTE micro-cell at 750 m, as detailed in (Sha-hab et al., 2013). The number of DCs varies be-tween the experiments and thus so, also the DC catch-ment defined as the ratio between DCs and RBSs,changes between (1:1) and (1:16). In abstract terms,the (1:1) catchment represents a setup with one Prox-imal DCs per RBS. In contrast, the (1:16) catch-ment approaches a more traditional case of RemoteDC serving all users in the domain.

To reveal the effects of DC catchment, all DCs areof the same capacity. The number of VMs in each DCis scaled proportionally to the number of users theyserve. The DC in the (1:16) catchment scenario has16 VMs, while the DC in the (1:1) scenario has justone VM. The workload is balanced among availableVMs, new sessions are forwarded to the least loadedVM. To reveal the full extent of the effect of usermobility, user states and requests are strictly migratedto the geographically nearest DC.

We use a request generation model with a ses-sion arrival rate of λses described by a Log-Normaldistribution with the parameters µ = 3 and σ = 1.1.The number of requests per session Nreq is taken froman Inverse Gaussian distribution with the parametersλ = 5 and µ = 3. Inter-request time is Dreq secondsand is modelled with with an Exponential distributionwith λ = 0.1. The simulation domain is populated by480 MDs, all subscribing to the same service. Due tothe size and simplicity of the network topology in thescenario, we are deploying a Markov-based mobilitymodel. The mobility model is based on a car andis as specified in Section 6.1, with parameters from(Bettstetter, 2001). To allow the mobility and work-load models to jointly reach a steady state, the simu-

lation is run for 8 simulated hours. This results in anaverage processing load of 30%, what gives enoughmargin for migrations to complete successfully.

The user state is proportional to the aggregate sizeof that user’s sessions with the application it sub-scribes to and is defined by a 5th order AR-processwith linearly decaying parameters. In our simulation,initialisation of a VM takes tinit = 81s, similarly as form1.small VM type in Amazon EC2 (Ostermann et al.,2010). A VM is terminated if it remains in the IDLEstate longer than tidle which is equal to the mean inter-session time. It takes tterm = 21s to terminate a VM.

To investigate the influence of Proximal DC catch-ment on the performance of the telco cloud we ob-serve the life cycle of the VMs by recording theamount of time they spend in each state. We runtwo sets of experiments. In the first set, end usersare static. The second set introduces mobility.

7.2 Results

Figure 5 shows the breakdown of the mean time spentin each VM state in the system per DC catchment.With a (1:1) DC catchment the utilisation suffers fromthe proportion of time spent in IDLE state due to therelatively low request arrival rate generated by onesixteenth of all users. The inefficiency is caused bythe time the system spends in the IDLE, INITIAT-ING, and TERMINATING states. The compositionof time spent in these states changes with DC catch-ment, and is a reflection of the number of VMs ina DC and load-balancing effort. Reducing the timespent on starting and terminating VMs would freeup more resources and perhaps also make the systemmore reactive to sudden workload changes. The intel-ligent management of VM scalability and placementis clearly something that needs to be optimised.

Figure 5(b) reveals the overhead of user mobilityand the migration effort it incurs. Depending on theDC catchment, different migration dynamics comeinto play. As migrations are more frequent in the (1:1)case than in the (1:8) case, user states do not have thetime to grow as much between migrations in the for-mer case. The migration effort is therefore not a fac-tor eight lower in the (1:8) case versus the (1:1) case,but rather, they spend 26% and 47% of their time inthe MIGRATING state, respectively. The system dy-namics revealed by Figure 5(b), where at worst, 47%of the execution time is spent migrating users, pointsto the need to find scaling mechanisms for the telcocloud that take into account mobility and inactivity,so that resources can be freed dynamically for otherrevenue generating applications. A policy of strictlymigrating user states and requests to the geograph-

(a) Static users

(b) Mobile usersFigure 5: DC catchment vs. time spent in each VM state.

ically closest DC, irregardless of DC catchment, inorder to obtain minimal propagation and communica-tion latency, is suboptimal.

8 CONCLUSIONS

In this paper we present a way to combine existingmodels of user mobility, mobile and core networks,and DCs into a meta-model capable of capturing dy-namics of the telco cloud. We also implement a pro-totype simulator based on a simplified meta-model.

The meta-model can be used by telecommuni-cation operators as well as equipment developersto model existing infrastructures and to plan futurechanges. Researchers can test algorithms for resourcemanagement, e.g., migration of services between geo-distributed DCs. Also developers can benefit from us-ing the simulator to observe how their mobile appli-cations behave in telco cloud environment.

Future work will be focused on enhancing thefunctionality of the simulator to incorporate other pa-rameters from the presented meta-model. Then, usingthe simulator we would like to explore the followingtelco cloud challenges: minimising the trade-offs be-tween costs and performance of telco cloud depend-ing on the DC placement and capacity, and optimalplacement and migration of services between DCs.

ACKNOWLEDGEMENTS

This work is funded by the Swedish Research Council(VR) project Cloud Control and the European Union’sSeventh Framework Programme under grant agree-ment 610711 (CACTOS). Maria Kihl and WilliamTarneberg are members of the Lund Center for Con-trol of Complex Engineering Systems (LCCC) fundedby the Swedish Research Council. Also, they aremembers of the Excellence Center Linkoping – Lundin Information Technology (ELLIIT).

We thank Johan Eker (Ericsson Research) whocontributed with feedback during several discussionsand Tania Lorido-Botran (University of the BasqueCountry) for her critical comments on early drafts ofthe paper.

REFERENCES

Ahmed, A. and Sabyasachi, A. S. (2014). Cloud computingsimulators: A detailed survey and future direction. In2014 IEEE International Advance Computing Confer-ence (IACC), pages 866–872. IEEE.

Armbrust, M., Fox, A., Griffith, R., Joseph, A. D., Katz,R., Konwinski, A., Lee, G., Patterson, D., Rabkin, A.,Stoica, I., et al. (2010). A view of cloud computing.Communications of the ACM, 53(4):50–58.

Atzori, L., Iera, A., and Morabito, G. (2010). The internetof things: A survey. Comp. net., 54(15):2787–2805.

Barker, S. K. and Shenoy, P. (2010). Empirical evaluationof latency-sensitive application performance in thecloud. In Proc. of the 1st annual ACM SIGMM con-ference on Multimedia systems, pages 35–46. ACM.

Baroncelli, F., Martini, B., and Castoldi, P. (2010). Net-work virtualization for cloud computing. Annals oftelecommunications, 65(11-12):713–721.

Bettstetter, C. (2001). Smooth is better than sharp: A ran-dom mobility model for simulation of wireless net-works. In Proc. of the 4th ACM Int. Workshop on Mod-eling, Analysis and Simulation of Wireless and MobileSystems, MSWIM ’01, pages 19–27. ACM.

Bosch, P., Duminuco, A., Pianese, F., and Wood, T. L.(2011). Telco clouds and virtual telco: Consolidation,convergence, and beyond. In 2011 IFIP/IEEE Inter-national Symposium on Integrated Network Manage-ment (IM), pages 982–988. IEEE.

Calheiros, R., Ranjan, R., Beloglazov, A., De Rose, C., andBuyya, R. (2011). CloudSim: a toolkit for modelingand simulation of cloud computing environments andevaluation of resource provisioning algorithms. Soft-ware: Practice and Experience, 41(1):23–50.

Fehske, A., Richter, F., and Fettweis, G. (2009). Energy ef-ficiency improvements through micro sites in cellularmobile radio networks. In GLOBECOM Workshops,2009 IEEE, pages 1–5.

Greenberg, A., Hamilton, J., Maltz, D. A., and Patel, P.(2008). The cost of a cloud: Research problems indata center networks. SIGCOMM Comput. Commun.Rev., 39(1):68–73.

Howell, F. and McNab, R. (1998). Simjava: A discreteevent simulation library for java. Simulation Series,30:51–56.

Kliazovich, D., Bouvry, P., and Khan, S. (2012). Green-cloud: a packet-level simulator of energy-aware cloudcomputing data centers. The Journal of Supercomput-ing, 62(3):1263–1283.

Kovachev, D. (2012). Framework for computation offload-ing in mobile cloud computing. Int. J. of InteractiveMultimedia and Artificial Intelligence, 1(7):6–15.

Ostermann, S., Iosup, A., Yigitbasi, N., Prodan, R.,Fahringer, T., and Epema, D. (2010). A performanceanalysis of EC2 cloud computing services for scien-tific computing. pages 115–131. Springer.

Papagiannaki, K., Moon, S., Fraleigh, C., Thiran, P., andDiot, C. (2003). Measurement and analysis of single-hop delay on an IP backbone network. IEEE J. onSelected Areas in Communications, 21(6):908–921.

Patel, C. D. and Shah, A. J. (2005). Cost model for plan-ning, development and operation of a data center.

Reyes-Lecuona, A., Gonzalez-Parada, E., Casilari, E.,Casasola, J., and Diaz-Estrella, A. (1999). A page-oriented www traffic model for wireless system simu-lations. In Proc. ITC, volume 16, pages 1271–1280.

Riley, G. F. and Henderson, T. R. (2010). The ns-3 net-work simulator. In Modeling and Tools for NetworkSimulation, pages 15–34. Springer.

Sakellari, G. and Loukas, G. (2013). A survey of mathemat-ical models, simulation approaches and testbeds usedfor research in cloud computing. Simulation Mod-elling Practice and Theory, 39(0):92 – 103.

Shahab, S., Kiong, T., and Abdulkafi, A. (2013). A Frame-work for Energy Efficiency Evaluation of LTE Net-work in Urban, Suburban and Rural Areas. AustralianJ. of Basic and Applied Sciences, 7(7):404–413.

Varga, A. et al. (2001). The OMNeT++ discrete event simu-lation system. In Proceedings of the European simula-tion multiconference (ESM’2001), volume 9, page 65.

Wang, A., Iyer, M., Dutta, R., Rouskas, G. N., and Baldine,I. (2013). Network virtualization: Technologies, per-spectives, and frontiers. Journal of Lightwave Tech-nology, 31(4):523–537.

Wickremasinghe, B., Calheiros, R., and Buyya, R. (2010).Cloudanalyst: A cloudsim-based visual modeller foranalysing cloud computing environments and applica-tions. In 2010 24th IEEE International Conference onAdvanced Information Networking and Applications(AINA), pages 446–452.