This is a repository copy of GA-Par: Dependable Microservice Orchestration Framework for Geo-Distributed Clouds.

White Rose Research Online URL for this paper:http://eprints.whiterose.ac.uk/166338/

Version: Accepted Version

Article:

Wen, Z, Lin, T, Yang, R et al. (5 more authors) (2020) GA-Par: Dependable Microservice Orchestration Framework for Geo-Distributed Clouds. IEEE Transactions on Parallel and Distributed Systems, 31 (1). pp. 129-143. ISSN 1045-9219

https://doi.org/10.1109/tpds.2019.2929389

© 2020, IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

eprints@whiterose.ac.ukhttps://eprints.whiterose.ac.uk/

Reuse

Items deposited in White Rose Research Online are protected by copyright, with all rights reserved unless indicated otherwise. They may be downloaded and/or printed for private study, or other acts as permitted by national copyright laws. The publisher or other rights holders may allow further reproduction and re-use of the full text version. This is indicated by the licence information on the White Rose Research Online record for the item.

Takedown

If you consider content in White Rose Research Online to be in breach of UK law, please notify us by emailing eprints@whiterose.ac.uk including the URL of the record and the reason for the withdrawal request.

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 9, NO. 99, 2019 i

GA-Par: Dependable Microservice OrchestrationFramework for Geo-Distributed Clouds

Zhenyu Wen, Tao Lin, Renyu Yang, Member, IEEE , Shouling Ji, Member, IEEE , Rajiv Ranjan, Senior

Member, IEEE , Alexander Romanovsky, Changting Lin, Jie Xu, Member, IEEE

Abstract—Recent advances in composing Cloud applications have been driven by deployments of inter-networking heterogeneous

microservices across multiple Cloud datacenters. System dependability has been of the upmost importance and criticality to both

service vendors and customers. Security, a measurable attribute, is increasingly regarded as the representative example of

dependability. Literally, with the increment of microservice types and dynamicity, applications are exposed to aggravated internal

security threats and externally environmental uncertainties. Existing work mainly focuses on the QoS-aware composition of native

VM-based Cloud application components, while ignoring uncertainties and security risks among interactive and interdependent

container-based microservices. Still, orchestrating a set of microservices across datacenters under those constraints remains

computationally intractable. This paper describes a new dependable microservice orchestration framework GA-Par to effectively select

and deploy microservices whilst reducing the discrepancy between user security requirements and actual service provision. We adopt

a hybrid (both whitebox and blackbox based) approach to measure the satisfaction of security requirement and the environmental

impact of network QoS on system dependability. Due to the exponential grow of solution space, we develop a parallel Genetic

Algorithm framework based on Spark to accelerate the operations for calculating the optimal or near-optimal solution. Large-scale real

world datasets are utilized to validate models and orchestration approach. Experiments show that our solution outperforms the

greedy-based security aware method with 42.34% improvement. GA-Par is roughly 4x faster than a Hadoop-based genetic algorithm

solver and the effectiveness can be constantly guaranteed under different application scales.

Index Terms—service orchestration, dependability, microservice

F

1 INTRODUCTION

M ICROSERVICES are excellent building blocks for com-posing applications whose components are dis-

tributed across multiple Cloud datacenters [1] for agility,performance and fault-tolerance [2] [3] [4]. The strengths ofmicroservices help enable distributed application deploy-ments by allowing quick provisioning and updating acrossgeo-distributed Clouds. For example, a recent study basedon benchmark Docker Containers on AWS EC2 showed thatthey need much less time to start or boot up (less than 50milliseconds) as compared to Virtual machines (VMs) (30-45seconds) as well as having no or negligible memory over-head [5]. Moreover, the containerized microservices provideconsiderable freedom in personalizing users’ preferences.For example, Docker Hub has stored over ten million im-ages that are available for users to create their customizedapplications. Therefore, the diversity significantly increasesthe configuration dimension of each single microservice aswell as the scale of the search space to find a suitablemicroservice compared to the VM-based Cloud services.

Dependability is a key system concern that incorporatesattributes of reliability, availability, safety, integrity, main-

• Z.Wen, R. Ranjan and A.Romanovsky are with Newcastle Uni-versity, United Kingdom. E-mail: {zhenyu.wen, Raj.Ranjan, alexan-der.romanovsky}@newcastle.ac.uk

• T.Lin is with the EPFL, Switzerland. E-mail: tao.lin@epfl.ch• R.Yang and J.Xu are with University of Leeds, UK and BDBC, Beihang

University, China. E-mail: renyu.yang1@gmail.com, j.xu@leeds.ac.uk. Dr.Renyu Yang is the corresponding author.

• S. Ji and C.Lin are with Zhejiang University, China. E-mail:{sji, lin-changting}@zju.edu.cn

Manuscript received ???; revised ???

tainability, etc. Meanwhile security is increasingly recog-nized as an important factor of dependability and bringsin considerations of confidentiality, availability and in-tegrity [6]. Given the abstract nature of dependability, wecan take security as an example and measurable attribute.Microservice orchestration is the procedure of determiningand selecting best microservice instances to compose anapplication workflow that satisfy application’s functionaland nonfunctional requirements QoS. In this context, sys-tem dependability will be influenced by both internal andexternal causes. Internally, topological layout and the se-curity attributes’ satisfaction quantitatively determine theconsequent dependability. Externally, unpredictable envi-ronmental variables such as network jitter and noisy neigh-bors [7] also impact the dependability across geo-distributedClouds. This is due to the logical satisfaction or optimizationthat is achieved by internal examination may no longerexist. For instance, [8] [9] report that network QoS uncertain-ties of Cloud-hosted services have been widely recognizedand it is extremely difficult to accurately capture such un-certainties. Therefore, to achieve a holistically dependableorchestration necessitates both internally quantitative mea-surement and environmental considerations.

Traditionally, security experts decide the service compo-nent placement based on their domain knowledge to guar-antee the overall application security level [10]. However,such method heavily relies on human expertise and fails todeploy in large-scale applications if the application consistsof a huge number of service components. Thus, a genericmechanism to express the security risks and quantitativelymeasure the security level of a microservice is urgently

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 9, NO. 99, 2019 ii

required. However, existing security-aware algorithms forservice composition randomly compute or statically deter-mine the security level by simply checking the adoptedmeans such as the encryption algorithm or security policy,etc. [11][12]. This will result in decreased accuracy in thecontext of massive-scale microservices. Also existing worksmerely regard the dependability as mathematical optimiza-tion ignore the exploiting the external system uncertainties,resulting in less realistic orchestration. Finally, the explo-sively increasing number of workflow scale and microser-vice candidates that available for application compositionleads to computation efficiency issues by using traditionalapproaches [13] [14].

In this paper, we propose GA-Par framework to provi-sion a dependable microservice orchestration and deal withthe involved computation efficiency issue. It aims to providean optimal solution of deploying microservice workflowacross geo-distributed datacenters. We take the orchestra-tion as an optimization problem to maximize the depend-ability satisfaction in terms of security requirement and net-work QoS. We adopt a hybrid (both whitebox and blackboxbased) approach to measure relevant factors. Internally, wepropose a topology-aware security satisfaction model fromwhitebox perspective to minimize the discrepancy betweenthe user’s requirements and actual service provision. Weexploit the discrepancy of individual microservices andadjunct communications between microservices to reachlogical optimization. To achieve this, we comprehensivelyanalyze the security technologies, pertaining risks and thetarget security goals to generate a quantitative measure ofthe security satisfactory levels. Regarding environmentaluncertainties that may influence dependability, we developa statistical model considering transmission rates and ser-vice response time in a black-box manner. This will beadded upon the security satisfaction model to formulatea holistic optimization problem for an actually improveddependability. Finally, we develop a novel parallel geneticalgorithm framework, GA-Par based on Apache Spark, toeffectively find the sub-optimal solution. A multi-phasehybrid parallelization management can adaptively enableboth fitness evaluation and population reproduction withinthe genetic algorithm to reach the maximized parallelism.Experiments show that GA-Par can fulfill up to 33x timesacceleration of the execution time compared against otherparallelization approaches and the effectiveness can be con-stantly guaranteed under different application scales. Themain contributions of this work are:

• A mathematic optimization model to formulate the de-pendable microservice orchestration considering bothinternal security requirement satisfaction and environ-mental uncertainty factors.

• A measurement to quantify the satisfactory level of se-curity goals by exploiting security risks and techniquesamong interdependent microservices.

• A data-driven statistical model describe the network QoSof Cloud based containerized microservices.

• A multi-phase parallelization framework to acceleratedthe execution of Genetic Algorithms, that can signifi-cantly improve orchestration efficiency.

Organization. We firstly present the motivation and require-

Fig. 1. e-banking microservices (a representative case [15])

US Datacenter

China Datacenter

UK Datacenter

Geo-distributed Clouds Environment

Scheduling Optimizer

Resource Monitor

Metric Aggregator

Deployment Manager

Requirement Interpreter

Job Dispatcher

Orchestrator

A

A’

B

C

C’

D E User-Defined Requirement (UDR)

submit

Fig. 2. Microservice orchestration in a geo-distributed Cloud environ-ment.

ments in Section 2. We describe the topology-aware securitysatisfaction measurement in Section 3 and the external un-certainty model in Section 4. In section 5, we formalize theproblem as an optimization problem and then develop aparallel framework to solve it in Sections 6. Experiments areshown in Section 7. Following a review of related work inSection 8, the paper draws conclusions and outlines furtherworks.

2 MOTIVATION

Geo-distributed Microservices. Fig 1 describes an e-banking example, which is one of representative appli-cations that adopt microservice architecture [15]. Regard-ing the front-end, a logined user can process a paymentfrom their account, pay or request the credit cards, browseinformation and request loans or mortgage, and acquirewealth management services. The back-end data-relatedmicroservices consist of in-memory or persistent databasesand relational database that underpin the information sys-tem and services. In the financial and banking scenario,dependability is a primary consideration not only in se-curity guarantee in a single microservice (e.g., the privacyprotection in lending and mortgage, the confidentiality inin credit card, the availability in database access, etc.), butthe secure data transmission between microservices. For fur-ther resiliency and high quality of service, a global serviceprovider can spread the microservices over different datacenters or clouds. By leveraging geographically distributedarchitecture (shown in Fig. 2), if one has serious connectivityor late-timing issues, the holistic service can still operatein other replicas. In this paper, we propose a microservicesorchestration framework which ensures the dependabilityof composed microservices across geo-distributed Clouds.Dependability and security. The original definition of de-pendability is defined as accepted dependence and theability to deliver service that can justifiably be trusted.Traditional dependability for a system incorporates avail-ability, reliability, maintainability. Nowadays, applying se-

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 9, NO. 99, 2019 iii

curity measures to the appliances of a system generallyimproves the dependability by limiting the number of exter-nally originated errors [6]. Therefore, reducing or mitigatingsecurity risks can reduce the consequences of unforeseendependability threats. The measurement of security levelplays a significantly important role in quantifying how wella service functions dependably in the presence of faults orattacks over a specified period of time.

Characteristics in geo-distributed and containerized en-vironment. A geo-distributed application consists of a setof microservices with different functionalities. Each typeof microservice has a huge number of candidates withdiverse configurations in terms of resource request, networkconfiguration, QoS requirement, etc. Due to the deviationsof sensitivity to network environment among different mi-croservices, different scheme of microservice placement ingeo-distributed environments also affects the overall de-pendability. We summarize the following characteristics thatmotivate this work:

• [C1] Customers are exposed to a variety of security risks.For example, main sources of security threats includehuman-caused sabotage of network, malicious attacks,etc. Hence, the achievability of security goals is highlyaffected by how microservices can mitigate or avoidpotential security risks.

• [C2] Differentiated security-levels are provisioned by differentmicroservices. There are more than ten million imagesstored in Docker Hub. A lot of images share the samefunctionalities, but with different non-functional config-urations such as security. For example, the seven mostwell known cryptographic algorithms are: DES, RSA,HASH, MD5, AES, SHA-1 and HMAC [16]. However,these algorithms have different ease of being attacked.We assume that the more dependable microservice usesstronger security techniques. Meanwhile, microservicemay use multiple techniques to tackle security. For in-stance, a storage service may include access control,data replication, resource isolation, etc. Therefore, it isextremely desirable to judge the security levels of amicroservice by exploiting and aggregating the adoptedtechniques.

• [C3] Microservices are exposed to uncertain and unpre-dictable network environments. In container managementsystems, the network is typically configured in a besteffort manner [7]. The host machine tries to provisionnetwork resources to running containers. Consequently,the network performance interference drastically man-ifests outside the confines of strict QoS guarantee dueto unpredictable neighbors [7]. This uncertainty evenmagnifies when microservices are geo-distributed acrossmultiple Cloud datacenters through wide area Internet.In fact, network jittering with unpredictable latency issometimes difficult to capture[8] [9].

Dependable Orchestration Requirements. Based on afore-mentioned analysis, the configuration diversity in termsof security capability and network uncertainty significantlyaffect the dependability of containerized microservices de-ployment, thereby reducing the satisfaction level of user’srequirement. To achieve a dependable microservice orches-tration, we need to deal with following problems: [P1]

TABLE 1Mapping between Cloud risks and security goals

Security Goal (G) Risk (R)

Accountability R1, R3, R5, R6, R7Audibility R1, R3, R6, R7, R8Authenticity R3, R4, R5, R8Availability R1, R2, R4, R7, R8, R9Confidentiality R1, R3, R4, R5, R7Integrity R1, R3, R4, R5, R7Non-repudiation R1, R7Privacy R1, R2, R3, R5, R6, R8

how to characterize new security risks associated with themicroservices and how to bridge and exploit risks andsecurity techniques to quantify the security goal satisfactionin microservice orchestration? [P2] how to capture the envi-ronmental impact of network QoS fluctuation (e.g. through-put and response latency) on microservice selection? [P3]How to deal with the increasing computation complexityof finding the most dependable microservices compositionunder different optimization constraints? The significantlyincreased scale of application workflow and increased num-ber of microservices candidate horizontally and verticallyboost the computing difficulty.Problem Formal Definition. Formally an orchestrated ap-plication can be represented as a directed acyclic graph(DAG) G = (S,E). Vertices S correspond to the servicecomponents and edges E correspond to data transferredbetween them. More specifically, each component i in theapplication has a group of candidate microservices Si anda candidate p is denoted by sip ∈ Si. We use sip and sjhto represent the selected microservice of components Si andSj in the microservice workflow respectively. Accordingly,the dependency Ei,j can be determined by dsip→sjh ∈ Ei,j ,where the data is transferred from the candidate p of Si

to the candidate h of Sj . To tackle above challenges, weleverage whitebox- and topology-dividing based methodto quantify the achievable security of G = (S,E). Eachcandidate sjh might have varying data transmission ratesand complicated patterns (e.g. latency between request andresponse) among different microservices at different times.We use statistically significant models to measure environ-mental uncertainties in a blackbox and statistic manner.

3 QUANTITATIVE SECURITY MEASUREMENT

Security is one of the indispensable non-functional aspectsfor Cloud-based applications. Most research focuses on im-proving the security of each service component within ap-plications. SecGraph for example is a technique that enablesdata owners to anonymize their data, while measuring thedata’s utility, and evaluating the data’s vulnerability againstmodern De-Anonymization (DA) attacks [17]. However,there are a number of security techniques that are appliedto a microservice. Therefore, how to measure the securitylevel of each candidate microservice becomes difficult, whileessential for composing a set of microservices to meet thesecurity requirements.

To deal with [P1], we first uncover the relationship be-tween security level, goal, risks and the techniques/policies.A microservice may face some security risks due to variousreasons such as deploying unreliable resource, using weakauthentication techniques and being accessed by insecure

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 9, NO. 99, 2019 iv

devices etc. On the other hand, there are different securitytechniques or policies provisioned by the microservice toalleviate the corresponding risks, thereby enhancing the se-curity. Therefore, the achievement of security goals of a mi-croservice depends on both potential security risks and themitigation degree of risk through techniques. We measurethe security level of each microservice by the aggregation oftargeted security goals. In next subsections, we investigatethe relationships among the top risks within Cloud comput-ing [18], the most popular security-enforcement techniquesand Cloud security goals. We further develop a genericmethod to measure the security level of each microservicebased on the multiple mapping relationships.

3.1 Mapping between Security Goals and Cloud Risks

Basic Idea. A complete collection of security goals withinthe CIA(Central Intelligence Agency) triad (including ac-countability, audibility, authenticity, availability, confidentiality,integrity, non-repudiation and privacy) has been illustratedby [19]. Cloud security is one of the subcategories withininformation security realms, thus we consider these goalsas factors to quantify the security level of each candidatemicroservice. As discussed above, the achievement of secu-rity goals is affected by the potential security risks includedin the microservice. Since microservice technology is stillat an early but rapid development stage, there are limitedstandards or whitepaper of microservice security risk thatcan be widely accepted and generally referred to. Therefore,we currently applied the commonly-used cloud securitiesand risks into our main framework.

In the microservice scenario, the vulnerabilities can beleveraged by attackers more easily, thereby aggravating themanifestation of risks. The first reason for this is the archi-tecture evolution. As monolithic application is de-composedinto hundreds of smaller microservices, the number andcomplexity of interactions and communications betweenthem is substantially increased. Attackers can thus exploitthe decomposition and the inherent complexity to launch at-tacks against applications. For example, many separate APIsand ports per app represent numerous doors for intruders totry to access within an application. Such microservices mayalso be deployed in a cloud environment that the applicationowner is unable to control. Hence, the manifestation ofspecific risk such as loss of governance will increase ac-cordingly. Secondly, container-based microservices provideattackers more chances to break the applications’ securityguard down. For example, unlike in a VM, the kernel isshared among all containers and host, which magnifiesand disseminates the kernel vulnerabilities. If a containermanages to attack the kernel, the corresponding threat canbe spread into the other parts of the system, resulting inthe whole host breakdown. Moreover, sharing and buildingcontainer images tends to be much easier and more efficientthan sharing VM images for DevOps purposes. This char-acteristic, however, increases the chances of privacy leaks,where the malicious code can be injected in both in-housewritten code and third-party libraries by the image owner[20] and image users are unaware of this.

The following investigates the top security risks in Cloudcomputing hinder the achievement of the listed securitygoals. These security risks have been summarized in [18]:

[R1] Loss of governance, [R2] Lock-in, [R3] Isolation fail-ure, [R4] Management interface compromise, [R5] Dataprotection, [R6] Insecure or incomplete data deletion, [R7]Malicious insider, [R8] Customers’ security expectations and[R9] Availability chain. In order to obtain the correlatedmapping between security goals and Cloud risks, we ana-lyze all the potential attacks caused by each risk and inspectwhether each attack will lead to unreachable security goals.The following example demonstrates how we map a risk tothe corresponding security goals.Example. Risk [R1] describes the case where a Cloud userloses the supervision to the Cloud provider in many aspects:lack of port scans, vulnerability assessment and penetrationtesting. The provider might outsource or subcontract ser-vices to third-parties (unknown vendors), resulting in uncer-tain security guarantees. Specifically, third-party providersmight not be able to meet the organization’s requirementsof certifications and responsibilities, thereby violating theAccountability. Similarly, Availability, Confidentiality, Integrityand Privacy of data cannot be guaranteed for the samereason. Since no user audit is allowed by the Cloud provider,the risk will also be closely relevant to Non-repudiation andAudibility. Based on this domain knowledge, we can figureout the complete mapping as shown in Table 1.

3.2 Microservice Security Level

In this subsection, we use the installed risk preventiontechniques of microservices to measure the security levelsof them. Table 2 shows the notations of the rest of the paper.We assume that the security level of a microservice dependson the quantity and quality of the application of securitytechniques. For example, a type of risk can be prevented bymany security techniques, and they have a variety of per-formance when they are applied independently. Moreover,we assume that there is incremental effect by applying moresuitable techniques to a microservice. The following detailshow to calculate the security level of a microservice.

We assume that a risk Rk can be handled by a set oftechniques and policies TechRk

. Thus, the total capacityof Rk that can be prevented is:

∑j∈TechRk

Vj , where Vj

represents the performance of technique j to reduce riskRk. Next, if a microservice s offers a set of techniques andpolicies Techs, the ability of s to handle risk Rk can be

represented as Rk(s) =

∑i∈{Techs∩TechRk

} Vi∑j∈TechRk

Vj. As mentioned

in Table 1, a security goal gq corresponds to a group ofrisks Rgq . Therefore s’s capability of satisfying the securitygoal gq can be defined as gqs =

∑Rk∈Rgq

Rk(s). We use

set Gs = {g1s , .., g|G|s } to represent s’s capacity of targeting

different security goals.

3.3 Microservice Security Measurement

Take the e-banking example in Figure 1 again. We areobliged to target security goals in essential microservicessuch as payment and customer information provider, etc.and enable secure data transmission (particularly user dataand monetary information) between microservices.

Microservice Security Satisfaction. To identify and sortout those microservices that can best satisfy customers’security requirements, we extend DSD (Degree of Security

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 9, NO. 99, 2019 v

TABLE 2Notations

Symbols DescriptionsSecurity Metrics

gq ∈ G The qth goal where |G| = 8

Rgq ∈ R The qth goal’s risks set where |R| = 9Rk(si) The ability of si to handle risk Rk

gqsi

si’s capabilities of targeting security goal gq

Gsisi’s capabilities of targeting all goals

Microservice Orchestrationu A customer/usersjh A candidate h selected by microservice component Sj

usjh u has selected sjh

dsip→sjhThe transfer of data from sip to sjh

Security MeasurementDSSD(u, sjh) Degree of sjh security deficiency, between its

capability and u’s requirementsDDSD(dsip,sjh

) Degree of data security deficiency between two

conjunct microservices sip and sjhV DDSD(u, sjh) The security satisfactory degree of moving input data

from other conjunct microservices to sjh for user uV SEC(u, sjh) All security-related metric value of sjh for user uUtil The utility function for selecting a microservice

Network QoS Measurementε(t) The reduction degree of time consumed or

transmission delays by using off-peak as the baselineDTL Data Transmission LatencySUR(t) Represents the possibility that a client can get the

response from a Cloud service after tT (mrt, sjh) Customized user requirements for each service sjh.

If it meets requirements, set to 1 (available), else setto ∞

λ Optimal solution for the given workflowλ′ One of the solutions for the given workflow

AlgorithmsS The set of all microservice candidatesSi The microservices candidate set for components i

M The total population size of each generationL The numbers of components in a workflowN The number of SlavesPartition Partitions are basic units of parallelism and each

partition is responsible for an atomic logicaldivision of data.

Q′ The partition number used for Fitness CalculationQ′′ The partition number used for Genetic OperationC The number of cores on each SlaveT The number T of CPUs requested by each taskR The threshold for the number of stable iterations in

terminationiter The total number of GA iterations

Ci The ith constant valueINSj

The inputs from Sj ’s immediate predecessors

IN A set of all data dependencies in the given workflowAlgorithms Analysis

E The size of the elitism listdCollecti The time delay for collecting the local elitism list

from ith partitiondBroadcastEi

The time delay for broadcasting the global elitism list

to ith Slave

Deficiency) [21] into DSSD (Degree of Service Security Defi-ciency) in our model to describe the discrepancy betweendesired security level and the supplied level. In this context,DSSD(u, sjh) = 0 if the selected microservice sjh can fullymeet the demands of customer (u). DSSD can be formalizedas:

DSSD(u, sjh) =

|G|∑

q=1

wqj ·M(gqu, g

qsjh

)

where 0 ≤ wqj ≤ 1,

|G|∑

q=1

wqj = 1 and

M(gqu, gqsjh

) =

{

0 if gqsjh ≥ gqu

gqu − g

qsjh otherwise

(1)

where wqj is the weight of the qth security goal for com-

ponent Sj and the customer specifies the weights to reflect

their priorities among those goals. M represents the dispar-ity between sjh security levels and u’s security demands interms of each security goal.

Data Transmission Security Satisfaction. For data trans-mission between two components Si and Sj , we use DDSD(Degree of Data Security Deficiency) to depict the satisfactionof security of the data movement:

DDSD(dsip→sjh ) =

|G|∑

q=1

wqij ·M(gqsip , g

qsjh

)

where 0 ≤ wqij ≤ 1,

|G|∑

q=1

wqij = 1 and

M(gqsip , gqsjh

) =

{

0 if gqsjh ≥ gqsip

gqsip − g

qsjh otherwise

(2)

In Eq. 2, dsip→sjh indicates the data transfer from sourcesip to the destination sjh. We assume that the candidatemicroservice in component Si satisfies the expected securitylevel of customers. Therefore, M(gqsip , g

qsjh

) directly indi-cates the customer’s security satisfaction from sip to sjh.Customers can also define the priority or importance ofeach security goal when transferring data from Si to Sj byadjusting wq

ij . Thereafter, we can estimate the satisfaction oftransferring holistic data:

V DDSD(u, sjh) =∑

{dsip→sjh}∈INSj

DDSD(dsip→sjh ) (3)

where INSjis all input data pipelines of Sj from all its

immediate predecessors. Finally, we can calculate sjh’s sat-isfaction of security by Eq. 4:

V SEC(u, sjh) = V DDSD(u, sjh) +DSSD(u, sjh) (4)

4 ENVIRONMENTAL UNCERTAINTY MEASURE-

MENT

To solve [P2], we leverage a data-driven and statistical mod-eling method to capture and describe the environmentalimpact on the dependable microservice orchestration.

4.1 Data Transmission Latency

Due to the network bandwidth fluctuation, the data trans-mission latency varies between upstream and downstreammicroservices. The throughput of file transfer to run in-stances within an Amazon EC2 datacenter (US East North-ern Virginia Region) is evaluated in [22]. The experimentcaptured the network bandwidth every 30 minutes between20th and 21st May 2013. A throughput surge can be detectedduring the period 7.00-8.00 a.m. while the throughput dur-ing other periods is much less and remains stable. Therefore,we can take 7.00-8.00 am as the peak time and distinguish itfrom other off-peak times. We define a latency factor ε,

ε(t) =

{

mean(thr(t))mean(thr(t))

if t = peak time

1 if t = off-peak(5)

where thr(t) indicates the throughput of time periods t(time period except for t). According to real data statistics[22], the peak throughput is approximately twice that of off-peak throughput, resulting in ε = 1

2 . In practice, ε is config-urable and can be adjusted according to different conditions.Moreover, the total time and resource consumption also rely

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 9, NO. 99, 2019 vi

(a)

1000 1500 2000 2500 3000

Τηρεσηολδ ξ

0

500

1000

1500

2000

2500

3000

Μεαν Ε

ξχεσσ α

(ξ)

Ριγητmοστ δατα

Παρετο

(b)

0 500 1000 1500 2000 2500 3000

Τιmε (mσεχ)

0.0

0.2

0.4

0.6

0.8

1.0

ΧDΦ

Οτταωα

Βυmαβψ

Νεωχαστλε

Dυρηαm

ΝεωΨορκ

Χηιχαγο

Πεψτον

Λανσινγ

Σεχαυχυσ

ΛοωερΒουνδ

(c)

0 500 1000 1500 2000 2500 3000

Τιmε (mσεχ)

0.0

0.2

0.4

0.6

0.8

1.0

Τ

h�

�����������

��� � �v

(d)

Fig. 3. Distributions; (a) RTT from different locations (b) ME of lower bound data and the fitted distribution (c) The lower bound of the end to endnetwork delay in Cloud datacenter (d) The survival distribution of the measured datacenter

on the amount of data. We can thus define the overall DTL(Data Transmission Latency). DataSj

and Total are the inputsize of component Sj and the overall data consumed withinthe entire microservice workflow respectively. The giventime tsjh represents the local time of sjh.

DTL(tsjh ) = ε(tsjh ) ·DataSj

Total(6)

Note that the size of transferred data and the executiontime of each microserivce can be taken from provenancelogs. Numerous works [23][24] describe how to estimate thetime and size based on the execution logs.

4.2 Uncertainty Modeling

4.2.1 Data-driven Benchmarking and Observations

In order to elaborate on the uncertainty model and itsimpact on service provisioning efficiency and availability,we probe the request response latencies and regard themas a main indicator. To explore a generic distribution ofthe request response time of a specific location, we deploythe same implementation of clients that are geographicallydistributed around the world, and send requests to onecentral datacenter (in Dublin). The round trip timespan(RTT) is collected, and the experiments are repeated 1,400times for each individual client within a continuous 24-hourperiod. As shown in Fig. 3(a), requests from different citiesare sent to Dublin and we display the response time withCDF (Cumulative Distributed Function). Apparently, distanceand location have a significant impact on the distribution.In general, a longer distance indicates a higher probabilityof traffic delay. In addition to these observations, we haveto form a uniform model and find a marginal distribution tocover all distributions of different requests comprehensively, andsatisfy the worst case for each specific datacenter. In Fig. 3(a),the target distribution is supposed to be within the coloredarea, and it is highly desirable to obtain an optimal distri-bution that fits the marginal bound as closely as possible.Due to space limitations, we only briefly demonstrate howto find the required distribution.

4.2.2 Mathematical Modeling

Basically, the exponential distribution class provides a rig-orous mathematical framework. In reality, our target distri-bution belongs to a subclass of exponential distribution –heavy-tailed distribution [25]. We intend to find a regularlyvarying tail and determine the most appropriate distribu-tion with the relevant properties.

Firstly assume a non-negative random variable X , theCDF of X is F (x) = P [X ≤ x](x ≥ 0) and F (x) = 1−F (x).

The data (shown in Fig. 3(a)) conforms to Eq. 7, and thus thedistribution is subexponential (i.e. F ∈ S) [26].

limx→∞

P [X1 + · · ·+Xn > x]

P [max(X1, . . . , Xn) > x]= 1 for some (all) n ≥ 2, (7)

where X1, ..., Xn are independent and identically distributedrandom variables (IID). [27] further substantiates the impli-cation that such F is regularly varying-tailed (heavy-tailed).The heavy-tail distribution will have:

limx→∞

F (x)

e−δ·x→ ∞ ∀δ > 0, F ∈ S (8)

In fact, this indicates that the tail of F decreases moreslowly than any exponential tail. Meanwhile, for a positivemeasurable function f which regularly varies with index α,if Eq. 9 is satisfied, we can write f ∈ R(α) [28]:

limx→∞

f(t · x)

f(x)= tα ∀t > 0 (9)

Due to the heavy-tailed pattern, we conclude F (x) ∈R(−α) and further represent F (x) = x−α·l(x) where x > 0,α > 0, and l(x) ∈ R(0) is a slowly varying function. In gen-eral, Pareto, Brr and Log-gamma distributions are examplesof distribution functions with such regularly varying tails.

Afterwards, we have to determine which offers the best-fit distribution. To this end, we use the mean-excess (ME)metric (see Eq. 10) to sort out the desired distribution fromall these candidtates [29].

ME(x) = E(X − x|X > x) =

∫∞x

F (y)dy

F (x), x > 0 (10)

We extract the data which corresponds to the lowerbound in Fig. 3(a), and calculate the ME value with differentx ranges as shown in Fig. 3(b). We examine the ME functionof each distribution, and only the Pareto Type I distributionincreases linearly [30]. Specifically, the ME of a standardPareto I distribution is shown in Eq. 11 (the blue line inFig. 3(b)).

ME(x) =x

α− 1, α > 1 (11)

Therefore, we choose Pareto as our approximate fittingdistribution and use Eq. 12 to calculate the cumulativeprobability within a specified time window.

F (t) =

∫ ∞

t

α · xα

tα+1dt (12)

Fig. 3(c) illustrates the effectiveness of the applicationwith Eq. 12. The LowerBound curve can always guarantee thecoverage of the response time boundary and all measuredrequest cases. In this manner, we have the survive function(Eq. 13), which represents the possibility of a client getting

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 9, NO. 99, 2019 vii

a response from a Cloud service after t (as illustrated inFig. 3(d)). By leveraging the survive function, we quantita-tively describe the uncertainty in request handling.

F (t) = 1 − F (t) = 1 −

∫ ∞

t

α · xα

tα+1dt (13)

5 FORMULATION AND SOLUTION

5.1 Orchestration Problem

Based on the model proposed in Section 3.3 and Section4, the overall function of user u to run a microservicecandidate h in the component Sj can be expressed as:

Util(u, sjh, t) = θ · DTL(tsjh ) + (1− θ) · V SEC(u, sjh) (14)

Users can customize the weight θ to manifest their re-quired importance and tradeoff between data transmissionefficiency and the security requirements.

Single component utility function. For each componentSj , customers have their specific requirements to avoid theuncertainty. Based on our early experience, we assume thatthe survival rate of Sj ought to be higher than 10%. In orderto apply the uncertainty factor into our model, we depictT (mrt,sjh) as in Eq. 15 where mrt is the maximum responsetime that the customer can tolerate. In this formalized def-inition, if the survival rate is lower than 10%, we set theoutput to be +∞.

T (mrt, sjh) =

{

1 if F (mrt) ≥ 10%

+∞ otherwise(15)

Thus, the utility of u selecting an sjh is:

V alue(u, sjh,mrt, t) = T (mrt, sjh) · Util(u, sjh, t) (16)

where sjh ∈ Sj and Sj ∈ S.Multiple component orchestration. We assume that the

optimal orchestration is that each selected microservice sjhin microservice component Sj within an application canmeet user u’s requirements. It can be defined as:

Optimal(u, λ) =∑

sjh∈λ

V alue(u, sjh,mrt, t) = 0 (17)

where λ is one of the optimal solutions for the target orches-tration workflow. However, it is very difficult to guaranteethe optimal solution. The optimal solution may not evenexist due to the lack of candidate services. Therefore, ourobjective is to find an orchestration that minimizes the valuefrom all possible solutions Λ:

Minimize:∑

sjh∈λ′,λ′∈Λ

V alue(u, sjh,mrt, t);(18)

where λ′ is one of the solutions for the target workflow.

Definition 1 (Optimal Solution). As in the statement ofoptimization problem in Eq. 18, the number of selectedmicroservices is fixed to |λ′|. Therefore the optimal solution

can be obtained by traversing m|λ′| solutions, where eachcomponent in the orchestration workflow has m candi-date microservices. The given optimization problem can beproofed an NP-hard problem.

Conjecture 1. The dynamic programming method is not suitablefor solving our problem.

We assume that a set SK = {s1, ..., sk} is the optimalsolution of composing k types of microservice, and a setSK+1 is the optimal solution of composing k + 1 types ofmicroservices. If Sk ⊂ SK+1, the dynamic programmingmethod will take more time than the direct calculation of theoptimal SK+1. However, in our optimization problem, thissituation happens very frequently due to the impact of datadependencies. Thus, a new and more efficient algorithm isdesired to solve this optimization problem.

5.2 Basic Solution Principle

With respect to optimization algorithms, [13] demonstratedthat Genetic Algorithms (GA) outperform other solutionssuch as Mixed-Integer Non-linear Programming (MINLP) orLinear Programming (MIP), etc. The high time complexityof MIP and MINLP solvers make it infeasible to applythose in large orchestration problems. Therefore, we presentGA-Par (Genetic Algorithm Parallelism) – a novel ParallelGenetic Algorithm designed to optimize the microserviceorchestration in terms of the security under uncertainty.

In our Genetic Algorithm (GA), a string w = (β1, ..., βL)with length L = |λ′| is used as a chromosomal repre-sentation for a workflow and each βi = Si ∈ S rep-resents a component. Each gene segment derives from afinite set of microservices Si = {si1, ..., sim}. Accordingto our optimization objective, we set the fitness functionfit(λ′) =

∑sjh∈λ′ V alue(u, sjh,mrt, t). Basically, the GA

usually includes two primary parts: Fitness Calculation (FC)and Genetic Operations (GO) [31]. The effect value of eachchromosome (individual) is calculated in FC, and thenthe elites are sorted out from all population members. InGO, the new chromosomes are generated by operations:Selection, Crossover and Mutation. After iterations of severalgenerations, a sub-optimal solution can be found.

6 GA-PAR: DESIGN AND IMPLEMENTATION

To tackle [P3], we propose a novel multi-phase parallelizedGA to accelerate the orchestration procedure. We will brieflyoverview existing solutions and introduce our architectureand detailed implementations.

6.1 Parallelized GA Overview

According to the philosophy of divide-and-conquer, GAparallelism can be categorized into two approaches:

1) Fitness Evaluation Parallelism – The parallelism is con-ducted merely within the calculation of fitness value bydistributing individuals to different computing nodes (i.e,slaves). The centralized master manages all the GO opera-tions and the generated population. This method is suitablefor cases that have a time-consuming fitness function calcu-lation. Obviously, the GO will become extremely inefficientwith an increase in population scale due to the single pointof performance bottleneck. The computing power of themaster tends to be limited once dealing with the huge num-ber of individuals for generating new populations. More-over, a significant network latency might be aggravatedconsidering the communication of individual distributionwhen computing the fitness value over different slaves.

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 9, NO. 99, 2019 viii

Συβ−ποπ

ΓΑ−Παρ Σλαϖε

ΓΑ−Παρ Μαστερ

ΓΑ−Παρ Σλαϖε

Παρτιτιον

Μαστερ

Παρτιτιον

ΜαστερΠαρτιτιον

Μαστερ

Παρτιτιον

Μαστερ

ΠοπΠαρτιτιον

Μαναγερ

ΣηαρεδΕλιτιστ

Μαναγερ

Χοmπλετε ποπυλατιον (ποπ)

Συβ−ποπ Συβ−ποπ Συβ−ποπ

Συβ−ποπυλατιον

(συβ−ποπ)

Ελιτε ρεπορτ ανδ υπδατε

ΣλαϖεΑγεντ ΣλαϖεΑγεντ

Παρτιτιον δψναmιχ αδϕυστmεντ

... ...

Γενετιχ Οπερατιονσ

...

Fig. 4. The architecture and module interactions in GA-Par

2) Population Reproduction Parallelism – This approachallows the process of creating a new population indepen-dently over different slaves, and each slave generates andmanages a subset of the new population. As a result, thetime consumed in the population generation can be reduced.However, if the GO is only performed inside each isolatedcomputing node, the probability of duplicated individualswill rise. Thus, it is extremely likely to fall into a localoptimum.

Recently, [32] introduced a framework for running ge-netic algorithms with map reduce paradigm in Hadoop.However, the design is on the basis of dumping eachgeneration of chromosomes onto disk, thereby dramaticallyintroducing huge IO costs and decreasing the operationalperformance. Furthermore, a large scale workflow optimiza-tion problem requires a very large search space in terms ofhorizontal (dependencies of each components) and vertical(different attributes of each candidate microservice). Forthese reasons, in-memory data operations naturally fit inthe GA operations, especially for the crossover operation inwhich the data for each individual can be efficiently shuffledamong different nodes. We therefore propose a new parallelalgorithm GA-Par based on Apache Spark and it adaptivelyincorporates the parallelization of both fitness evaluationand population reproduction.

6.2 GA-Par Framework

6.2.1 Architecture

GA-Par adopts the master-slave architecture and two-phaseparallelization management to fully explore the efficiency ofthe fitness calculation and genetic operations. The paralleldegree can be dynamically adjusted in order to maximizethe calculation velocity whilst maintaining the optimizationquality. Mechanisms such as adaptive partition control,global population shuffling and elitist sharing are proposedto achieve these objectives.

Fig. 4 details the architecture and multiple modulesunderpinning the implementation of GA-Par. The GA-ParMaster is responsible for the overall management of par-allelization and population life-cycle (i.e. generation andexchange). The SlaveAgent is the daemon agent that runs oneach GA-Par slave node and handles instructive messagesfrom the GA-Par Master. More specifically, the populationwill be divided into a number of sub-populations and eachof them is firstly initiated on each partition. This procedureis controlled by the PopPartitionManager (PPM) with themaster. Correspondingly, on each slave node, the Partition-Manager (PM) takes charge of each sub-population andmanages the new generation through FC and GO.

Algorithm 1: : GA-Par Master

Input:S: all candidate microservicesQ′: partition number for FCQ′′: partition number for GOλ′: required composition of microservices

1 Master(S, Q′, Q′′, λ′)

2 sharedElitist← Initialize(S, λ′)3 while do not meet termination condition do4 setPartitionNumberToSlaves(Q′)5 // wait all PartitionMasters to finish FC

6 sync for FC7 // collect all elites from partitions to a list in master node

8 localElitistList← collect()9 // find the best individual

10 s← findBest(localElilistList)11 // update the shared elitist list

12 sharedElitist← update(sharedElitist,localElilistList)

13 // broadcast the shared elitist list to slaves

14 broadcast(sharedElitist)15 setPartitionNumberToSlaves(Q′′)16 // wait all PartitionMasters to produce new generation

17 sync for GO18 end19 return s

Also, the PPM coordinates the dynamic partition config-urations to make partitions fully parallelized over differentslaves whilst considering the data shuffling cost (AppendixA.1 explains the execution model). To increase the diversityof the new generation, we merge several partitions into asingle partition for population reproduction and then re-partition it appropriately to ensure the parallelization ofFC (detailed later). Moreover, the local elite results of eachpartition will be collected and synthesized into a globalelitist list by SharedElitistManager(SEM) in GA-Par Masterbefore the selection phase. The manager will then broadcastthe global elitist list to the running partitions to ensureeach of them has the best chance to generate advantageoffsprings.

We adopt Spark to implement the aforementioned func-tionalities. Compared with Hadoop, Spark can provisiona more flexible approach to manipulate data via ResilientDistributed Datasets (RDD), which a collection of elementsthat will be partitioned across different slaves [33]. More-over, the in-memory computation on large clusters strikesthe balance between the maximized parallelism and thediversity control.

6.2.2 Two-phase Parallelization

A candidate microservice within a workflow componentmust satisfy all security and efficiency requirements, whileguaranteeing service availability. Herein, we describe theasynchronized messaging between GA-Par Master and Par-tition Managers and the main procedures on both sidesrealize the adaptive partition operations. GA-Par Masterwill finally output the optimal orchestration solution.

Alg. 1 and Alg. 2 depict the collaborative and interactiveprocedure of the master and pertaining slaves. The master

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 9, NO. 99, 2019 ix

will read candidate list, configurations before initializing theshared elitist information. Regarding the slave, after receiv-ing the candidates from the GA-Par Master, individuals willbe initialized on Q partitions, followed by the GO withineach partition. The population size per partition is M

Q, where

M is the total population size of each generation, whichhas been pre-defined. Q is the number of partitions selectedfrom the set {Q′′, Q′}, where Q′′ < Q′. Q can thereforebe dynamically adjusted in different execution phases toensure the tradeoff between parallelism and diversity. Inparticular, during the FC phase, in order to rapidly finishthe FC computation and obtain the elitist list, the degree ofparallelism should be maximized to Q′. In other words, theindividuals of the whole population (M size) are distributedto the maximal number of partitions, in order to fully utilizethe computational resources. On the other hand, in the GOphase, individuals within the same partition are involved toproduce a new generation. Therefore, the more individualsinvolved, the more diverse generations can be produced. Asa result, there are increasing chances to obtain individualadvantage. Accordingly, as shown in Alg. 1 (Lines 4 and 15)and Alg. 2 (Lines 6 and 13), the partition number is shrunkfrom Q′ to Q′′ to guarantee the offsprings’ diversity.

It is noteworthy that the elitist list is computed by theextension method from [31] to guarantee the diversity ofeach generation. The local elitist list will be sorted out,sent within the PartitionMaster (Alg. 2 Line 10-12) andcollected in the GA-Par Master (Alg. 1 Line 8). The locallist will be aggregated into a sharedElitist by selecting theindividuals that have better fitness values. Afterwards, theglobal sharedElitist will be disseminated to each Partition-Master across different slaves (Alg. 1 Line 9-11). Comparedwith conventional methods, our elitism method also benefitsfrom the global elitist list over the partitions, giving rise tothe fact that the outstanding individuals of each partitioncan be selected and synchronized. This can significantlyminimize the fitness value and reduce the execution time.The PartitionMaster continues the GO phase by using thelatest synchronized list. When new generations are gener-ated and notified from the slave (Alg. 2 Line 18 to 20),GA-Par Master will update by selecting the individuals thathave better fitness values (Alg. 1 Line 17) and repeat theiteration loop until a certain condition emerges. Finally, GA-Par Master outputs the best fit microservice selection for thesubmitted workflow (Alg. 1 Line 19) .

6.2.3 Algorithm Termination

The GA algorithm will eventually compute the best so-lution as the total number of iterations iter goes to ∞[34]. However, limited by computation resources, the valueof iter should be decided in some sense of optimal. Thetermination control in GA-Par assumes that the processwill be terminated if there is no further improvement inthe global fitness value for a fixed number of iterations R.Literally, the value of R can be defined by users accordingto the complexity of their problems. In our experiments, weperform the grid search to find the optimal R within therange of [10, 20, 30, 40]. Eventually 20 is selected since wecannot observe any efficiency gain of reaching the optimalwhen R surpasses 20.

Algorithm 2: : GA-Par Slave and PartitionMaster

Input:shareElitist: shared elitist listQ′: partition number for FCQ′′: partition number for GOpop: populationM : the population size

1 Slave(shareElitist, Q′, Q′′, M , pop)2 // initialize a M

Q′ size population for each partition

3 pop← Initialize(M , pop)4 while not be terminated by master do5 // get the partition number Q′ from Master

6 PartitionNumber(Q′)7 // compute the fitness value for each individual

8 fitpop← FC(pop)9 // find the best N individual

10 localElitist← find(fitpop, N , pop)11 // send localElitist to Master

12 Send( localElitist)13 // sync latest shareElitist from Master

14 shareElitist← Sync()15 // get the partition number Q′′ from Master

16 PartitionNumber(Q′′)17 // generate a new generation

18 pop← GO(pop, sharedElitist)19 // notify new generation pop to Master

20 Notify()

21 end

6.3 Time Complexity Analysis

Since our algorithm involves intensive computation whilehaving low I/O consumption, we simplify our analysis byonly providing the majority cost for one iteration (genera-tion) under the case of a single core executor: 1) populationcreation/reproduction: O(M |S|); 2) fitness score computa-tion: O(M(L3)); 3) elitism list updating: O(O(Mlog(M

Q′ ) +EQ′log(EQ′)) where E is the size of the elitism list;4) crossover: O(ML); and 5) mutation: O(M). The totalcomplexity of one iteration for the single core executor isO(M(|S|+ L3 + log(M

Q′ ))).In contrast, under the parallel cluster environment, the

time complexity of GA-Par computation can be reduced toO( T

NC× [M(|S| + L3 + log(M

Q′ )) + EQ′log(EQ′)]) whereT is the requested CPU cores of each task, and C and Nrepresent the number of CPU cores on each node and nodenumber respectively. The analysis has been omitted anddetails are in Appendix A.

7 EXPERIMENTAL EVALUATION

7.1 Experimental Setup

Platform. We perform all experiments on a 20-node clusterhosted on Google Cloud Platform. Each node is hosted asa n1-standard-8 instance with 8 vCPU (single hardwarehyper-thread and chosen from 2.5 GHz Intel Xeon E5 v2,30GB RAM, and 100GB storage.

Dataset. The used datasets consist of: 1) a list of 4, 532web services’ response-times distributed in 150 computing

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 9, NO. 99, 2019 x

nodes across over 20 countries [35]1; 2) one week through-put records that are derived from Dublin Microsoft Cloudservers [22]; 3) 51 collected techniques and policies forsecurity improvements of Cloud-based services2.Workloads. In the following experiments, we assume thatthe customers prefer the highest security that Cloud-basedmicroservices can provide, given by gqcj = 1, q ∈ {1, .., 8}.To evaluate our algorithm, we firstly assign a number from0 to 24 to each microservice, representing the local timeperiod. Next, we randomly select n techniques out of total51 techniques to each microservice and n is also randomlygenerated. In addition, we randomly generate four types ofDAGs as workflows: The number of vertices in the DAG#w is selected from (50, 100, 150 or 200) and dependenciesbetween vertices are randomly generated to simulate theworkflow (e.g., the example in Figure 1). For each vertexwithin a workflow, we then randomly select #s = (100, 300,500, 1000 or 1500) microservices as the candidates. We usedifferent mappings of workflow and service configurationsfor the application (marked as App(w, s)) in our experiments.We deploy App(w, s) and set up the same initial population.Methodology. We firstly evaluate the security enhancementby comparing GA-Par with the SU (a generalized security-unaware method) and SA-Greedy (a greedy-based security-aware approach). Afterwards, the overall orchestration ef-fectiveness and efficiency of our GA-Par (multiple slaveswith two-phase parallelization) will be evaluated againstother schemes: SGA (a standalone GA, running GA in asingle machine) and HGA (a Hadoop-based ParallelizedGA, following the elephant56 [32]). Since the setting-up ofmapper and reducer in [32] is equivalent to our partitionsetting during the two-phase parallelization, we evaluateGA-Par and HGA under completely the same configura-tions for fairness considerations. Furthermore, we conductthe analysis of impact on the GA-Par efficiency by varyingparameters such as diverse application scale, parallelizationsetting, etc. The scalability of the proposed method is alsoassessed by varying the number of machines.Metrics. In particular, we use three complementary metrics:

• Security Discrepancy. The discrepancy between the re-quirement and the targeted security level that the givenapproach can achieve.

• Consumed Time. The time consumption of the orchestra-tion by different methods.

• Utility Value. The generated utility value of Eq. 18by applying different algorithms. The value implies thereachable distance to the theoretically optimal solution.

7.2 Effect of Using Security-Aware Approach

In this section, we investigate the security improvementintroduced by the security-aware mechanism by runningthe App(100, 100). Basically, we evaluate 1) the presentedsecurity discrepancy between the user requirements and theactual provision by the selected service and 2) the achievableutility value by using different schemes.

As shown in Figure 5, the resultant discrepancy canbe significantly reduced by 67.34% and 42.34% against

1. The original dataset is not available now, we therefore publish thedataset used in our experiments in [36]

2. The list of the techniques and policies is available in [36].

(a) Security Discrepancy (b) Utility Value

Fig. 5. Security Enhancement Effects by Different Methods

TABLE 3Overall Evaluation by Submitting App(100, 100)

Exp Group Time Ratio Value Ratio

SGA/GA-Par 5.85 1.85HGA/GA-Par 2.54 1.39

SU and SA-Greedy respectively. This substantial decreasewithin GA-Par is because the proposed algorithm is awareof the security satisfaction in both services themselves andthe data transmission. Additionally, an approximately 63%and 40% reduction of utility value can be observed com-pared with SU and SA-Greedy respectively. This is becauseonly the network uncertainty (Eq. 6) is considered in theutility function when adopting SU, while our approachtakes both uncertainty and security into account as shownin Equation 14. Although the uncertainty and security areincluded in SA-Greedy, the greedy algorithm only has oneshot to compute the optimal solution, which makes it veryhard to have a correct design to find the optimal solution.Moreover, this type of algorithm is limited in generalizationin that each single optimization requires a new and carefullydesigned algorithm. The results indicate that holisticallyGA-Par can minimize the utility values and effectively finda more reasonable solution from the candidate space.

7.3 Overall Effect Evaluation

In this experiment, we firstly deploy an application App(100,100) by using the SGA, HGA and GA-Par. The initial popu-lation size in all cases is uniformly set to be 5, 000.

SGA vs GA-Par. It is observable from Table 3 that GA-Par obtains a better solution for the optimization problemin terms of fitness(utility) values, compared to SGA (i.e. theratio between them is 1.85). Moreover, the time consumptionof SGA is approximately 5.85 times more than that ofGA-Par which is far beyond the range of tolerance. Thereason is that the stage of reproduction requires a great

TABLE 4Time and Value Ratio of HGA/GA-Par under Different App(w,s)

Time Ratio Comparisonw \s 100 300 500 1000 1500 avg

50 3.901 2.844 3.407 4.789 4.132 3.815100 2.182 2.535 2.670 2.684 2.691 2.553150 2.415 4.065 2.391 2.571 3.775 3.043200 3.277 3.965 4.544 3.720 4.416 3.985avg 2.944 3.352 3.253 3.441 3.754 3.349

Value Ratio Comparison50 1.648 2.114 2.118 1.726 1.745 1.870100 1.506 1.389 1.326 1.347 1.374 1.388150 1.345 1.240 1.348 1.317 1.385 1.327200 1.361 1.197 1.253 1.260 1.249 1.264avg 1.465 1.485 1.511 1.413 1.438 1.462

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 9, NO. 99, 2019 xi

(a) Time of GAPar (b) Value of GAPar

Fig. 6. Time and Value of GAPar.

number of computing resources and the capacity of a singlecomputation node typically cannot satisfy this vast demand.In contrast, by using two-phase parallelization, GA-Par canaccelerate the procedures of both fitness value calculationand generation reproduction. Considering the intolerabletime inefficiency, we will not take SGA into account in thefollowing experiments.

HGA vs GA-Par. Table 3 demonstrates the ratios oftime and value are roughly 2.54 and 1.39 for App(100, 100),indicating that GA-Par outperforms HGA with respects toboth time efficiency and the fitness calculation. Table 4shows more detailed comparison results when deployingapplications with different configurations of w and s. It isobservable that the average time and value ratio are 3.349and 1.462 respectively. There is a stable behavioral ratiobetween HGA and GA-Par in spite of some marginal fluctu-ations. Although the value ratio slightly decreases with theincrement of workflow size, there still exists a guaranteedimprovement of utility value by GA-Par. The improvementsare predominately caused by the in-memory processing andcommunication and the efficient parallelization level withinGA-Par. In effect, GA-Par obtains a trade-off for efficiencybetween fitness and time, as elite list sharing and syn-chronization guarantee the fitness efficiency and populationdivision ensures the time efficiency. Furthermore, we willillustrate the significant reduction of consumed time and theimprovement of effectiveness by varying parameters (suchas Q, Q′′ ) in GA-Par in following subsections.

7.4 Impact of Different Application Scales

Figure 6 demonstrates the experimental results under differ-ent workflow size and candidate number of microservicesby using GA-Par. We can observe that with the increment ofworkflow size, the time consumption increases accordingly.The linear increase demonstrates that the growth of tasknumbers in the workflow will increase the search range tofind optimal solution, thereby taking longer time to finishthe overall computation. In Figure 6(a), the number of ser-vice candidates is not an obvious factor that influences thetime consumption. The consumed time slightly fluctuateswhen the topology and size of the workflow is determined.Apparently, given the workflow size w and each componentin the orchestrated workflow has s candidates, the searchspace is sw. Thus, the impact of s on the consumed timewill not be as significant as that of w.

Likewise, a similar phenomenon can be observed interms of the fitness calculation. In particular, the increasedworkflow size will naturally degrade the optimization ef-fectiveness given the fixed setting of the total population.

(a) Consumed Time (b) Fitness Value

Fig. 7. The impact of population size setting

(a) Consumed Time Comparison (b) Fitness Value Comparison

Fig. 8. Impact analysis of partition configuration

Compared with a smaller-scale workflow, larger workflowswith soaring component numbers are less likely to convergeand obtain the optimal result once the population is set up.

7.5 Impact of Parallelization Parameters

Herein, we discuss the impact of some parallelization pa-rameters on the execution performance. We will evaluatethe impact on the fitness value and execution time underdifferent population size and partition numbers during thetwo-phase parallelization depicted in Section 6.2.2.

Firstly, we fix App(50, 100) and change the initial pop-ulation size ranging from 1000 to 5, 000 and 10, 000; and itis observable from Fig. 7(a) that the consumed time willdramatically increase with the population size going upwhile the fitness value reduced when the population sizesoars, as illustrated in Fig. 7(b). This is because the expandedpopulation will increase the opportunities of emerged elitistsince more individuals are involved in the genetic opera-tions. Undoubtedly, it leads to increased effectiveness to-wards the optimal result, but with an expected extension oftime consumption. Therefore, within the orchestration weneed to strike the balance between the time consumptionand orchestration quality.

Secondly, we fix App(100, 100) with the 5, 000 popula-tion and dynamically adjust Q′ and Q′′. Q′ ranges from 40to 160 with an equal interval while Q′′ is initially set tothe same as Q′ and will shrink to 1/2, 1/4 or 1/8 of Q′.Figure 8(a) illustrates that with the decrement of Q′, thetime consumption grows accordingly. It is obvious that theincrease of parallelization will accelerate the procedure offitness calculation, where the computation time differencebecomes more significant when Q′ is large, while for smallQ′, the benefit of parallelism is trivial and can even begoverned by the randomness introduced by our holisticalgorithm. Additionally, given Q′ is fixed, the executiontime reaches the bottom when the partition number in theGO phase reduces by 50%. In fact, the resource capacityin our Spark-based computation model will determine themaximal number of co-locating executors within the sameslave node, and the resultant merge scheme considering IO

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 9, NO. 99, 2019 xii

(a) Time (b) Value

Fig. 9. Fixed number of partitions per node

(a) Time (b) Value

Fig. 10. Fixed number of total partitions

consumption and messaging communications. Specifically,if three vcores are requested per executor and the phys-ical capacity is eight vcores per slave node, at most twoexecutors can be simultaneously launched within a nodeduring the fitness calculation phase. Equivalently, whenQ′′ = 1/2Q′, the partitions can be merged locally for GOphase without any global shuffles across nodes, therebyminimizing the execution time. Afterwards, the reducednumber of Q′′ will increase the network communicationoverheads, and thus slow down the holistic computing. Thisis identical to our theoretical discussion in Section 6.3 andAppendix A. The trade-off between the scale of parallelismQ′ and the communication overhead of Q′′/Q′ further mo-tivates us to better configure our algorithm based on thehardware characteristics.

However, the corresponding fitness value for differentQ′ and Q′′/Q′, as in Fig. 8(b), indicates the enlarged par-tition number during the fitness calculation phase has anindirect impact on the quality of the optimization solution.In fact, the overall effectiveness is merely relevant to thequality of generated individuals, thus the fitness value ispredominantly influenced by the proportion of elites in theGO phase. This will result in weak correlation between theutility value and the Q′. Assuming the total population sizeand Q′ is fixed, the decreased Q′′ indicates the growth ofsub-population per partition in the subsequent GO phase.In this case, the proportion of elite individuals in the popu-lation descends. From Fig 8(b), we observe a slight decreaseof fitness value at the beginning followed by an increasewith the decrease of Q′′. Therefore, GA-Par trades off thepopulation diversity through controlling Q′′ to find suitableparameters.

7.6 Scalability Evaluation

We evaluate how the machine number involved in GA-Paraffects the execution performance. We fix the applicationtype as App(100, 100) and the total population is initializedas 5000. Because of the relationship between Q′ and Q′′

found in Section 7.5, we only conduct the experiments underthe pre-assumption of Q′′ = Q′/2 with fixed Q′ = 160.

Figure 9 demonstrates the scalability of our algorithm ifthe machine number is scaled from 5 to 20 nodes, where

each node maintains the same number of partitions. In thismanner, the total partition number will be proportional tothe number of nodes. Due to the increased number of parti-tions, it leads to less computation workload involved withina partition. As a result, the increased level of parallelizationcan be achieved, substantially reducing the consumed timeas shown in Fig. 9(a). Regarding the achievable optimallevel, as demonstrated in Fig. 9(b), there is no direct cor-relation between the machine number and the fitness value.Despite the fact taht increased machines will result in theincrement of partition number Q′, the varying partitionnumber Q′′ will not directly impact on the fitness value(similar to the explanation for Fig. 8(b)).

Figure 10 describes another case where we fix the num-ber of total partitions in different experiments. We can ob-serve a consistent time reduction with the increment of thecluster size. Apparently, the increased computation capabil-ity from adding more machines will considerably acceleratethe computation over a fixed population and partition num-ber, in spite of increased communication overheads amongpartitions. Furthermore, for the aforementioned reason, thenumber of machines will not affect the fitness value.

8 RELATED WORK

To deal with load spikes and vendor locks-in issue whilstfully exploiting the cloud diversities with reduced costand guaranteed QoS, geo-distributed cloud techniques andarchitectures are proposed recently such as SuperCloud[37][38], MultiCloud[39], JointCloud [1]. In many cases, theuse of sound security engineering techniques can reducerisks. An alternative approach is to directly utilize thereliable platform-as-a-service (PaaS) components providedby different Cloud providers, instead of designing andbuilding applications from scratch. [11] [10] introduced aset of algorithms to orchestrate applications over federatedClouds. However, the security level of each microserviceis randomly assigned. [12] assigned the security level bythe performance of a specific security technique such asencryption algorithm applied in a Cloud-based service.However, it is extremely difficult to handle the Cloud di-versities. Additionally, the complexity of large-scale geo-distributed Cloud environments results in a great numberof uncertainties which cannot be modeled by the availabilityin conventional information security approaches [40][19]. Inthis paper, we propose a generic method to quantitativelymeasure the security level and its availability aspects underuncertainties of Cloud-based service components. Further-more, how to optimize the service selection or componentplacement to meet different requirements has become aresearch hotspot in the last few years. [41] [42] [43] focusedon QoS-aware service composition and its implementationin middleware. The QoS attributes typically include theresponse time, availability, reliability, price, and reputation,etc., without quantitatively depicting the secure objectiveswith risks and solving them in real Cloud environments.Meanwhile, the above reviewed works have not yet con-sidered the uncertainties within federated Clouds and thecorresponding impacts of massive-scale microservices onthe security aspects. These motivate us to combine them inthe emerging microservice orchestration to realize a reliableservice-oriented system.

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 9, NO. 99, 2019 xiii

As for the optimization solution, solvers such as Mixed-Integer Non-linear Programming (MINLP) or Linear Pro-gramming (MIP) etc. are infeasible in the proposed largeorchestration scenario due to the high time complexity [13].Genetic Algorithm has been widely used to optimizeresource management and task placement [44][45], andparallel Genetic Algorithms also have been explored fordecades [14]. However, there is no general and easily-usedframework that allows for implementing the GA over large-scale clusters. [46] first proposed how to implement GAover Hadoop and conducted detailed studies on the fac-tors that would affect the algorithms’ scalability. Similarly,this paper also studies the scalability factors of GA-Parwithin in-memory computing scenarios. The most recentworks [32] [47] introduced a general framework using amap reduce paradigm, but neglected the linked data amonginterdependent or interactive Cloud services and their non-functional requirements. By contrast, GA-Par can be easilyadapted into distributed models to accelerate the orchestra-tion. [48] proposed an architecture to deploy and executeparallel GAs based on the available resources over Clouds.This architecture can be considered as an effective supple-ment to GA-Par.

9 CONCLUSION

In this paper, we describe a new microservice orchestrationframework by considering both internal security threats andenvironmental uncertainties. The internal security satisfac-tion levels stem from application’s topology and diverseprovisioning capability of microservice candidates. The un-certainties are captured by black-boxed modelling in termsof response time and transmission rates over Clouds. Weadopt a novel paralleled GA-Par on Spark to acceleratethe solving of the proposed orchestration. Some importantfindings and conclusions can be drawn as follows:

• Improving dependability of massive scale orchestration isbecoming of increasing importance. Quantitatively for-malized modeling and measurement throughout bothinternal and external factors can comprehensively em-power maximized achievability of dependability.

• Standing on the security viewpoint reveals an effective meansto measure the system dependability. Although we focus onthe security-aware modeling and problem formulation,the presented methodology can be easily applied intoother dimensions for depicting dependability such asreliability, availability and safety, etc.

• Relying on real dataset and data-driven approach is criti-cal to understanding the real-world problems and formu-lating assumptions under realistic circumstances. Statisticapproaches can be exploited to deal with the long-standing environmental uncertainty issues and it is alsoan effective means to capture the long-tail responsecharacteristics of Cloud-based services.

• Integrating GA-Par with IoT and Fog eco-system to facilitatethe service orchestration. Within the coming decades, theconcept of the exascale system will become increasinglycommonplace, interconnecting billions of different sen-sors, things and other data sources across a vast numberof industries which will likely co-exist in some formof Fog and smart mobility eco-system [49][50]. Thechallenges of orchestration pertaining to security and

uncertainties will continue to play a critical concern fordesigning these systems.

More practical validations are underway to demon-strate the industrialized process into production-level clus-ter scheduling and container orchestration systems based oncollaborative works with Alibaba such as [51][52][53]. Tech-nically, we will further consider data partition to achieveindividual reduction of GA incurred by partitioning the datato different Slaves. Finally, studies are also needed to de-velop partition algorithms to make GA-Par more efficient.

APPENDIX A

PERFORMANCE ANALYSIS

A.1 Spark-based Execution Model

Apache Spark is built upon the concept of Re-silient Distributed Datasets (RDD) and provides ac-tions/transformations on top of RDD. More precisely, Sparkapplications include three steps: creating new RDDs, trans-forming existing RDDs, and calling operations/actions onRDDs to compute the results.

A Spark application consists of a driver process and aset of executor processes scattered across nodes on a clus-ter. A driver is a process that controls the high-level dataflow, while executor processes are responsible for executingtasks. Batches of tasks run concurrently on each executorthroughout the application’s lifetime. In other words, thesetasks are distributed to RDD partitions in a bijective wayand are executed concurrently. As a result, the parallelismof Spark is highly dependent upon the partition numberas well as the executor’s capability. Using Spark commandline flags is a flexible approach for tuning the utilizationof computational resources to available CPU cores. Forexample, the “executor-core” flag can specify the numberof CPU cores bound to each executor, controls which thenumber of concurrent tasks. Suppose a cluster has N slavesand each has C CPU cores. If we parallelize the execution ofNC tasks over the cluster, the average utility is NC

T.

The performance of GA-Par is evaluated with respect tothe following three factors: computation complexity, networkcommunication, and disk reads/writes. For ease of exposition,we simplify the time complexity analysis by first consider-ing one iteration of the algorithm.

A.2 Computation Complexity

The computation is the main constituent of the overall timecomplexity. For sequential GA computation, the time com-plexity of each iteration can be divided into the following:

Population Creation/Reproduction. According to ourdefinition in Spark preliminary, Q′ partitions correspond to

Q′ tasks to be executed. Each partition involves MQ′

∑Li=1 |Si|

computations. Therefore we have:

T1 = Q′ × (M

Q′×

L∑

i=1

|Si|) ≊ O(M |S|)

Fitness Score Computation. The computation of eachpartition can be summarized as M

Q′ × t1, where t1 is the

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 9, NO. 99, 2019 xiv

time complexity of workflow evaluation. More details areas follows:

t1 =∑

sjh∈λ′

V alue(ci, sjh,mrt, t)

=∑

sjh∈λ′

T (mrt, sjh) · {θ · ε(tsjh) ·DataSj

Total

+ (1− θ) · [V DDSD(ci, sjh) +DSSD(ci, sjh)]}

Thus, the time complexity t1 can be written as

t1 =∑

sjh∈λ′

T (mrt, sjh) · {θ · [ε(tsjh) ·DataSj

Total]+

(1− θ) · [∑

{dsip→sjh}∈INsj

|G|∑

q=1

wqij ·M(gqsip , g

qsjh

)+

|G|∑

q=1

wqj ·M(gqci , g

qsjh

)]} (19)

and approximated as follows:

t1 ≊

L∑

i=1

(C1 +

|INSj|∑

j=1

C2 + C2) ≊ C1L+ C2|IN |L+ C2L

≊ (C1 + C2)L+ C2|IN |L

where C1 and C2 are constants. As a result, INSjrepresents

the input data from Sj ’s immediate predecessors, and INis the set of all data dependencies. In the best case, wehave |IN | = L − 1, while in the worst case the numberof data dependencies is |IN | = L(L− 1). Consequently, forQ′ partitions, the time complexity T2 is:

T2 ≊ Q′ × (M

Q′× t1) ≊ O(ML3)

Updating the Elitism List. The time complexity of find-ing the elitism list on a single host is O(Mlog(M)). How-ever, the time complexity can be reduced by first generatingthe local elitism list for each partition. A local elitism list isobtained by running the top E nearest neighbors on eachpartition. The global elitism list with size E can then begenerated by sorting EQ′ individuals.

Since each partition has MQ′ individuals, the time used to

sort the fitness value in each partition can be approximatedto O(M

Q′ log(MQ′ )), and finding the global elitism list requires

O(EQ′log(EQ′)). Thus, the time complexity of updatingthe global elitism list is:

T3 ≊ O(Q′ × (M

Q′log(

M

Q′)) + EQ′log(EQ′))

≊ O(Mlog(M

Q′) + EQ′log(EQ′))

Crossover. Since Genetic Operations (GO) works on asmaller number of partitions Q′′ to ensure the diversity ofthe next generation, the time complexity of both Crossoverand Mutation should be considered under this case. The

population size of each partition is MQ′′ , which reduces the

time complexity of Crossover to:

T4 ≊ Q′′ × (M

2Q′′· L)s ≊ O(ML)

Mutation. Since it involves the rate of Mutation, the timecomplexity can be approximated as:

T5 ≊ O(M)

In summary, the time complexity for each iteration is:

Time(Computation) = T1 + T2 + T3 + T4 + T5

≊ O(M |S|) +O(ML3)) +O(Mlog(M

Q′) + EQ′log(EQ′))

+O(ML) +O(M)

≊ O(M(|S|+ L3 + log(M

Q′)) + EQ′log(EQ′))

Based on the above analysis, we prove that the over-all GA-Par computation can be reduced to O( T

NC×

Time(Computation)), which clearly demonstrates the scal-ability of the algorithm.

Proof. Assume we have a N -slaves Cluster, each slave has Ccores, and each task needs T cores of CPU. Then, there areNCT

execution slots that can run in parallel, i.e. the speed-

up is TNC

. Since Time(Computation) represents the totalworkload of a single machine with one single-core CPU, thetime complexity of our algorithm can reduce to O( T

NC×

Time(Computation)).

A.3 Network Communication

Shuffling also affects execution time, so data transmissionand partition operations should be carefully considered. Wesimplify our evaluation model by assuming that our idealSpark implementation will not block the Spark program andonly the observable and countable network communicationdelay is considered:

Shuffling Cost of Updating the Elitism List. Thisprocedure includes collection of the local elitism lists andbroadcast of the global elitism list. During collection ofthe local elitism lists, we primarily consider the maximalnetwork delay maxi∈[1,Q′](dCollecti), where dCollecti isthe time delay for collecting the local elitism list from theith partition. Similarly, for broadcasting, the time delayshould be maxj∈[1,N ](dBroadcastj), where dBroadcastj isthe time delay for broadcasting the global elitism list to thejth slave. As a reminder, either the size of local elitism listor global elitism list is E. Thus, we can evaluate the cost asT6 where C4, C5 and C6 are constants:

T6 = C4 maxi∈[1,Q′]

(dCollecti) + C5 maxj∈[1,N ]

(dBroadcastj) + C6

≊ O(E)

Shuffling Cost of Partition Adjustment. The perfor-mance depends on the value of Q′ and Q′′, where

Q′ = ⌊Q′

N⌋ × a1 + b1;Q

′′ = ⌊Q′′

N⌋ × a2 + b2

Based on the scheme mentioned above, at most ⌊ Q′

Q′′ ⌋ par-titions can be merged into a single partition. Also, thismerging process is not limited to the same slave and might

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 9, NO. 99, 2019 xv

occur among the partitions in different slaves. This willcause the data shuffling. In particular:

• If ⌊Q′

N⌋ = i × ⌊ Q

′

Q′′ ⌋ (i ∈ Z), no additional shufflingoccurs because the merging process operates within thesame slave.

• Otherwise, the partitions will cross over slaves. Assume

⌊Q′

N⌋ = i × ⌊ Q

′

Q′′ ⌋ + r, where i, r ∈ Z. The worst

case scenario of partition adjustment would be Q′

partitions each containing MQ′ individuals, participating

in the shuffling procedure. The communication cost isC7 ·Q

′ ·MQ′×θ, i.e., C7Mθ where θ is the bit transmission

rate for an individual. The worst case scenario rarelyhappens under the control of the optimized Sparkshuffling schema, but in the best case, each Slave onlyinvolves the remaining r partitions. The ideal shuffling

cost would be C8N × (⌊Q′

N⌋ mod ⌊ Q

′

Q′′ ⌋).

In summary, the shuffling cost T7 for partition adjust-ment would fall within the following:

C8N × (⌊Q′

N⌋ mod ⌊

Q′

Q′′⌋) ≤ T7 ≤ C7Mθ

A.4 Disk Reads/Writes

Our system involves intensive computations but the pro-cessed data can easily fit into memory. As a result, wecan ignore the cost of I/O in the initialization phase. Theshuffling process will involve some unavoidable I/O costs,which can follow the partial schema of A.3. Specifically, theanalysis includes:

Updating the Elitism List. We first collect the localelitism lists from each partition and read them on the Masternode. the I/O cost is 2EQ′. Similarly, the I/O cost forbroadcasting the global elitism list is 1 + Q′′. As a result,the total I/O cost in this phase is 2EQ′ +Q′′ + 1.

Partition Adjustment. Based on the analysis in A.3, the

I/O cost is approximately between 2N(⌊Q′

N⌋ mod ⌊ Q

′

Q′′ ⌋)and 2M .

ACKNOWLEDGMENT

This work is supported by the National Key Research andDevelopment Program (2016YFB1000103) and the NationalNatural Science Foundation of China (61421003). This workis also supported by Beijing Advanced Innovation Centerfor Big Data and Brain Computing (BDBC). For any cor-respondences, please contact corresponding author RenyuYang.

REFERENCES

[1] H. Wang, P. Shi, and Y. Zhang, “Jointcloud: A cross-cloud coop-eration architecture for integrated internet service customization,”in IEEE ICDCS, 2017.

[2] Z. Wen, R. Yang, P. Garraghan, T. Lin, J. Xu, and M. Rovatsos,“Fog orchestration for internet of things services,” IEEE InternetComputing, vol. 21, no. 2, pp. 16–24, 2017.

[3] A. C. Zhou, Y. Gong, B. He, and J. Zhai, “Efficient process map-ping in geo-distributed cloud data centers,” in Proceedings of theInternational Conference for High Performance Computing, Networking,Storage and Analysis. ACM, 2017, p. 16.

[4] A. C. Zhou, S. Ibrahim, and B. He, “On achieving efficient datatransfer for graph processing in geo-distributed datacenters,” inDistributed Computing Systems (ICDCS), 2017 IEEE 37th Interna-tional Conference on. IEEE, 2017, pp. 1397–1407.

[5] M. Natu, R. K. Ghosh, R. K. Shyamsundar, and R. Ranjan, “Holis-tic performance monitoring of hybrid clouds: Complexities andfuture directions,” IEEE Cloud Computing, vol. 3, no. 1, pp. 72–81,2016.

[6] A. Avizienis, J.-C. Laprie, B. Randell, and C. Landwehr, “Basicconcepts and taxonomy of dependable and secure computing,”IEEE TDSC, vol. 1, no. 1, pp. 11–33, 2004.

[7] A. Dusia, Y. Yang, and M. Taufer, “Network quality of service indocker containers,” in 2015 IEEE International Conference on ClusterComputing (CLUSTER). IEEE, 2015, pp. 527–528.

[8] Y. Chen, A. Gorbenko, V. Kharchenko, and A. Romanovsky, Mea-suring and Dealing with the Uncertainty of SOA Solutions. IGIGlobal, 2012.

[9] A. Gorbenko and A. Romanovsky, “Time-outing internet ser-vices,” IEEE Security & Privacy, 2013.

[10] P. Watson, “A multi-level security model for partitioning work-flows over federated clouds,” Journal of Cloud Computing: Advances,Systems and Applications, vol. 1, no. 1, p. 15, 2012.

[11] Z. Wen, J. Cala, P. Watson, and A. Romanovsky, “Cost effective,reliable and secure workflow deployment over federated clouds,”IEEE TSC, 2016.

[12] W. Liu, S. Peng, W. Du, W. Wang, and G. S. Zeng, “Security-aware intermediate data placement strategy in scientific cloudworkflows,” Knowl. Inf. Syst., 2014.

[13] S. Malek, N. Medvidovic, and M. Mikic-Rakic, “An extensibleframework for improving a distributed software system’s deploy-ment architecture,” IEEE TSE 2012.

[14] E. Cantu-Paz, “A summary of research on parallel genetic algo-rithms,” 1995.

[15] Y. Gan, Y. Zhang, D. Cheng et al., “An open-source benchmarksuite for microservices and their hardware-software implicationsfor cloud and edge systems,” in ACM ASPLOS 2019.

[16] N. R. Potlapally, S. Ravi, A. Raghunathan, and N. K. Jha, “Astudy of the energy consumption characteristics of cryptographicalgorithms and security protocols,” IEEE Transactions on mobilecomputing, vol. 5, no. 2, pp. 128–143, 2006.

[17] S. Ji, W. Li, P. Mittal, X. Hu, and R. Beyah, “Secgraph: A uniformand open-source evaluation system for graph data anonymizationand de-anonymization,” in USENIX Security, 2015.

[18] D. Catteddu and G. Hogben, “Cloud computing: Benefits, risksand recommendations for information security,” enisa:EuropeanNetwork and Information Security Agency, Tech. Rep., 2009.

[19] M. E. Whitman and H. J. Mattord, Principles of Information Security,2004.

[20] D. Trihinas, A. Tryfonos, M. D. Dikaiakos, and G. Pallis, “Devopsas a service: Pushing the boundaries of microservice adoption,”IEEE Internet Computing, vol. 22, no. 3, pp. 65–71, 2018.

[21] T. Xie and X. Qin, “Performance evaluation of a new schedulingalgorithm for distributed systems with security heterogeneity,”Journal of Parallel and Distributed Computing, 2007.

[22] M. Forshaw, “Operating policies for energy efficient large scalecomputing,” Ph.D. dissertation, Newcastle University, UK, 2015.

[23] M. Dobber, R. van der Mei, and G. Koole, “Effective prediction ofjob processing times in a large-scale grid environment,” in HighPerformance Distributed Computing, 2006 15th IEEE InternationalSymposium on. IEEE, 2006, pp. 359–360.

[24] T. Miu and P. Missier, “Predicting the execution time of workflowactivities based on their input features,” in High Performance Com-puting, Networking, Storage and Analysis (SCC), 2012 SC Companion:.IEEE, 2012, pp. 64–72.

[25] R. J. Adler, R. E. Feldman, and M. S. Taqqu, Eds., A Practical Guideto Heavy Tails: Statistical Techniques and Applications, 1998.

[26] A. B. Downey, “Evidence for long-tailed distributions in the inter-net,” in SIGCOMM, 2001.

[27] V. P. Chistyakov, “A theorem on sums of independent positiverandom variables and its applications to branching random pro-cesses,” Theory of Probability & Its Applications, 1964.

[28] E. Seneta, Ed., Regularly Varying Functions. Springer, 1976.[29] P. Embrechts, T. Mikosch, and C. Kluppelberg, Modelling Extremal

Events: For Insurance and Finance, 1997.[30] P. Cirillo, “Are your data really pareto distributed?” Physica A:

Statistical Mechanics and its Applications.[31] D. Bhandari, C. Murthy, and S. K. Pal, “Genetic algorithm with

elitist model and its convergence,” IJPRAI, 1996.[32] P. Salza, F. Ferrucci, and F. Sarro, “elephant56: Design and imple-

mentation of a parallel genetic algorithms framework on hadoop

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 9, NO. 99, 2019 xvi

mapreduce,” in Proceedings of the 2016 on Genetic and EvolutionaryComputation Conference Companion. ACM, 2016, pp. 1315–1322.

[33] M. Zaharia, M. Chowdhury, T. Das, and et al., “Resilient dis-tributed datasets: A fault-tolerant abstraction for in-memory clus-ter computing,” in NSDI, 2012.

[34] C. Murthy, D. Bhandari, and S. K. Pal, “ε-optimal stopping timefor genetic algorithms,” Fundamenta Informaticae, vol. 35, no. 1-4,pp. 91–111, 1998.

[35] Y. Zhang, Z. Zheng, and M. R. Lyu, “Wspred: A time-aware per-sonalized qos prediction framework for web services,” in ISSRE,2011.

[36] “Dataset used for the experiments,”https://github.com/lukewen427/GA-Par-data, accessed: 2019-04-25.

[37] Z. Shen, Q. Jia, G.-E. Sela, W. Song, H. Weatherspoon, andR. Van Renesse, “Supercloud: A library cloud for exploiting clouddiversity,” ACM TOCS, 2017.

[38] Q. Jia, Z. Shen, W. Song, R. Van Renesse, and H. Weatherspoon,“Supercloud: Opportunities and challenges,” ACM SIGOPS Oper-ating Systems Review, 2015.

[39] F. Paraiso, N. Haderer, P. Merle, R. Rouvoy, and L. Seinturier, “Afederated multi-cloud paas infrastructure,” in IEEE Cloud. IEEE,2012.

[40] D. B. Parker, Fighting Computer Crime: A New Framework for Protect-ing Information, 1998.

[41] T. Yu, Y. Zhang, and K.-J. Lin, “Efficient algorithms for webservices selection with end-to-end qos constraints,” ACM Trans.Web, 2007.

[42] M. Alrifai, D. Skoutas, and T. Risse, “Selecting skyline services forqos-based web service composition,” in Proceedings ACM WWW.ACM, 2010, pp. 11–20.

[43] S. Rosario, A. Benveniste, S. Haar, and C. Jard, “Probabilistic qosand soft contracts for transaction-based web services orchestra-tions,” IEEE TSC, 2008.

[44] C. Guerrero, I. Lera, B. Bermejo, and C. Juiz, “Multi-objectiveoptimization for virtual machine allocation and replica placementin virtualized hadoop,” IEEE Transactions on Parallel and DistributedSystems, vol. 29, no. 11, pp. 2568–2581, 2018.

[45] Z. Zhu, G. Zhang, M. Li, and X. Liu, “Evolutionary multi-objectiveworkflow scheduling in cloud,” IEEE Transactions on parallel anddistributed Systems, vol. 27, no. 5, pp. 1344–1357, 2016.

[46] A. Verma, X. Llora, D. E. Goldberg, and R. H. Campbell, “Scalinggenetic algorithms using mapreduce,” in Intelligent Systems Designand Applications, 2009. ISDA’09. Ninth International Conference On.IEEE, 2009, pp. 13–18.

[47] F. Ferrucci, P. Salza, and F. Sarro, “Using hadoop mapreduce forparallel genetic algorithms: A comparison of the global, grid andisland models,” Evolutionary computation, no. Early Access, pp. 1–33, 2017.

[48] P. Salza, F. Ferrucci, and F. Sarro, “Develop, deploy and executeparallel genetic algorithms in the cloud,” in Proceedings of the2016 on Genetic and Evolutionary Computation Conference Companion.ACM, 2016, pp. 121–122.

[49] A. Longo, M. Zappatore, M. Bochicchio, and S. B. Navathe,“Crowd-sourced data collection for urban monitoring via mobilesensors,” ACM Transactions on Internet Technology (TOIT), 2017.

[50] A. Longo, M. Zappatore, and S. B. Navathe, “The unified chartof mobility services: Towards a systemic approach to analyzeservice quality in smart mobility ecosystem,” Journal of Parallel andDistributed Computing, 2019.

[51] Z. Zhang, C. Li, Y. Tao, R. Yang, H. Tang, and J. Xu, “Fuxi: afault-tolerant resource management and job scheduling system atinternet scale,” Proceedings of the VLDB Endowment, 2014.

[52] R. Yang, Y. Zhang, P. Garraghan, Y. Feng, J. Ouyang, J. Xu,Z. Zhang, and C. Li, “Reliable computing service in massive-scalesystems through rapid low-cost failover,” IEEE TSC, 2016.

[53] X. Sun, C. Hu, R. Yang, P. Garraghan, T. Wo, J. Xu, J. Zhu, andC. Li, “Rose: Cluster resource scheduling via speculative over-subscription,” in IEEE ICDCS, 2018.

Zhenyu Wen is currently a postdoc researcherwith the School of Computing, Newcastle Uni-versity, UK. He received M.S and Ph.D. degreesin computer science from Newcastle University,Newcastle Upon Tyne, UK in 2011 and 2015respectively. His current research interests in-clude Multi-objects optimisation, Crowdsources,AI and Cloud computing.

Tao Lin is currently a Ph.D. student at EPFL,Switzerland. Prior to that, he received his M.scand B.E. from EPFL and Zhejiang University. Hisresearch interests include distributed machinelearning, optimization etc.

Renyu Yang is a Research Fellow with Univer-sity of Leeds, UK. He is also an adjunct researchscientist with BDBC, Beihang University, China.He received his Ph.D. degree from Beihang Uni-versity. He was previously with Alibaba Groupand had industrial experience in building large-scale systems. His research interests includereliable resource management, distributed sys-tems, data processing, etc. He is a member ofIEEE.

Shouling Ji is a Professor at Zhejiang Univer-sity and a Research Faculty in the School ofElectrical and Computer Engineering at GeorgiaInstitute of Technology (Georgia Tech). He re-ceived two Ph.D. degrees from Georgia Instituteof Technology and Georgia State University, andB.S. (with Honors) and M.S. degrees from Hei-longjiang University. His current research inter-ests include data-driven security and privacy, AIsecurity and big data analytics. He is a memberof IEEE.

Rajiv Ranjan is a Professor in Computing Sci-ence at Newcastle University, UK. Before mov-ing to Newcastle University, he was Julius Fel-low (2013-2015), Senior Research Scientist andProject Leader in the Digital Productivity andServices Flagship of Commonwealth Scientificand Industrial Research Organization. Prior tothat he was a Senior Research Associate (Lec-turer level B) in the School of Computer Sci-ence and Engineering, University of New SouthWales. He has a Ph.D. from the department of

Computer Science and Software Engineering, the University of Mel-bourne.

Alexander Romanovsky AlexanderRomanovsky is a Professor in ComputingScience at Newcastle University, UK. He isthe investigator of the EPSRC platform granton Layers for Structuring Trustworthy AmbientSystems (STARTA) and a co-investigatorof the EPSRC PRiME programme grant onPower-efficient, Reliable, Many-core Embeddedsystems. Before this, he coordinated the majorEU FP7 DEPLOY IP that developed the Rodintooling environment for formal stepwise design

of complex dependable systems using Event-B. His main researchareas are system dependability, fault tolerance, safety, modelling andverification.

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 9, NO. 99, 2019 xvii

Changting Lin is currently a Ph.D. candidate atZhejiang University. He received B.E. and M.S.degrees from Zhejiang Gongshang University in2009 and 2012. His research interests includeSoftware Defined Network, reconfigurable andnetwork security.

Jie Xu is Chair Professor of Computing at Uni-versity of Leeds, Director of UK EPSRC WRGe-Science Centre, Chief Scientist of BDBC, Bei-hang University, China. He has industrial ex-perience in building large-scale networked sys-tems and has worked in the field of depend-able distributed computing for over 30 years. Heis a Steering/Executive Committee member fornumerous IEEE conferences including SRDS,ISORC, HASE, SOSE and is a co-founder forIEEE IC2E. He has led or co-led many research

projects to the value of over $30M, and published in excess of 300academic papers, book chapters and edited books. He is a memberof IEEE.