an anomalous behavior detection model in cloud computing

TSINGHUA SCIENCE AND TECHNOLOGYISSNll1007-0214ll08/11llpp322–332Volume 21, Number 3, June 2016

An Anomalous Behavior Detection Model in Cloud Computing

Xiaoming Ye, Xingshu Chen�, Haizhou Wang, Xuemei Zeng, Guolin Shao, Xueyuan Yin, and Chun Xu

Abstract: This paper proposes an anomalous behavior detection model based on cloud computing. Virtual

Machines (VMs) are one of the key components of cloud Infrastructure as a Service (IaaS). The security of such

VMs is critical to IaaS security. Many studies have been done on cloud computing security issues, but research into

VM security issues, especially regarding VM network traffic anomalous behavior detection, remains inadequate.

More and more studies show that communication among internal nodes exhibits complex patterns. Communication

among VMs in cloud computing is invisible. Researchers find such issues challenging, and few solutions have

been proposed—leaving cloud computing vulnerable to network attacks. This paper proposes a model that uses

Software-Defined Networks (SDN) to implement traffic redirection. Our model can capture inter-VM traffic, detect

known and unknown anomalous network behaviors, adopt hybrid techniques to analyze VM network behaviors, and

control network systems. The experimental results indicate that the effectiveness of our approach is greater than

90%, and prove the feasibility of the model.

Key words: virtual machine; network behavior; anomaly detection; cloud computing

1 Introduction

Cloud computing infrastructure is a hybrid networkingsystem, that integrates hybrid technology, hybridoperating systems, and hybrid hardware. Cloudcomputing aims to provide on-demand, low-cost,high-performance computing resources, and leveragesvirtualization technologies to deliver storage, server,network services, CPU, and memory[1].

Cloud computing has to face traditional securitythreats and new generations of security threats. Cloudcomputing vulnerabilities include core technologyvulnerabilities (e.g., Web applications and services,virtualization, and cryptography), essential cloudcharacteristic vulnerabilities (e.g., unauthorized

�Xiaoming Ye, Xingshu Chen, Haizhou Wang, XuemeiZeng, Guolin Shao, Xueyuan Yin, and Chun Xu are withthe College of Computer Science, Cybersecurity ResearchInstitute, Sichuan University, Chengdu 610065, China. E-mail:[email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected].�To whom correspondence should be addressed.

Manuscript received: 2016-01-09; accepted: 2016-03-07

access to management interfaces, Internet protocolvulnerabilities, etc.), and defects in known securitycontrols, and prevalent vulnerabilities (e.g., injectionvulnerabilities and weak authentication schemes)[2].Attackers find vulnerabilities and use them to undertakeattacks. There have been many attacks against virtualmachines on cloud computing platforms, such asvarious port scanning attack, attacks on hypervisors,attacks on virtualization, backdoor channel attacks,flooding attacks, user-to-root attacks, and insiderattacks (e.g., internal denial-of-service attacks viazombies in the cloud)[3].

Virtualization technology is a core technology incloud computing. Virtual Machines (VMs) are keycomponents of cloud infrastructure. For example,virtualization technology enables the execution ofmultiple operating system environments, or VMinstances, on a single hardware system. Each VMowns an operating system and applications. AVM executes programs like a physical machine.Cloud computing contains both physical and virtualnetworks[4]. Virtualization creates blind spots ofnetwork traffic, or invisible networks, in the sameserver infrastructure. Gartner[5] represented six of the

www.redpel.com +917620593389


Xiaoming Ye et al.: An Anomalous Behavior Detection Model in Cloud Computing 323

most common virtualization security risks, includingnoting that “the lack of visibility and controlson internal virtual networks created for VM-to-VM communications blinds existing security policyenforcement mechanisms”. He said that more than 60%of virtual machines in production are less secure thantheir physical counterparts. VMs are losing their abilityto detect and control this communication. Attacks anddata can move through the VMs without ever going outto the physical network, which means these attacks willnot be detected by traditional tools. To deal with thisvulnerability, making all VM communications trafficvisible is the first problem that needs to be solved.

Currently, the challenge is how to establish aneffective network behavior detection system for eachVM in a cloud computing network, so that it canaccurately identify deviations from normal networkbehavior of the virtual machines, and reduce cloudsecurity risks.

This paper proposes a model to detect anomalybehavior for the VMs in cloud computing. Thismodel is a time-varying system with a number ofnetwork traffic features. Here are the main work andcontributions of this paper:� Communications among VMs in cloud are

invisible. The model uses Software-Defined Networks(SDN) to build a virtual network, so that the virtualswitch network traffic is through the physical networkcard, then to the node where the deployed systemresides.� The model aims to detect known and unknown

anomalous behaviors.� This paper designs a control model, and adopts

hybrid techniques to analyze VM network behaviorsand control network systems.

The remainder of this paper is organized as follows.Section 2 introduces state machine definitions andcomponents of the model and methods of stateanalysis. Section 3 introduces Snort, data processing,application behavior analysis, and decision analysis.The algorithm and technologies used in this paper arealso discussed. Experiments were conducted and theresults are discussed in Section 4. Conclusions andfuture work are presented in Section 5.

2 Model Overview

2.1 State definitions

Network behavior has various forms and means of

changing characteristics. We cannot describe andidentify all the anomalous behaviors of networks butcan describe states that characterize a VM under attack.

Before the attack, a malicious user tries to scan VMsand search for vulnerabilities or ports to find the cloudcomputing infrastructure security “holes”. The attackerthen has a planned, purposeful, step-by-step process toundertake the attack, including an attack action plan,tests, and a complete attack process. Normal VMnetwork behavior is a state of dynamic equilibrium.Network attacks will affect this state, which is definedas follows:

Definition 1 (Homeostasis, S1): Currently, thevirtual machine is running properly, the network trafficsituation is in dynamic equilibrium. Virtual machineshave vulnerabilities and other security threats, but theyhave not been detected or used.

Definition 2 (Before imbalances, S2): Supposeanomalous behaviors of network traffic are detected,such as vulnerability scanning. In this state, VMsecurity threats have been detected, but have not yetbeen utilized by an attacker.

Definition 3 (Imbalances early, S3): Supposeanomalous network traffic behaviors are detected morethan once. An attacker has detected vulnerabilities inthe virtual machine, and exploited them.

Definition 4 (Imbalance, S4): Network trafficanomalies are repeatedly detected. The VM is undercontinuous cyber-attacks.

Figure 1 depicts the transition of virtual machinestates under attack. The sequence starts at state S1.Attack behaviors make VM state S1 activate statesS2, S3, and S4. When anomalous behavior has beencontrolled, the VM state returns to a state of dynamicequilibrium.

Fig. 1 VM state transition.



324 Tsinghua Science and Technology, June 2016, 21(3): 322–332

Through application behavior analysis, the modeldetermines whether or not application behavior deviatesfrom normal. According to this, the model can be usedto describe VM state transitions. The details of itsalgorithm will be given in Section 2:3.

2.2 Components

This paper proposes a cloud computing anomalousbehavior detection model. The model can detect knownand unknown anomalous behaviors. Hybrid techniquesare used to detect anomalies. The model determineswhether the network behavior of a virtual machinedeviates from normal.

Figure 2 describes the model components anddetection processes. This model consists of VMprofiles, Snort, data processing, application behavioranalysis, state analysis modules, and decision analysis.� The VM profile module is a dataset used to store

and manage VM profiles based on traffic analysis.Application behavior states are used to build a set ofVM profiles. The information includes the services, thesoftware version number, open port, IP address, MACaddress, and rules. In addition, it also includes rules forcommunication among virtual machines, and betweenvirtual machines and physical machines. These profilesinclude VM security rules among other features.� VM network traffic passes through Snort first. This

module is used to detect known anomalous behaviors.

Fig. 2 Model components.

Snort uses detection rules based on signature. Themodel first executes a Snort module, which providesknown anomaly detection, improves the detection rate,and reduces the computational cost. Then networktraffic flows into the next detection module. The Snortmodel not only uses the known anomaly behavior rulebase, but also reduces the volume of traffic that must beprocessed in the next module.� The model then performs application behavior

analysis. This module has two parts. In the firstpart, traffic classification is performed to identifyapplications. This part manipulates the trainingexamples and produces multiple classifiers to improvethe application classification accuracy. In the secondpart, the application behavior analysis module usestime series to build a baseline for each application.Considering the normal network behavior of VMs,time series analysis is used. For example, peoplework during the day and rest at night. People workfrom Monday to Friday and rest on Saturday andSunday. Other regular behaviors include data backup,“application heartbeat”, and periodic behaviors thatare repeated. This module aims to detect unknownanomalous behaviors. So the properties of applicationsfor each VM are stored. The algorithm of this moduleis given below.� Finally, the results of detection from Snort and

the application behavior analysis module are savedas anomaly records. In order to improve detectionaccuracy, the decision analysis module uses the recordsfor in-depth analysis. The algorithm is below.� After the application behavior analysis, the VM

profile information is updated. According to this, themodel can describe the states of the VMs in cloud. Theformulas are described below.

2.3 State analysis

VM profiles have summary information about each VMin the cloud collected from traffic. For each application,detection results from Snort and behavior analysis areadded to the VM profiles. Other information includesnumber of services, open port number, number of flows,number of outgoing connections, number of incomingconnections, maximum value of each connection,and duration. In addition, it also includes rules forcommunication between virtual machines, as well as forcommunication between virtual machines and physicalmachines.Ak represents the anomalous performance of the k-




th VM in the cloud as discussed in Section 2.1. In thismethod, the state of each VM is shown in three formsA.1/, A.2/, and A.3/. Its value is calculated by Eq. (1),where n.t/ is the random noise, and r1, r2, and r3 areparameters. vmk denotes the weight of the k-th VMusing Eq. (20). A.1/ represents the degree of deviationof traffic periodicity of the VM using Eq. (6). A.2/

denotes the anomalous status of known applications(app) using Eq. (11). A.3/ denotes anomalous statusof unknown applications (uapp) using Eq. (12).

The anomalous performance of VMs isAk D vmk.r1A

.1/C r2A

.2/C r3A

.3//C n.t/ (1)Here’s how to compute A.1/. A VM profile is a time-

varying matrix with network traffic features that candescribe the state of network traffic. A time series isa sequence of data usually at regular intervals of timeduring a specific period. The most important featureof this type of data is that neighboring observations aredependent on each other. This paper takes into accounthistory data before time T .T1, T2, and T3 are threeadjacent time before detection time T ), but also lastweek’s value WT, last month’s value MT, and last yearsvalue YT at each observing time as shown in Fig. 2.

Thus, in Eq. (2), here are six values associated withgiven time, where m represents the total number ofobservation characteristics. Create a time matrix Stvmas follows:

Stvm D

0B@ w11 : : : w16:::

: : ::::

wm1 : : : wm6

1CA (2)

Build a vector base on each time window Wi attime t, where W1 represents T1, W2 represents T2, W3represents T3, W4 represents WT, W5 represents MT,and W6 represents YT.

Wi D .w1i ; w2i ; :::; wmi /T (3)

Stvm D .W1;W2;W3;W4;W5;W6/ (4)

The model then uses Euclidean distance to measurethe transformation. It means the likelihood of ananomalous VM state performance can be expressed bythe distance spanned by the time window vector. TheEuclidean distance can be expressed as

dist.Wi ;Wj / D

vuut mXkD1

.wki � wkj /2 (5)

A weight ˇi is associated with each time window toexpress its importance in relation to time T.

A.1/ D1

6

6XiD1

.ˇi � dist.WT ;Wi // (6)

Here’s how to compute A.2/. In the followingequations appi represents the i-th application. Thelikelihood of anomalous application performance canbe expressed in detail by considering factors such as theprobability of presence of the application in traffic:Fi DPrfappig � Prfanomalyjappig D

Prfappig � fappi is suspiciousjappig�

Prfappi is anonalousjappi is suspiciousg D

F1i � F2i � F3i D

3YjD1

Fj i (7)

In Eq. (7), Fi denotes the status of the i-thapplication, which consists of three viewpoints F1i ,F2i , and F3i . F1i represents the probability of the i-th application in traffic, F2i represents the probabilityof a detected anomaly in Snort or application behavioranalysis in the i-th application, but not in the resultsof the decision analysis module. F3i represents theprobability of an anomaly being found in the decisionanalysis module. F1i , F2i , and F3i can be calculatedby Eqs. (8) – (10).

F1i DNumber of connections to appi

Total number of connections(8)

F2i DNumber of anomaly alert appiNumber of connections to appi

(9)

F3i DNumber of anomaly appi

Number of anomaly alert appi(10)

A weight � is associated with the importance of theappi . k represents the number of the applications. Thenormalized A.2/ from Eqs. (8) – (10) can be given as

A.2/ D1

1C e�

kPiD1

�i

3Qj D1

Fji

(11)

Below is the formula for computing A.3/. Thelikelihood of anomalous behavior in unknownapplications (uapp) can be expressed by consideringfactors such as the probability of presence of theunknown applications in traffic:A.3/ D Prfuappg � Prfanomalyjuappg D N1 �N2

(12)

N1 DNumber of connections to uapp

Total number of connections(13)

N2 DNumber of alerts to uapp

Total number of alerts(14)

So the anomalous performance of the k-th VM from




Eqs. (6), (11), and (12) can be calculated by Eq. (15).

Ak Dvmk

r1

n

nXiD1

.ˇi � dist.Wt ;Wi //C

r2

1C e�

kPiD1

�i

Q3j D1 Fji

C r3

2YiD1

Ni

!C n.t/ (15)

Even a single VM is considered important in the cloud

if it is connected to many VMs, which multiply theimpact of each VM. vmk is an impact factor associatedwith the VM’s importance in the cloud. Now we showhow to compute vmk .

Figure 3 shows a sample connection graph. Eachnode represents a VM, where Vk denotes the k-th VM,and Pj denotes the j-th port of the VM. A connectionbetween V1 and V3 exists if a flow record having theseaddresses is observed. Between nodes V1 and V3 thereare three edges representing three flow records from IPaddress V1 to IP address V3 with different port numbers.

According to given sample, there are three edgesbetween V1 and V3. The vector V.k/ represents theconnections of the k-th VM with other VMs, whereV.1/, V.2/, and V.3/ can be expressed by Eqs. (16) –(18). The matrix V3�3 denoting the connections of thethree VMs, is expressed by Eq. (19).

V.1/ D . 0 0 3 /T (16)

V.2/ D . 0 0 2 /T (17)

V.3/ D . 3 2 0 /T (18)

V3�3 D .V.1/;V.2/;V.3// D

0B@ 0 0 3

0 0 2

3 2 0

1CA (19)

The normalized vmk can be calculated by Eq. (20),where u represents the total number of VMs.

Fig. 3 Flow record sample for VMs connection graph.

vmk D sum .V .W; k//

� uXiD1

sum .V .W; i// (20)

The method proposed here can be used to describethe anomalous performance of VMs. Estimating theanomalous performance of VMs involves evaluating thesituation and trend of the states of the VMs in the cloud.

3 Model Methodology

3.1 Snort

Most security concerns have been addressed, andapplying traditional security can prevent mostintrusions by setting up defenses for each VM[6].Deploying Intrusion Detection Systems (IDS) on thecritical network flow entry is also a feasible solution[7].Traditional IDS[8, 9], intrusion prevention systems,and firewalls can be used to detect attacks in cloudcomputing.

Snort[10] is a free and open source Network IntrusionPrevention System (NIPS) and a Network IntrusionDetection System (NIDS). Snort has the ability toanalyze traffic in real time and log packets. Based ondifferent configurations, Snort has a sniffer mode, apacket logger mode, and a network intrusion detectionsystem mode[11].

We propose using a Bayesian classifier and Snortto detect network intrusions in cloud computingenvironments (see also closely related work inRef. [12]). This approach has few false positivesand affordable computational cost. An OpenFlowand Snort-based Intrusion Prevention System(IPS) is integrated to detect intrusions and deploycountermeasures by reconfiguring cloud computing.Our experimental results demonstrate the feasibilityof this approach (see also closely related work inRef. [13]).

3.2 Data processing

3.2.1 OpenFlowOpenFlow is an open protocol to program a flowtable to deploy new protocols, without changing anynetworking devices, and it implements programmablenetworks. It thus makes it possible to experiment onproduction networks, without danger to operations.McKeown et al.[14] pioneered the control andforwarding separation architecture of OpenFlow.OpenFlow maintains a FlowTable in various switchesand routers. The FlowTable includes packet-forwarding




rules. According to the FlowTable, when a packetarrives at the network device, the rule set determinesthe packet forwarding. With programmable features,OpenFlow enables networks to reconfigure basedon new rules. The paper proposed a new frameworkthat implements network security monitoring usingOpenFlow in cloud computing (see also closely relatedwork in Ref. [15]).

3.2.2 Traffic redirectionInternal virtual networks are invisible in cloudcomputing because their communication traffic does notflow in the same physical machine. Insider threats couldincrease the chance of malware infection of internalVMs and hosts from unknown neighbor applications.Therefore, a large volume of traffic is out of control.This model employs OpenFlow to build a virtualnetwork, so that the virtual switch network traffic runsthrough the physical network card, and the networktraffic flows to the deployed system with our programs.OpenFlow then allows all the network flows to beinspected.

Figure 4 shows the virtual machine network trafficredirection. The model makes use of OpenFlowtechnology. OpenFlow can redirect the network trafficof VMs in the same physical machine to the deployedsystem. This solves the problem that the inter-VMtraffic cannot be monitored and managed. And then themodel employs OpenFlow to reconfigure control rulesto prevent attacks.

3.2.3 AlgorithmWe designed Algorithm 1 to get information fromflows or packets. The data processing module includesdata packet parsing, reorganization of flow session,packet statistics, flow statistics, and a data accessinterface. NPC is the captured network packagecollection, which cannot use Snort to detect anomalous

Fig. 4 Traffic redirection.

Algorithm 1: Data Processing

Input Data: NPCOutput Data: F, G, Mp, Mf

1: While NPC is not null2: Get packet p from NPC;3: Add data packet p to queue p

0

;4: pi get data packet from queue p

0

;5: if (p not null) then6: gi parse the header fields of data packet pi ;7: add gi to G;8: mi compute statistic vector of data packet gi ;9: add mi to Mp;10: if(pi 2 flow fi ) then11: add pi to flow fi ;12: fi update attributes of flow fi ;13: if (pi is the last packet of flow fi ) then14: ni compute statistic vector of flow fi ;15: add ni to Mf;16: end if17: else18: create fi ;19: add pi to flow fi ;20: fi initialize attributes of flow fi ;21: add fi to F ;22: end if23: end if24: end while

network behaviors; F is a flow attribute vector set; andfi is a property of the flow. G is a data packet attributesvector set and gi is an attribute of a packet. Mp is avector of statistical properties of a packet. Mf is a vectorof statistical properties of the flow.

A function of the data processing module is toprepare the dataset used by other modules. The systemprovides a uniform data access interface in order toperform quick and effective behavior detection.

3.3 Application behavior analysis

3.3.1 Application classificationThe variety of network applications in cloud computinghas dramatically increased along with the growth ofusers. Accurate application traffic identification andclassification is important for anomaly detection. Thispaper represents four goals of traffic classification,one of which is detecting unknown application ormalicious flows[16]. Based on different grainedfeatures of network traffic, our research focuses onpacket and flow data for traffic classification. At thepacket level, the information is collected from packetheaders and, optionally, parts of the payload. The




IP quintuple of transport protocol, source IP address,destination IP address, source port, and destinationport are common properties of a flow. At the flowlevel, the information can be collected from flowstatistics. Network traffic classification has attractedmany researchers over the past few years[17–20]. Wefocus on behaviors of applications when they deviatefrom normal behavior. This is a motivation of the workpresented in this paper.

The main characteristics of the network traffic usedto identify the application are number of packets orbytes per second, number of packets payload (only onebyte), number of packets payload (greater than onebyte), sequence of number of byte on the first fivepackets payload, Dstatis of packets payload, Dstatis ofpackets interval, and Dstatis of TTL. Dstatis representsthe statistical value of one characteristic, which containsminimum, maximum, variance, mean, median, anddeviation.

Application behavior analysis consists of twosteps. The first step aims to identify applications.This module manipulates the training examples andproduces multiple classifiers to improve the applicationclassification accuracy. The second step aims to detectanomalous behaviors of the application. This paperadopts the AdaBoost algorithm given in Ref. [21].AdaBoost produces a sequence of k classifiers, such asK-Means, Support Vector Machines (SVM), etc. Theweight for all training examples is equal at beginning.In each iteration, the error of the previous classifier iscalculated. If it is too large, delete the iteration andexit. Training examples that are incorrectly classifiedby the previous classifiers are given higher weights forthe next classifier[22]. The iteration stops until the errorrate reaches a predetermined value.

Figure 5 shows the process of applicationclassification. A application classifier is learnedfrom the labelled training samples during the trainingphase and then the class label of every application isobtained from the trained classifier in the classificationphase. Traffic samples that contain various applications(such as HTTP, QQ, PPLIVE, DNS, SSH, MSN, POP3,etc.) are collected. The module then uses time seriestechnology to analyze applications. As mentionedpreviously, each module will get information from thedata processing module. After identifying applications,this module gets various applications as input and thenwe use time series analysis method to detect anomaliesbased on application behaviors.

Fig. 5 Classification processing.

3.3.2 Time series analysisThe characteristics of networking behaviors are alsoclosely correlated with history data (T1, T2, T3, WT,MT, and YT using Eq. (21)) as mentioned in Section2.3. The time series is defined as in Ref. [23].

TS D fT1; T2; T3;WT;MT;YTg (21)C D .C1; C2; :::; Cm/ represents the value of the time

T , where m is the total number of the applicationcharacteristics, and Ci represents the value of thei-th feature can be any characteristic of a networkapplication (such as byte counts, packet counts, numberof connection requests, source mask bits, destinationmask bits, incoming and outgoing traffic, duration,average connection duration, protocol, packet rate,maximum or average packet, etc.). OC is the predictedvalue at time T. � determines whether the applicationbehavior deviates from normal. This means that somedeviation between the forecast values and the values canbe observed. This deviation is given by Eq. (22).

�.T / D C.T / � OC.T / D .�1; �2; :::; �m/ (22)Here’s how to compute OC using Eq. (23). jTSj

denotes the size of the set TS.OCi D

1

jTSjsum .Ci .t// D

1

jTSj.Ci .T1/C Ci .T2/C

Ci .T3/C Ci .MT/C Ci .YT// (23)

However, if the detection time is too short, youcannot show a regularity; if the time is too long youwill have a lot of historical data as a basis, which is thenext key issue to be resolved, along with determiningthreshold � .

3.4 Decision analysis

In order to improve detection accuracy, the decisionanalysis module uses the anomaly records for in-depth




analysis, where the various computation processesare described in Algorithm 2. This module usesa self-training algorithm, which is an incrementalalgorithm. The known and unknown records ofanomalous behaviors are used to construct a samplelibrary. In this module, a Naive Bayesian classifier[22]

is trained with the labeled set, which is applied toclassify the unlabeled set. Then, the highest-confidencesamples are added to the labeled samples. This processiterates until all the unlabeled samples are added tothe labeled samples.

The features A D .A1; A2; :::; Am/ are extractedfrom anomaly records, which are used to construct asample library. If a number of labeled samples meetthe condition, the system will get results through self-learning. The number of labeled samples will affectthe final result, which is the next problem to besolved. Naive Bayesian is a classifier F. The task ofclassification can be regarded as estimating the classposterior probabilities. In this module, there are twoclasses. One is anomalous behaviors, the other is normalbehaviors.

Each sample is assigned to its most probableclass. The self-training algorithm[22] is given. Z islabeled samples, .a1; a2; :::; am/ represents observedattributes. Using label samples Z, train Naive Bayesian

Algorithm 2: Decision Analysis

Input Data: Features of Network TrafficOutput Data: y D fyes, nog1: Train classifier F use labeled samples Z;2: While unlabelled samples is not null3: F.Q/;4: for r 1 to jQj do5: Compute per class Pr.C D cj /6: Compute per feature Pr.Ai D ai jC D cj /

7: c1 D Pr.c D yes/ �jAjQ

iD1

Pr.A D ˛i jc D yes/

8: c2 D Pr.c D no/ �jAjQ

iD1

Pr.A D ˛i jc D no/

9: if (c1 > c2 and c1 > ") then10: q:y D yes;11: end if12: if (c2 > c1 and c2 > ") then13: q:y D no;14: end if15: Add q:y to Z;16: remove q from Q;17: end for18: end while

classifiers F. This classification is then used to classifythe unlabeled sample Q; then the highest confidencesamples are added to the labeled samples. This processiterates until all the unlabeled data have been given classlabels. This module aims to find out which applicationshave anomalous behaviors.

4 Experiments

The approach is able to establish a behavioral baselineof normal network activity for each service, andthen when network activity deviates from a baseline,anomalous activity will be detected. Zhao et al.[24]

proposed detection botnets for classifying networktraffic behavior, and that it is possible to identifythe presence of existing and unknown botnet activitywith high accuracy. Lin et al.[25] proposed a behavior-based approach that can detect known and evenunknown malware. Koch et al.[26] used behavior-based techniques to detect intrusions in encryptedenvironments. Behavior profiles of each VM andservice are used to detect cooperative anomalousbehavior in our approach.

In order to detect anomalous network behaviors incloud computing, we propose the model presented inFig. 2. For illustration purposes, a cloud environmentwith several nodes is set up and we have usedthis platform to develop the security architecture forIaaS[27]. We deploy an experimental cloud computingplatform based on a QEMU emulator v2.0.0 (Debian2.0.0+dfsg-2ubuntu1), OpenStack IceHouse, and Open-Flow v1.3.

We use the KDD-99 dataset as training data, which isused for the Third International Knowledge Discoveryand Data Mining Tools Competition[28]. It contains4 898 431 network connections with 41 network trafficfeatures. There are seven discrete-valued features,and others are continuous-valued features. KDD-99is well-known and widely used for network attackdetection[29–31]. The system will first preprocess sometext features into numeric features. As shown in Table1, the service type “UDP” is mapped to 2. Thenthe system transforms continuous-valued features intodiscrete-valued features.

KDD-99 is partitioned into ten equal-size disjointsubsets as training data, including six services in Table1. For testing purposes, our system focuses on thesame types of application traffic. Table 2 shows datadistribution of connection records on six services, and




Table 1 Data transformation.Types Class Value

ProtocolTCP 1UDP 2ICMP 3

Service

login 1http 2shell 3smtp 4ssh 5

telnet 6

average accuracy of classification.In this experiment, the dataset is partitioned into ten

equal-size disjoint subsets. The 10-fold cross-validationmethod is used. As shown in Fig. 6, this approachis able to classify almost one hundred percent ofnormal traffic. Detection of attack traffic decreases byapproximate 3%-8% when the dataset is unbalanced foreach class, which is left for future work. The resultsshow that the proposed algorithms are able to classify amajority of the attack traffic. The experimental resultsindicate that the effectiveness of our approach is morethan 90%, and the model can detect attacks accurately.

A Receiver Operating Characteristic (ROC) curve isused to evaluate classification results. We aggregate theclassification results, and demonstrate the effectivenessof this model. Figure 7 shows ROC curves for six

Table 2 Data distribution on six services.Service type Attack (%) Accuracy (%) Attack precision (%)

login 100.0000 99.9020 99.9899http 0.6491 99.6813 95.8351shell 99.5243 99.7106 99.7106smtp 1.2252 99.8299 92.7589ssh 99.3488 99.8095 99.9900

telnet 47.9308 92.1981 90.6363

services. Considering some acceptable behaviors can beclassified as unacceptable, we plan to further evaluatethe proposed approach using false negative analysis inthe future. In terms of the per-service attack sample rate,“login”, “shell”, and “ssh” have the best classificationperformance across all services, due to the existenceof large and long-duration attack flows in the trainingdata. The effectiveness of the algorithms are evaluatedin terms of its ability to distinguish attack traffic fromnormal traffic. We focus on the six services in this workand leave other types of services for future work. Theexperimental results show the feasibility and accuracyof our proposed approach.

5 Conclusion

This paper presents an anomalous behavior detectionmodel in cloud computing that takes into account hybriddata sources and hybrid approaches. Our proposeddetection model can deal with both discrete andcontinuous attributes. Experimental results show thatit has high precision values and low recall values. Themodel uses SDN programmable technology to solvethe inter-VM network traffic that cannot be monitored.The VM states are analyzed to propose efficientcountermeasures to fuse several analysis approaches forpreventing and handling the anomalous traffic of VMs.

A good direction for future work would be to studyweights of samples and optimizing parameters of theproposed algorithm. We also hope to combine a deeplearning algorithm and genetic algorithms to improvethe accuracy of the model.

Acknowledgment

This work was supported by the National Natural ScienceFoundation of China (No. 61272447) and the National

Fig. 6 Classification precision on ten subsets.




Fig. 7 ROC curves of six services.

Key Technologies Research and Development Program ofChina (No. 2012BAH18B05).

References

[1] N. Antonopoulos and L. Gillam, Cloud Computing:Principles, Systems and Applications. Springer ScienceBusiness Media, 2010.

[2] B. Grobauer, T. Walloschek, and E. Stocker, Understandingcloud computing vulnerabilities, IEEE Security & Privacy,vol. 9, no. 2, pp. 50–57, 2011.

[3] U. Oktay and O. K. Sahingoz, Attack types andintrusion detection systems in cloud computing, in2013 6th International Information Security & CryptologyConference, 2013, pp. 71–76.

[4] R. George, Cloud Application Architectures: BuildingApplications and Infrastructure in the Cloud. O’ReillyMedia, Inc., 2009.

[5] Gartner Press Release, Gartnesr says 60 percentof virtualized servers will be less secure than thephysical servers they replace through 2012, http://www.gartner.com/newsroom/id/1322414, 2015.

[6] J. H. Lee, M. W. Park, J. H. Eom, and T. M. Chung,Multilevel intrusion detection system and log managementin cloud computing, in Advanced CommunicationTechnology (ICACT), 2011 13th International Conferenceon. IEEE, 2011, pp. 552–555.

[7] U. Tupakula, V. Varadharajan, and N. Akku, Intrusiondetection techniques for infrastructure as a servicecloud, in Dependable Dependable, Autonomic and SecureComputing (DASC), 2011 IEEE Ninth InternationalConference on, 2011, pp. 744–751.

[8] P. Casas, J. Mazel, P. Owezarski, P. Casas, and J.Mazel, Unsupervised network intrusion detection systems:Detecting the unknown without knowledge, ComputerCommunications, vol. 35, no. 7, pp. 772–783, 2012.

[9] L. Koc, T. A. Mazzuchi, and S. Sarkani, A networkintrusion detection system based on a hidden naı̈ve Bayesmulticlass classifier, Expert Systems with Applications, vol.39, no. 18, pp. 13492–13500, 2012.

[10] Snort, https://www.snort.org, 2015.[11] Snort Users Manual, http://manual.snort.org, 2015.[12] C. N. Modi, D. R. Patel, A. Patel, and R. Muttukrishnan,

Bayesian classifier and Snort based network intrusiondetection system in cloud computing, in ComputingCommunication & Networking Technologies (ICCCNT),2012 Third International Conference on, 2012, pp. 1–7.

[13] T. Xing, D. Huang, L. Xu, C. J. Chung, and P. Khatkar,Snortflow: A openflow-based intrusion prevention systemin cloud environment, in Research and EducationalExperiment Workshop (GREE), 2013 Second GENI, 2013,pp. 89–92.

[14] N. McKeown, T. Anderson, H. Balakrishnan, G. Parulkar,L. Peterson, J. Rexford, S. Shenker, and J. Turner,OpenFlow: Enabling innovation in campus networks,Computer Communication Review, vol. 38, no. 2, pp. 69–74, 2008.

[15] S. Shin and G. Gu, CloudWatcher: Network securitymonitoring using OpenFlow in dynamic cloud networks(or: How to provide security monitoring as a service inclouds?), in Network Protocols (ICNP), 2012 20th IEEEInternational Conference on, 2012, pp. 1–6.

[16] A. Callado, C. Kamienski, G. Szabo, B. P. Ger, J.Kelner, S. Fernandes, and D. Sadok, A survey on internettraffic identification, IEEE Communications Surveys andTutorials - COMSUR, vol. 11, no. 3, pp. 37–52, 2009.

[17] J. Zhang, Y. Xiang, W. Zhou, and Y. Wang, Unsupervisedtraffic classification using flow statistical properties and IPpacket payload, Journal of Computer and System Sciences,vol. 79, no. 5, pp. 573–585, 2013.

[18] J. Zhang, Y. Xiang, Y. Wang, W. Zhou, Y. Xiang, andY. Guan, Network traffic classification using




correlation information, IEEE Transactions on Paralleland Distributed Systems, vol. 24, no. 1, pp. 104–117, 2013.

[19] Y. Jin, N. Duffield, J. Erman, P. Haffner, S. Sen, and Z.Zhang, A modular machine learning system for flow-leveltraffic classification in large networks, ACM Transactionson Knowledge Discovery From Data (TKDD), vol. 6, no.1,p. 4, 2012.

[20] A. Tongaonkar, R. Torres, M. Iliofotou, R. Keralapura,and A. Nucci, Towards self adaptive network trafficclassification, Computer Communications, vol. 56, no. 1,pp. 35–46, 2015.

[21] Y. Freund and R. E. Schapire, Experiments with a newboosting algorithm, in Int’l Conf. Machine Learning(ICML), 1996, pp. 148–156.

[22] B. Liu, M. J. Carey, and S. Ceri, Web Data Mining.Springer, 2011.

[23] G. E. P. Box, G. M. Jenkins, and G. C. Reinsel, Time SeriesAnalysis: Forecasting and Control. John Wiley & Sons,2008.

[24] D. Zhao, I. Traore, B. Sayed, W. Lu, S. Saad, A. Ghorbani,and D. Garant, Botnet detection based on traffic behavioranalysis and flow intervals, Computers & Security, vol. 39,pp. 2–16, 2013.

[25] Y. D. Lin, Y. C. Lai, C. N. Lu, P. K. Hsu, andC. Y. Lee, Three-phase behavior-based detection and

classification of known and unknown malware, Securityand Communication Networks, vol. 8, no. 11, pp. 2004–2015, 2015.

[26] R. Koch, M. Golling, and G. D. Rodosek, Behavior-based intrusion detection in encrypted environments,Communications Magazine, vol. 52, no. 7, pp. 124–131,2014.

[27] L. Chen, X. S. Chen, J. F. Jiang, X. Y. Yin, and G. L.Shao, Research and practice of dynamic network securityarchitecture for IaaS platforms, Tsinghua Science andTechnology, vol. 19, no. 5, pp. 496–507, 2014.

[28] KDD Cup 1999 Data, http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html, 2015.

[29] P. A. R. Kumar and S. Selvakumar, Detection of distributeddenial of service attacks using an ensemble of adaptive andhybrid neuro-fuzzy systems, Computer Communications,vol. 36, no. 3, pp. 303–319, 2013.

[30] S. S. Sathya, R. G. Ramani, and K. Sivaselvi, Discriminantanalysis based feature selection in kdd intrusion dataset,International Journal of Computer Applications, vol. 31,no. 11, pp. 1–7, 2011.

[31] P. Casas, J. Mazel, and P. Owezarski, Unsupervisednetwork intrusion detection systems: Detecting theunknown without knowledge, Computer Communications,vol. 35, no. 7, pp. 772–783, 2011.

Xiaoming Ye is a PhD candidate atCollege of Computer Science of SichuanUniversity. She got the BE degree fromCollege of Information Engineering ofJiangnan University in 2005 and MSdegree from College of Computer Scienceof Sichuan University in 2008. Herresearch interests include cyber security

and big data analytics.

Xingshu Chen received the PhD degreefrom Sichuan University in 2004. Sheis now a professor of the College ofComputer Science and CybersecurityResearch Institute of Sichuan University.She is the member of China InformationSecurity Standardization TechnicalCommittee. Her research interests include

cloud computing, cloud security, distributed file system, bigdata processing, network protocol analysis, and new mediasupervision.

Haizhou Wang received the BE degreeand PhD degree from College of ComputerScience, Sichuan University, China, in2008 and 2014, respectively. From 2013to 2014, he visited University of Toronto.He is currently a lecturer in the Collegeof Computer Science, Sichuan University,China. His research interests include peer-

to-peer streaming system, information security, and networkmeasurement.

Xuemei Zeng is a PhD candidate atCollege of Computer Science of SichuanUniversity. She received the MS degreefrom Computer Science College of SichuanUniversity in 2004. Her current researchinterests include computer and networksecurity, big data, and cloud computingsecurity.

Guolin Shao is a PhD candidate ofCollege of Computer Science of SichuanUniversity. He got the BE degree fromSichuan University in 2013. His generalresearch interests lie in cyber security.

Xueyuan Yin is a PhD candidate atCollege of Computer Science of SichuanUniversity. He got the BE degree fromSichuan University in 2008. His researchinterests mainly focus on computernetwork and information security.

Chun Xu received the PhD degree fromSichuan University in 2008. He is nowan associate professor of the Collegeof Cybersecurity Research Institute ofSichuan University. His research interestsmainly focus on computer network andinformation security.

