Obfuscating Function Call Topography to TestStructural Malware Detection against Evasion

AttacksAndrew Choliy

School of Arts and SciencesRutgers University

New Brunswick, New Jersey, U.S.A.Email: ayc41@scarletmail.rutgers.edu

Feng Li and Tianchong GaoDepartment of Computer and Information Technology

Indiana University-Purdue University IndianapolisIndianapolis, Indiana, U.S.A.

Email: (fengli, tgao)@iupui.edu

Abstract—The incredible popularity of the Android mo-bile operating system has resulted in a massive influxof malicious applications for the platform. This malwarecan come from a number of sources as Google allowsthe installation of Android App Packages (APKs) fromthird parties. Even within its own Google Play storefront,however, malicious software can be found.

One type of approach to identify malware focuses on thestructural properties of the function call graphs (FCGs)extracted from APKs. The aim of this research work is totest the robustness of one example method in this category,named the ACTS (App topologiCal signature throughgraphleT Sampling) method. By extracting graphlet statis-tics from a FCG, the ACTS approach is able to efficientlydifferentiate between benign app samples and malwarewith good accuracy.

In this work, we obfuscate the FCG of malware in severalways, and test the ACTs method against these evasionattacks. The statistical results of running ACTS againstunmodified real malware samples is compared with theresults of ACTS running against obfuscated versions ofthose same apps.

I. INTRODUCTION

Android is and likely will continue to be a remarkablysuccessful mobile platform. In fact, it powers more thanjust smartphones and tablets - in recent years, several set-top boxes and wearable devices have been released withthe operating system. However, as Android becomesmore and more ubiquitous, it also becomes a morefavorable target for malware developers.

Several methods have been proposed to detect and/ordefend against this malware, each with their own ben-efits and drawbacks. App topoligiCal signature throughgraphleT Sampling (ACTS) is one such approach, andis the focus of this paper. It examines certain structuralfeatures of an APK’s FCG, so chosen for its immunity tolower-level instruction obfuscation and higher-level con-tent encryption, among other things [1]. Namely, ACTSuses the frequency distribution of various graphlets ina function call graph. A graphlet is small connected

subgraph; in ACTS, they consist of no fewer than 3nodes and no more than 5. The method takes advantageof the fact the much of the malware on Android canbe grouped into families which often share some sourcecode and have some degree of similarities in theirstructure [2].

There are some potential challenges to ACTS, how-ever. While a FCG may be immune to certain afore-mentioned obfuscation techniques, it may be particularlyvulnerable to useless functions and function calls; thatis, functions and function calls whose sole purpose isto manipulate the FCG obtained from an APK file. Amalicious developer could strategically create functionsand/or function calls to manipulate the FCG of theirsoftware to resemble a legitimate app, which in turn willfool ACTS. Creating graphlets made up of the uselessfunctions will change their distribution statistics, whichis exactly the information that ACTS uses for detection.We experiment with a few potential strategies a malwaredeveloper may utilize to evade detection by ACTS. Assource code for Android malware is not readily available,we instead directly modify the function call graphsextracted from the malicious APK files via Androguard.While less convenient, it is still reasonably feasible fora malware developer to appropriately add functions andfunction calls in the source code of their application.Note that any modification we make to the FCG willinvolve adding a function or function call, as removingthose may impair intended operation of the software.

The obfuscation strategies we test ACTS against in-clude random edge (i.e. function call) insertion, creat-ing graphlets via specific edge insertion, and creatinggraphlets via specific edge and node (i.e. function calland function) insertion. We examine two slight variationsof the latter two techniques that aim to reduce the amountof new edges and/or nodes required to successfullyobfuscate the FCG. We also briefly discuss ways that

___________________________________________________________________

This is the author's manuscript of the article published in final edited form as:Choliy, A., Li, F., & Gao, T. (2017). Obfuscating function call topography to test structural malware detection against evasion attacks. In 2017 International Conference on Computing, Networking and Communications (ICNC) (pp. 808–813). https://doi.org/10.1109/ICCNC.2017.7876235

a malicious developer could obfuscate his program thatwe cannot adequately model by only modifying a FCG.

To summarize, the main contributions of this paperare as follows:

• We identify and design various obfuscation tech-niques a malicious Android application developercan utilize to evade function call graph based mal-ware detection methods.

• We implement and test these techniques using realmalware against the ACTS program and present ourresults.

The rest of the paper is organized in the followingway: In section II, we provide the reader with anoverview of the ACTS scheme and give some necessarybackground information. Section III details the variousobfuscation techniques we used in an attempt to evadedetection by ACTS. Section IV presents the results ofour experiment. The final three sections present relatedwork, potential future research, and concluding remarks,respectively.

II. ACTS OVERVIEW

Now we will explore how ACTS works with a littlemore detail. This section provides enough information tounderstand how our evasion techniques are implemented,but for a more comprehensive explanation of the pro-gram, see [1]. The notation and visual representation weuse for graphlets, nodes, etc. are the same as they appearin that paper. The steps ACTS takes to classify maliciousAPK files is as follows:

A. Extracting the Function Call Graph

First, a function call graph is extracted from an APKfile. This FCG is directed graph that represents functioncaller and callee relationships in a piece of software.Androguard is used to extract a FCG from the Javabytecode of an APK file.

B. Estimation of Graphlet Frequency Distribution(GFD)

Next, ACTS considers the graphlets in the FCG. Aspreviously mentioned, a graphlet is a small, connected

Fig. 1. The thirteen unique graphlets with 3 nodes ω3,i(i = 1, 2, ...13)

subgraph of a FCG. These graphlets capture topologicalinformation on local neighborhoods. The set of uniquegraphlets with 3 nodes may be enumerated as ω3,i (i =1, 2, ... 13). Graphical representation of this set is shownin Fig. 1. There are 199 unique 4-node graphlets, 9,3645-node graphlets, and 1,530,843 6-node graphlets. ACTSonly considers graphlets up to size 5, as any more thanthat will significantly increase computational costs whileproviding little extra accuracy.

ACTS uses the number of occurrences for each typeof graphlet to find the Graphlet Frequency Distribution(GFD) of a Graph G. GFD is a vector of the probabilitydistribution of each graphlet type in G. To elaborate,suppose that we enumerate every 3-node graphlet in G(which is finite). Then, define f3,i to be the numberof times that graphlet ω3,i appears in G. Then, thefrequency distribution density d3,i = f3,i/Σ13

i=1f3,i. Thevector (d3,1, d3,2, ...d3,13) is the 3-graphlet frequencydistribution (3-GFD) of G. Concatenate this with the 4-GFD and 5-GFD (which will have 199 elements and9,364 elements, respectively) to get the full 9,576-dimensional Graphlet Frequency Distribution vector ofG.

Unfortunately, actually enumerating every graphlet ina FCG is not feasible in a reasonable time frame. It ispossible to estimate graphlet distributions without doingso, however. To accomplish this, ACTS makes use ofthe Metropolis-Hastings algorithm (M-H algorithm) [3],which is a Markov chain Monte Carlo (MCMC) method,to estimate the GFD of graph G. For further details onthe estimation method, see [3] or the appropriate sectionof [1] for more information.

C. Projecting GFD Vector to Lower Dimension

In practice, ACTS does not use the entire GFD vector.Instead, we consider the set of Android applicationsfor which the full GFD vectors have been obtained byPeng et al [1]. From this set, we select graphlets whichhave a density of 2 percent or more in at least oneapplication. As a result, the number of dimensions ofthe GFD vector is reduced from 9,576 to 96 (5 3-nodegraphlets, 20 4-node graphlets, and 71 5-node graphlets).This step is necessary to avoid issues that many machinelearning algorithms have when dealing with vectors ofhigh dimensionality [4]. The new GFD vector is referredto as the topological signature of an app.

D. Classification via Support Vector Machine

Lastly, ACTS uses the reduced GFD with the LIB-SVM support vector machine (SVM) library to classifyits corresponding APK file as malicious or not, withresults of up to 87.9 percent accuracy.

Fig. 2. From left to right, the three obfuscation methods are: Random Edge Insertion (REI), Edges Only Graphlet Insertion (EOGI), and Edgesand Nodes Graphlet Insertion (ENGI). In REI, we see two new edges randomly added to the FCG. In EOGI, we see three edges added in sucha way to increase the distribution of ω3,4 graphlets, and in ENGI, we edges and nodes added in such a way to increase the distribution of ω3,5

graphlets.

III. METHOD

Here, we elaborate on the various obfuscation tech-niques we employed on real malicious software in anattempt to evade detection by ACTS. They can be brieflysummarized as follows:

1) Random Edge Insertion: We simply insert randomedges (i.e. function calls) on an APK’s FCG. Thisserves as a sort of baseline that shows some degreeof robustness for ACTS, as well as how our otherstrategies perform.

2) Edges Only Graphlet Insertion: We create onlyedges in the FCG to link existing nodes so that theyform graphlets that do not appear often enough ina malicious FCG.

3) Edges and Nodes Graphlet Insertion: We createboth edges and nodes to form graphlets that donot appear often enough in a malicious FCG, andform a few more edges to link back to the mainbody of the graph.

Fig. 2 shows a visual representation of how each pre-sented obfuscation strategy works.

A. Random Edge Insertion

The first technique we employ is Random Edge Inser-tion (REI). It simply consists of randomly finding twounconnected nodes in a graph G, and adding a directededge from one to the other. The amount of total edgesin G is increased by 50 percent to 100 percent.

This method may seem ultimately inconsequentialand a malware developer would likely not consider thisstrategy in a real-world scenario, but there is somerationale behind our decision to include it here. Aspreviously mentioned, Android malware can generally beclassified into one of several families which usually sharesome code and underlying structure. This commonalitybetween malware is crucial to how ACTS functions.Benign apps, on the other hand, presumably are notnearly as closely related, so perhaps something as simpleas adding random edges to a FCG would enable themalware to avoid detection.

At the very least, REI could be a heuristic indicationof ACTS’ resistance to basic obfuscation and a baselineto measure our other methods against.

Fig. 3. These graphlets have the highest frequency distributions amongthe function call graphs of benign Android applications.

B. Edges Only Graphlet Insertion

The second obfuscation strategy we use is EdgesOnly Graphlet Insertion (EOGI). ACTS stores the GFDof every APK sample it analyzes, as well as statisticsabout the most common graphlets in malware and benignsoftware by average distribution. Essentially, we treat thestatistics of the benign software as a sort of target. Weexamine the top five graphlets that appear in benign soft-ware (see Fig. 3; it is identical to the results obtained by[1]). We then look at the distribution of those graphletsin the malicious FCG, and add edges to existing nodesin order to increase the number of a particular graphlet.These (unconnected) nodes are chosen randomly. So, forexample, if the extracted FCG G of some malware hasa relative shortage of graphlet ω3,5, we could randomlyselect three unconnected nodes a, b, and c and add anedge from a to b and from a to c. Each graphlet typewill require its own specific manipulation of the FCG.

The above example illustrates the basic concept ofEOGI, but we often deal with relatively large FCGsthat can have thousands of nodes. In the interest ofminimizing edge additions to optimize obfuscation, weconsider two slightly more sophisticated versions of thismethod:

• The first involves adding a substructure that con-tains multiple instances of a graphlet while loweringthe number of edges added per graphlet. Fig. 4(a) illustrates the following example: consider theprevious scenario where a malicious FCG has ashortage of graphlet ω3,5. Now we randomly select6 nodes d, e, f, g, h, and i. Add an edge from d toevery other selected node. As the illustration shows,this structures results in ten ω3,5 graphlets beingadded even though only five edges were added;much improved over adding three edges for onegraphlet. Similar to before, each graphlet will needa specially designed substructure to achieve thiseffect.

• The second version attempts to find existing sub-graphlets in a FCG and add edges to transition itinto a specific graphlet. Going back to the scenario

Fig. 4. In (a) inserting that particular substructure increases the amountof ω3,5 graphlets by 10. In (b), we add only one edge to create anotherinstance of graphlet ω3,5.

where there were too few ω3,5 graphlets, the processis as follows (depicted in Fig. 4 (b)): select arandom node j. Ensure that it already has at leastone outgoing edge, and then add another edge to anunconnected node. This forms at least one instanceof graphlet ω3,5. Once again, different graphletshave different variations of this technique.

C. Edges and Nodes Graphlet Insertion

A weakness of EOGI is the potential ripple effectthat each added edge can have. Edges are often partof numerous graphlets, and an added edge may inad-vertently cause the distribution of undesirable graphletsto increase. Additionally, it is possible to not even addthe correct graphlet with EOGI. Consider an attempt toincrease the distribution of graphlet ω3,5 by adding anadd edge to an existing subgraphlet that consists of twonodes with one edge between them, as shown in Fig. 4(b). However, if the node that an edge is being made toalready has an edge with another node in the graphlet(i.e. if the bottom node is already connected to the topnode), it will cause the distribution of graphlet ω3,9 toincrease instead.

To mitigate the impact of this phenomenon, we con-sider a third obfuscation strategy: Edges and NodesGraphlet Insertion (ENGI). By and large, it follows thesame logic as EOGI, but nodes are created instead ofselected. The two variations of ENGI are similar to thoseof EOGI as well. The example presented in Fig. 4 canjust as easily be used to illustrate to ENGI. In the caseof the substructure insertion variant, ENGI’s distinction

is that nodes e, f, g, and h are created. Nodes d andi are ones that already existed in the FCG, and areused to connect the substructure to the main graph. GFDestimation will not consider unconnected sections. Whenusing the graphlet transition variant of ENGI, the middleand upper nodes as well as the upper edge are alreadypresent in the FCG, while the bottom node and edge arecreated.

IV. RESULTS

To test our obfuscation techniques, we considered 20random FCGs that came from known malware. In ACTS,these were all true positives; that is, correctly classifiedas malware. We applied each technique on the samplesand recorded any difference in ACTS’ classification.The results of our experiments are promising. As Fig. 5shows, both ENGI’s and EOGI’s substructure insertionvariant performed well, both causing ACTS to change itsoriginal correct malicious classification of 17 maliciousFCGs from a total of 20. EOGI’s substructure insertionperformance came as a surprise, however. Intuitively, thenumber of unintended graphlet additions would leavetoo much of an impact for good results (which we feelmay be the case for the graphlet transition variant ofEOGI), but the data indicates otherwise. The graphlettransition version of ENGI also performed well with14 misclassifications, though EOGI’s version did poorlywith only 6. REI’s poor performance suggests that eventhough much of Android can fit into one of severalstructural molds, simply disrupting this structure is notenough. Legitimate pieces of software may not by aseasy to categorize, but there are still some conventionsin the source code that are reflected in a FCG. REI wasnot able to reflect this in the modified graph.

A. Discussion

An important element of this obfuscation to consideris practicality and cost for the malicious developer. Wefound that the number of graphlets required to changethe GFD to a desirable value was anywhere from a fewhundred to several thousand, depending on the size of theFCG and its current GFD. Often, however, this meansthat the number of edges and nodes (when applicable)increases by several times. While not necessarily dif-ficult, adding all of those functions and function callsto a program can be time-consuming. EOGI is morefeasible than ENGI, as only function calls are added.There are several things that can be done to mitigate thisissue, however. Our implementation of the substructurestrategy involved six nodes and five edges, and resultedin 10 graphlets being added. Adding another node andedge to it would cause 15 graphlets to be added; yetanother node and edge would further increase that to21, and so on. A large enough substructure will reduce

Fig. 5. This figure shows how many apps were misclassified by ACTSafter our obfuscation techniques were applied to them. Here, v1 is thesubstructure insertion variant of EOGI/ENGI, and v2 is the graphlettransition variant.

the workload. Adopting a similar idea with the graphlettransition strategy would make it virtually indistinguish-able from structure insertion; the only difference is thatit would guarantee the presence of a subgraphlet thatexisted in the original FCG. Perhaps a more advancedversion of graphlet transition could identify multiplesubgraphlets or graphlets and connect them together tocreate a desired substructure. The process of findingsuitable subgraphlets and graphlets may be non-trivial,however. Additionally, the malicious developer is able toremove or condense functions, which means he or shecan modify a FCG in new ways.

V. RELATED WORK

There is little to no research that focuses specificallyon obfuscating the function call graphs of applicationsas we have done here. The idea itself of using a FCG todetect malicious software is relatively novel. Obviously,Peng et al. [1] created ACTS. [5] implements a similarapproach, though they also take into account features ofeach node in the graph, not just the overall structure.Testing our obfuscation techniques and devising newones to combat their detection could be interesting. FCGanalysis can also be used for other purposes; for instance,Zhou and Jiang [6] use it to detect specific vulnerabilitiesin Android applications.

VI. FUTURE WORK

Aside from further investigation and refinement of thepresented obfuscation techniques, there are two mainareas we believe will result in the most interesting andpractical related research. The first is studying otherFCG obfuscation techniques, especially ones that requiredirect editing of the source code (and thus are impossible

to do with our own approach). One such example wouldbe function inlining, which essentially involves condens-ing several function into one larger function. This couldenable the deletion of nodes and edges in the FCG, whichour approach was unable to do.

The second interesting area of future study is obfusca-tion detection. Certain bytecode or structural clues couldpotentially indicate FCG obfuscation. For instance, thenodes (i.e. functions) added by ENGI would generallyexist only for other functions to call or be called by. Find-ing these ”useless” functions even with just bytecode ispossible, and indeed could be gleaned by using [5]’sFCG node and neighborhood analysis. [5] also foundthat the average number of outgoing function calls froma node is slightly above two, so detecting REI or EOGIcould be made possible by using that and perhaps otherstatistics.

VII. CONCLUSION

In this paper, we present practical obfuscation tech-niques that malicious Android software developers couldimplement in order to avoid FCG-based malware detec-tion. Namely, we show the effectiveness of some of thesetechniques against an example of this detection software:ACTS. By adding edges or nodes and edges, we are ableto manipulate a malicious FCG to sufficiently changeits graphlet frequency distribution vector, thus foolingACTS into classifying it as benign software.

REFERENCES

[1] W. Peng, T. Gao, D. Sisodia, T. K. Saha, F. Li, and M. A. Hasan,ACTS: Extracting Android App Topological Signature throughGraphlet Sampling.

[2] Y. Zhou and X. Jiang, Dissecting Android malware: Characteri-zation and evolution. In Proc. of IEEE Symposium on Securityand Privacy (S&P), 2012

[3] N. Metropolis, A. W.Rosenbluth, M. N.Rosenbluth, A. H. Teller,and E. Teller. Equation of state calculations by fast computingmachines. The Journal of Chemical Physics Physics, 21(6):1087 - 1092, 1953.

[4] P. Indyk and R. Motwani. Approximate nearest neighbors: towardsremoving the curse of dimensionality. In Proc of ACM Sympo-sium on Theory of Computing (STOC), 1998

[5] H. Gascon, F. Yamaguchi, D. Arp, and K. Rieck. Structuraldetection of Android malware using embedded call graphs. InProc. of ACM Workshop on Artificial Intelligence and Security(AISec), 2013

[6] Y. Zhou and X. Jiang Detecting Passive Content Leaks andPollution in Android Applications. In Proc. of NDSS Symposium,2013