Repucall: Towards Describing the Reputability of Phonenumbers

Saran Ahluwaliassaluwa@ncsu.edu

AbstractAutomated calls, known as robocalls, have significantlydegraded the usefulness and trustworthiness of the phonenetwork. Individuals, enterprise organizations, andproviders have no effective solution to address this prob-lem. A significant barrier to progress in this space is thelack of visibility into the practices of automated callers.

In this paper, we are the first to apply an unsupervisedretrieval technique to crowd-sourced data (COC) in or-der to describe the authenticity of phone numbers. Weconsider over 100,000 complaints collected via crowd-sourced efforts (e.g., 800notes.com). Overall, our resultsshow that the proposed generative probabilistic modelproduces topics that are quantitatively and qualitativelybetter than matrix decomposition techniques, such as La-tent Semantic Analysis (LSA). In this paper, we proposethis model as a more efficacious mechanism for authen-ticating phone numbers. The proposed model identifiesmalicious phone-numbers adaptively. Additionally, themodel labels and ranks numbers by their re-use acrossdifferent scam campaigns. This last insight, in particu-lar, possesses implications for research combating phish-ing attacks, account fraud and identity theft.

1 IntroductionThe phone voice channel has many vulnerabilities and af-fords a very appealing attack surface for spam campaigns,robocalls, and phishing scams. Moreover, there have beenover seven million complaints made to the FTC, regardingunsolicited calls made to consumers.[1]. There are numer-ous solutions, such as YouMail [2], and TrueCaller[3] thathave surfaced in order to combat these unwanted calls.

Previous state-of-the-art work introduced the firstmethodology for blacklisting the following data sources:honeypot call detail records (CDR), and Federal TradeCommission public reporting [27]. In this work, the au-thors leverage latent semantic indexing (LSI) in order todescribe their data sets. Through this procedure, the au-thors do not assess topic coherence [24], but instead revertto manual selection for both the topics and for the opti-mal number of topics to use for modeling a call transcript.This results in manual analysis of topics and the discard-

ing of topics from a transcript’s topic mixture vector. Thisis due to the inherent issues in LSI: that topic mixture vec-tors are not humanly readable [20]. Evaluation is possiblethrough finding similar words for each word in the latentspace, but are otherwise not human interpretable.

In addition, the previous state-of-the-art work decidedon the number of topics based on certain heuristics thatrely on domain expertise. A conventional approach is tosort the cumulative singular values in descending orderand then a cut-off [21]. However, there is variability be-tween samples; hence manual analysis is often needed fornew training.

Within this context, we organized our research ques-tions around the following:

1. What proportion of COC corpora contain a phonenumber, and if so, does a given complaint containimportant information (wireless service provider andgeographic location of the source of the call etc.)?

2. Of those numbers from the previous question, areany of these malicious numbers seen across differentscam campaigns?

3. Can we use any other information retrieval tech-niques to measure phone reputation? If so, how dowe assess these techniques?

To fill this knowledge gap, in this paper we systemati-cally investigate COC data sources that may be leveragedto automatically learn new facets of crowd-sourced report-ing.

In summary, we provide the following contributions:

• We cluster and describe phone numbers with thecrowd sourced online complaints. Our analysisuncovers numerous malicious behaviors, includingbulk scam and phishing campaigns that share themeswith specific phone numbers. Most critically, ouranalysis shows that numbers, which are being usedto deliver malicious scam campaigns, directly under-mine mitigations from previous work [17].

• In order to associate phone numbers that are partof a long-running scam campaign, we apply a novelprobabilistic generative model on user reported com-plaints. We then identify the top campaigns from

1

each data source, and introduce an evaluation met-ric to assess the coherency of the topic model. Thishas particular implications for blacklist construc-tion and for systems-level application developmentthat would be consumed by users. Specifically, ourmodel enables keyword-specificity for searching forscam campaigns and for similar complaints associ-ated with a scam campaign theme and geographicallocation.

• We show that our model is more resilient to threats tointernal validity and to the reputability of the resultsthat we are generating from the model. These threatsare described, in detail, in Section 3.

The remainder of this paper proceeds as follows. Sec-tion 4 describes the implementation of our pipeline (pre-sented in Figure 1.). Section 5 demonstrates various meth-ods to assess the quality of our topic modeling method-ology. Section 7 discusses implications for aggregatingCOC complaint data and shortcomings in our approach.Section 8 describes related work. Section 9 provides con-cluding remarks.

2 Data Collection

2.1 Crowd-Sourced Online ComplaintsThis dataset contains the online comments obtained frompopular online forums, such as 800notes.com). Thedataset contains over 100,000 actual raw complaints filedby users, containing the phone number that made the un-wanted call, a timestamp, and a description of the contentof the call. The complaints stem from May 10, 2010 un-til September 17, 2017. Since the comments are enteredmanually by anxious and frustrated users, the text describ-ing the content of the call is typically comprised of manymisspellings, grammatically inaccurate sentences, exple-tives, and in some cases, irrelevant information. In addi-tion, many of the complaints contain URLs to resourcesand links to external fraud-prevention resources. Anotherexternality to consider is that the phone number and times-tamp provided by the users could be mistyped.

2.2 A Note on the Inclusion of Spam Mes-saging Feeds

Our original proposal for this research investigation stip-ulated that we also apply this same model to spam mes-saging data provided by Project HoneyPot [4]. From thiswe would evaluate a COC-only trained blacklist, a spammessaging-only trained blacklist and a COC and spammessaging blacklist in terms of unwanted call blockingrates. We would then conduct experiments that show

whether or not spam and COC data sources provide addi-tional insights into the longevity of spam campaigns. Thiswould complement insights into both the origin and thereuse of phone numbers for robocalls. Finally, we wouldcompare our blacklists’ classifications to two third-partyservice providers.

However, we were unable to obtain the dataset fromProject HoneyPot. In addition, the length of time forconstructing, training and evaluating each blacklist’s CallBlocking Rates (CBR) - as detailed by S. Pandit et al.[27] - was beyond the allotted time that was availableto the researchers that first implemented these experi-ments. Therefore, we were unable to validate many ofthese aforementioned hypotheses. We will leave this forfuture work.

3 Threat Model

Our threat model is grounded in the validity of our classi-fication of specific phone numbers. While it is difficult toensure the authenticity of numbers, we are assuming thatan external adversary is attempting to pollute the modelclassification task. This is a direct threat to the internalvalidity of our testing through selection bias of the in-put samples. For example, complaints associated witha phone number could be flooded with new commentsthat are favorable or describe the phone number with en-tirely different sets of words. This causes the model tomis-classify or produce topics that are not human inter-pretable. This also has another assumption that the adver-sary understands the model’s classification task. Hencewe are specifically assuming that the adversary is con-cerned with undermining the reputation of the classifica-tion task. In section 5 we propose and we evaluate theresiliency of our model to word intrusion and internal va-lidity threats.

4 Methodology and Design

As depicted in Fig. 1, the main components of our spamcampaign detection, analysis and investigation frameworkare the following:

4.0.1 Persistent Storage

Due to the volume of corpora, a NoSQL database - par-ticularly a document-based database - was used insteadof a SQL-based relational database management system(RDBMS). Document-oriented databases were favoredfor their flexibility and high scalability. However, onecaveat is the lack of a powerful query language like SQL.Moreover, the ability to represent relationships between

2

Figure 1: Architecture of pipeline

comments, scam campaigns and domain names is essen-tial. These relationships can be stored and processed ef-ficiently in a graph database. In our proposed system, weemploy MongoDB (MongoDB, 2017), which is an open-source NoSQL database [5]. MongoDB has the flexibil-ity of document-based databases as well as the capabilityof graph-based ones for managing relationships betweenrecords. It has the advantages of being performant, scal-able and supports SQL-styled queries. Complaints arestored in the database as documents with different fields,such as title, textual content, embedded URLs, etc. Addi-tionally, user information, phone-numbers, domain namesand timestamps of the correspondence are also saved asdifferent document types. For each document type, thedatabase maintains a hash table of a specific field to elim-inate duplications. The database also stores the relation-ships between different documents, such as between theforum correspondence, scam campaign, and phone num-ber.

4.0.2 Elimination of Stopwords

Our implementation of Latent Dirchlet Allocation (LDA),and the pre-processing of corpora have been influencedheavily by the seminal work of Blei at al [11]. Specifi-cally, previous work has detailed that one should removecommon stop words in a document before running LatentDirichlet Allocation [12]. Stop words are words that arevery common and do not alter the semantic meaning of asentence. Typical stop words include ”in”, ”the” and ”as”etc. Kim et al. automatically filtered out words from thevocabulary that were present in more than 50 percent orless than 5 percent of the documents [33]. Kim et al. statethat this heuristic is an effective way of removing bothstop words, misspelled words and non-words. Therefore,in our implementation, a static list of common stop wordswas used as suggested by Blei et al [12].

4.0.3 Stemming

In this project the NLTK version of the Snowball Stem-mer [6] was used. This project is originally based on thestemmer proposed by Lovins [22]. Stemmers do not takeinto consideration polysemy. For example, the Snowballstemmer stems both ”informational” and ”informality” tothe word inform. Because ”space” and ”spacial” do notpossess the same semantic meaning there is a risk of los-ing information when using stemming [32].

However, due to prior works’ common practice to usea stemmer when performing natural language processing,all experiments were conducted with the snowball stem-mer.

• Phone Number: Each phone number is extractedand parsed into a format that demarcates it’s countrycode, time zone and carrier provider. Initial parsingand validation of the phone number was conductedusing Google’s open sourced libphonenumber mod-ule [7].

• Complaint: We discarded complaints that containedless than 5 words or that only contained URLs. Inaddition, we only considered complaints that hadbeen ”upvoted” (an indicator for how useful the com-plaint was to other users). Finally, we discarded anycomplaints that only contained profanity and/or com-ments that derided another user. This resulted in a 91percent reduction in the total number of individualdocuments that were considered for topic modeling.

4.0.4 Online Learning for Latent Dirichlet Alloca-tion (LDA) Models using (Online VariationalBayes) applied to COC datasets

The Latent Dirichlet Allocation can intuitively be de-scribed as a model that identifies topics of documentsbased on the word frequency distribution in each docu-ment. Therefore, the words in a document are dependenton the latent topic distribution. The Latent Dirichlet Al-location relies on the Dirichlet prior that the words in adocument are generated based on the topic distribution ofthe document [12]. If one were to assume that this is thecase we can use any method of posterior inference to in-fer the latent variables in the Latent Dirichlet Allocationmodel. Both Blei et al. and Kim et al. used an onlineversion of the variational Bayes inference model [33].

We have adopted a similar implementation proposed byHoffman et al. [20]. Hoffman et al. provides an on-line version of Latent Dirichlet Allocation. The differ-ence from the original Latent Dirichlet Allocation is thatit requires only one pass over the data set. Hoffman et al.define online learning for Latent Dirichlet Allocation asshown in Algorithm 3; definitions of all parameters can be

3

Figure 2: Online Variational LDA Inferential Step

found in the proceeding, sub-table, Table 1. While oper-ating in a similar manner to the ”regular” Latent DirichletAllocation there are several differences. κt is a parameterthat adjusts the rate at which prior iterations are ”forgot-ten”. τ0 varies the weight of the early iterations while κdefines the decay rate of previous λ. λ is the result of theprevious mini-batches. More precisely, it is the variationalparameter on the matrix with word frequency spectrumof all the topics. While Algorithm 3, above, shows thatthe algorithm can process each document individually, itis common practice to collect a mini-batch of documentsand then run the algorithm on all documents in each mini-batch [20] [11]. Furthermore Hoffman et al. show that ifand only if κ is within the range of (0.5, 1] convergence isguaranteed [20].

For the COC datasets, we created 10 different OnlineVariational LDA models by varying the K parameter (i.e.the number of topics) from 1 to 100 and repeating this pro-cess five times. The assessment for optimizing the numberof topics is included in Section 5.0.1.

In our implementation, the Dirichlet parameters are setto be symmetrical for the smoothing of words within top-ics, η, η = 1/V and topics within the set of documents,K, α = 1/K. By keepingα < 1, the modes of the Dirich-let distribution are close to the corners, thus favoring just afew topics for every document and leaving the larger partof topic proportions very close to zero. The LDA modelsare created using the Python library Gensim [36].

Gensim uses variational inference from the onlinemodel in order to approximate the posterior [19]. Theconvergence iteration parameter for the expectation step(i.e. the E-step) is set to 1000; the part where per doc-ument parameters are fit for the variational distributionsare detailed in Algorithm 2 of the original implementa-tion [19].

Figure 3: Empirical evaluation of parameters

5 EvaluationAn experiment was set up to find the parameters thatyielded the best measurements on a run with 9,278 com-plaints. The measurements were made with perplexity -which Blei et al. and Hoffman et al. also used [12] [20].

The parameters used were found by, first, empiricallytesting different values for each parameter. This led to theset of parameter ranges found in Figure 3. A validationstep permuted all combinations of these these parameterson the same 9,278 complaints. The algorithm assessedperplexity after each mini-batch together with the aver-age perplexity and the standard deviation of the perplex-ity with a decaying window of 50 mini-batches. The bestparameters were found in the permutation that yielded thelowest sum of the average perplexity and standard devia-tion of the perplexity.

5.0.1 Empirically Choosing the Optimal Number ofTopics

One of the parameters in LDA is the number of topics K.Optimizing K can be accomplished by measuring the in-formation gain provided in each topic compared to a base-line measures [8]. We use log likelihood, because thisempirical measure evaluates how well the corpora fits themodel. In this case, this is the topic space model pro-duced by LDA. When performing parameter estimation, acommon strategy is to maximize the log likelihood [18].We employ this technique to measure the effectiveness ofeach LDA model, varying the number of topics K. Theresults are presented on Figure 4. Notice that the log like-lihood is maximal after approximately 10 topics. Hencewe chose ten topics as the optimal parameter in our sub-sequent experiments for classifying complaints.

5.0.2 Evaluating Topic Models

Because topic modeling is an unsupervised task the im-plementer is not aware of what the topics will be, aftertraining. This makes evaluation of the topic model a verydifficult problem. This dilemma is not apparent in super-vised learning (for example, logistic regression, classifica-tion), in which one compares predicted labels to expectedlabels. There are no ”expected labels” in our the pipelinefor topic modeling.

4

Figure 4: Empirical evaluation of parameters

Moreover, each topic modeling method - for example,LDA and LSA, possesses a unique method for measur-ing the ”internal quality” (perplexity and reconstructionerror, for example)[11]. However, what is often not citedin evaluation methods, is that these internal measures arean artifact of the particular approach taken when applyingthe model. These include, for example, Bayesian methodsand matrix factorization. In addition, there is no evidence-based approach to compare such scores across differenttypes of topic models. The most feasible method to assessthe quality of unsupervised models is to evaluate how eachmodel can improve the superordinate task of the modeltask for which the implementer is training each collectionof models for.

For example, when our goal is to retrieve semanticallysimilar documents, one can manually tag a set of simi-lar documents and then assess how well a given semanticmodel maps those similar documents together.

Such manual tagging can be resource intensive. Wal-lach et al. suggest a ”word intrusion” method that workswell for models where the topics are meant to be ”humaninterpretable”, such as LDA [13]. For each trained topic,they obtain that topic’s first ten words, then substitute oneof those words with another, randomly chosen word (theintruder) and see whether a human can reliably tell whichone it was.

If so, the trained topic is topically coherent; alterna-tively, if the topic does not have a discernible theme, themodel is considered to be poorly fitted.

We performed this evaluation as follows:

• Select the 50 words for each of of the 10 LDA topics

• Collect all 50 words from all 10 topics, as one set

• For each of the 10 topics we replace a word at a dif-ferent index

• We then split each document (complaint) into twoparts, and verify that first, topics of the first half of

the document are similar to topics of the second halfof the document; and second, each of half of eachrespective complaint is mostly dissimilar with othercomplaints.

In order to benchmark our results, we applied theaforementioned result on the same set of documentsas were trained using LDA, using LSI.

We then tested our evaluation method on 1000 doc-uments that were not used in either LDA or LSItraining. Our results are presented on Table 1 andon Table 2. We used the cosine similarity to mea-sure first, the similarity between corresponding doc-uments (first row) and second, a randomized selec-tion of 10,000 halves (second row). A higher co-sine similarity between corresponding documents isconsidered to be an attribute of a better-fitted model.Conversely, a lower cosine similarity score betweenrandomized documents are attributes of a better-fitted topic model [13].

Table 1: LDA ResultsModel Average cosine similarityCorresponding parts 0.776225069646Randomization 0.254734527925

Table 2: LSI ResultsModel Average cosine similarityCorresponding parts 0.606533434328Randomization 0.0748434974254

We should note, at this juncture, that this does not com-pletely address the security threat to the internal validityof our model. We understand that within a more volatileenvironment, there is no guarantee as to what the inter-topic distance will be, nor will we be able to controlthe the quality of data that is ingested by our proposedpipeline.

5.0.3 Evaluating Coherence

In order to further benchmark our online model to othertopic modeling implementations, we compared the fol-lowing topic models coherence measures [28]. The fol-lowing models were compared:

• LSI (Latent Semantic Indexing)

• HDP (Hierarchical Dirichlet Process)

• LDA (Latent Dirichlet Allocation)

5

Figure 5: Coherency Analysis for Justifying choice for theuse of LDA over other model variants

• LDA (optimized to find optimal number of topics -(LDA Mod))

• Projecting LDA into lower dimensional space (LDALSI)

Each model was parameterized to 10 topics. We usedthe CV measurement, due to the fact that CV hasbeen shown to resemble most closely human mea-surements and ratings. The details of the CV mea-surement are described in detail by Both et al. [28]. Itshould be noted that a higher coherence value, closerto 1, is associated to more human - interpretable top-ics. Our results indicated that LDA, and its vari-ants, scored above 0.4 versus LSI (0.28). This isagreement with our previous evaluation evaluatingthe similarity exhibited between corresponding doc-ument halves for LDA.

6 Analysis of Phone Numbers

After applying the model described in Section 4.0.4, wenow have a set of topics, Z, that are labeled with a relevantcampaign theme, and we aim to do two things:

1. Decide what source numbers from the online com-plaints should be be considered to be labeled asscam.

2. Leverage the topic assignments with their corre-sponding complaint (and hence the phone number(s)extracted from the complaint) to group together com-plaints and related source phone numbers that likelybelong to the same spam campaign.

To this end, one can tune the LDA model to performassignments to estimate two things:

Figure 6: Top 10 thematic topics for 9728 complaints

3. The topic mixtures of each document (by countingthe proportion of words assigned to each topic withinthat document).

4. The words associated to each topic (by counting theproportion of words assigned to each topic overall).

From the aforementioned heuristic, a distribution oftopics, for each document, is constructed. From this dis-tribution, we select the topic with the highest probabilityfor a given complaint. The calls with similar textual con-text can be clustered together to discover spam campaigns(hence, the phone numbers related to the campaigns). Fig-ure 6. illustrates a generated set of topics after conver-gence of the model. After associating each topic witheach complaint we achieved the following discrete statis-tics (displayed on Table 3).

Table 3: Statistics (Post-LDA)Variable CountComplaints 9278Unique Phone Numbers 1851Phone numbers with > 1 scam association 123

In particular, we observed that only 6 percent of thephone numbers were reused for another scam campaign(Table 3). This observation supports earlier hypothesesthat the phone numbers used by scammers in their cam-paigns are less diverse [15]. In particular purchasing newphone numbers is more expensive than the purchase ofother mediums of deploying scam campaigns, such asemail.

6.0.1 Scam Campaign Distributions

We clustered phone numbers by shared campaigns. Fig-ure 7 displays the varying distributions across all scamcampaign themes generated by the online LDA model.

In the event that a phone number was associated withmore than two scam campaigns we selected the scam cam-paign topic that was most often associated with the phonenumber. 25.6 percent of the phone numbers were asso-ciated with technical support campaigns. 18 percent ofnumbers were associated with gift card scams. It shouldbe noted that the content of these complaints for thisgroup were associated with Nokia and Best Buy Gift Card

6

Figure 7: Top 10 thematic topics for 9728 complaints

Figure 8: Distribution of Source Phone Numbers byCountry

scams. All scam campaigns associated with Nokia werecited by victims in the United Kingdom. Surprisingly, wediscovered that many themes were associated with whatwe define as meta comments: comments discussing legalpolicy and discussion regarding spoofing and challengesassociated with discovering scammers (spoofing numbers,for example).

6.0.2 Geographic Distribution of Source PhoneNumbers

We observed that over 95 percent of source phone num-bers that were extracted were sourced to the United States,Canada and the United Kingdom (Figure 8.). This char-acterization, noted as sampling bias, is elaborated on insection 7.

Within the United States (Figure 9.), our analysis dis-covered that the normalized distribution of complaintsthat cited malicious phone numbers were primarily repre-sented by North Carolina, Oregon, Nevada, New Mexico,

Figure 9: Distribution of Source Phone Numbers withinthe United States

Illinois and the District of Columbia. We discuss furtherimplications for further research in section 7.

6.0.3 Service Providers

We further characterized numbers by their wireless ser-vice provider. An interesting observation is that certainoperators are used more often than others to register scamnumbers. Figure 10 shows the distribution of phone num-bers used by scammers among the providers. We ob-serve that, in our dataset, the top 4 operators (out of32) provide more than 65 percent of fraud-related num-bers. These four cell providers, in particular, representone of the largest telecom service providers in Africa(MTN), and the Caribbean and Asia-Pacific (Digicel), re-spectively. After further investigation we discovered thatthese service providers offer one or more of the followingservices:

• Have an online registration service

• Offer low-cost or free international call forwarding

• Provide bundled packages for wireless services,along with subsidized cash-back call services

Although we did not confirm this, independently, wehypothesize, based on prior evidence, that communitiesof scammers find these service providers appealing [10].

7 Discussion and Limitations ofFindings

Our analysis alone is not sufficient to draw complete con-clusions. For instance, we are still uncertain how preva-lent phone number re-usability is. Given that 6 percent ofphone numbers are used across different scam campaignsbegs the following question:

7

Figure 10: Wireless Service Providers

1. Does this observation imply that the remaining num-bers are discarded or does this observation imply thatthey are simply not reported by users on that partic-ular forum?

This question lends itself to further scrutiny into our re-sults. Most saliently, we note that there is inherent bias inour sampling methods. 800notes.com, in particular, repre-sents user complaints from North America and the UnitedKingdom. Therefore, our sample data set does not capturethe full breadth of source phone numbers used by com-munities of scammers. In future work, we would like todiversify our data sources in order to include other publiccomplaint forums.

Moreover, our methods for selecting valid complaintswill also need to be revised. For example, due to thefact that English is predominantly represented in theseforums, we do not account for other languages in ouranalysis. Therefore, countries where scam campaigns areubiquitous, but under-reported are not representative ofour sample. In addition, it is possible that the discardedcomplaints were provided by non-English speakers thatdo not possess English fluency to elaborate or providedetailed accounts of their experiences. Future work willinclude countries that our initial analysis of numbers didnot account for. Namely, we will include COC datasetsfrom China, India, the United Kingdom, and from CentralAmerica.

We also understand that during training, only a subsetof source phone numbers in the COC datasets can be at-tributed to distinguishable campaigns. However it is aworthwhile experiment to see how effectively the COC-only model that has been trained can accurately describea test sample.

Therefore, a more thorough evaluation of our model,which we will leave for future work, is an evaluation ex-periment to further assess the effectiveness of our models’

label of the scam type, by creating a blacklist. We willthen assess the effectiveness of that blacklist based on theblacklists’ ability to block future unwanted calls. We will,subsequently, compare our blacklists against two com-mercial third-party carriers. We then will assess how wellthe blacklists formed from different data sources, namelyspam and COC, can complement each other. Finally, wewill describe how the blacklists we construct can blockfuture spam messages and phone calls. Through this, wewill demonstrate how effective the blacklists will be inblocking these campaigns.

To estimate the overlap between our blacklists and thirdparty blacklists, we will select a random sample of 10,000source phone numbers from all of our datasets, and per-form reverse phone lookup queries. This will be per-formed using Twilio and YouMail service providers. Thegoal will be to verify that our numbers, each labeled witha topical theme, are described similarly by the third-partyservice provider. We will provide specific proportions anddetermine to see if our blacklists can be leveraged as aproxy to estimate how effective third-party phone black-lists are in defending users from malicious calls. We willalso use that same random sample of numbers in order tovalidate false positives.

8 Related Work

8.1 Stability of Phone numbersTelephony abuse has emerged as a nefarious mechanismfor deploying a multitude of fraudulent and malicious pri-vacy violations [30], [29].

The exploration of such cross-channel abuse and spe-cific scams, has included the analysis of global operatorsand the origins of mobile operators. For example, Costinet. al. [15] investigated how malicious actors from vari-ous geographic regions, particularly Nigeria, deploy var-ious scams, based on analysis of crowd sourced web list-ings. In this investigation, the researchers also provided adetailed analysis of the economic implications of the busi-ness models of such scams, along with the information ex-changed between actors across telephony channels. Fur-thermore, their analysis provided detailed insights into theonline and phone infrastructure used by malicious actors.Finally, Costin et. al. also examined tactics, leveraged byscammers, that actively conducted live analysis of phonenumbers through calls. However, these previous resultsfocus on demonstrating that phone numbers are a stableindicator of the identification of scam campaigns. More-over, a more detailed analysis, mostly considered phonenumbers related to outgoing calls, whereby the user con-tacted a given tech-support number [25].

In contrast to their prior work, our scope of our analysisinclude a broader range of campaigns. Furthermore, our

8

analysis extends to incorporate crowd-sourced data sets.Finally, our study instead focuses on incoming call reportsand their subsequent responses from communities. Fromour longitudinal analysis we are able to capture and todescribe the authenticity and associated scam campaignof the phone number.

8.2 Spam and Email Classification

Forensic analysis of spam messages has been a very thor-oughly interrogated area of research [23]. In this studyMa et al. overcame the sparsity problem in SMS messageclassification by, first, using K-means to group messagesinto disparate classes and then, second, aggregate all thespam messages of a class into a single document. Symbolsemantics was accounted for this in this model. The au-thors designed specific rules and introduced specific back-ground terms in order to make the model appropriate tofully represent SMS spam. Wei et al. [34] propose an ap-proach based on the agglomerative hierarchical clusteringalgorithm and the connected components with weightededges model to cluster spam emails. Only spam emailsused for advertising are tested by the authors.

In contrast, our motivation is to describe the context ofa phone-number, and it’s potential reuse across differentspam campaigns. Our target documents are context-richforums that provide nuanced details that are enriched withpolysemy and ambiguity. Finally, the documents that weanalyze, and our model evaluation account for nuances ina victim’s regional differences. For example, the reportingof Nokia scam campaigns are reported differently than aCanadian lottery scam. Our procedure for describing thecomplaint, and for the evaluation of these complaints, ac-count for these differences.

8.3 Classifying Using (Call Detail Records)CDR Lookup

Phoneypot, [17], is a phone honeypot that is comprised ofa set of phone numbers - along with infrastructure to col-lect CDRs when honeypot numbers are called by externalparties. Phoneypot has demonstrated the utility of honeypots in telephony research.

Although the data collected from the honeypot was ana-lyzed to source various types of calls (attributes of the calltype), the longitudinal analysis of the phone number wasnot studied. Moreover, the features mentioned in Phoney-pot did not distinguish the specificity of the geographicalorigins of the phone number. In contrast, our analysis in-cludes geographical attributes in order to identify the ge-ography of the call, along with the service provider.

8.4 VoIP Call Detail Records

Similarly, Chiapetta et al. [14] present an analysis of alarge dataset of VoIP CDRs. Subsequently, their inves-tigation afforded insights into different call patterns andgroups callers using unsupervised techniques. The fea-tures mentioned in this work are appealing for population-level uses but are not designed to differentiate betweenspam numbers and legitimate callers. A similar heuristicfor clustering of transcripts, recorded by a phone honey-pot, have also provided similar findings in identifying and,subsequently blocking actors [29], [26], [16].

However, one caveat in this previous approach lies inthe fact that since the transcript is the only source of in-formation, it can only serve as the single indicator forwhether or not to block calls from the campaigns that areseen at the source honeypot. This cannot be generalized toother domains in the telephone network. Applying a gen-erative and probabilistic approach based on feature sets,as in our study, allows for greater generalization beyonda specific dataset. Moreover, our analysis provides morehuman-interpretable output features for both the identityof and the details of a given call record, as documentedby a given victim. Finally, our model continuously un-dergoes training in order to be flexibly deployed to novelcontexts in the phone network.

8.5 Domain and IP Address Analyses

Other reputation systems have been investigated for do-main names, IP addresses and other online resourcessuch as URLs. These systems have offered robust de-fenses to email spam and malware infections. However,these investigations typically utilize network, and otherapplication-specific features to label such resources whichdiffer significantly from information available in user-facing devices.

Finally, caller profiling has been investigated by sev-eral researchers [31]. However, these previous studiesare predicated on access to large volumes of private calldata records. These often possess accompanying privacyconcerns from service providers that do not readily makesuch datasets available. Yardi et al. [35] characterizedmacro-level behavioral patterns in order to differentiatemalicious users from authentic users. Call duration andsocial network connections were used to separate legiti-mate callers from spam/scam callers in [9] by developinga global reputation based system for callers. The caveat,however, is duration information and social connectionsof phone users are not available. Furthermore, both thefull evaluation for evaluating a human-interpretable ori-gin of a spam campaign’s numbers, and a given number’scross-validation with other ground truth data sets is alsonot addressed by this work. To the best of our knowledge,

9

our work is the first one to systematically explore crowd-sourced datasets using Online-Variational LDA samplingfor modeling topics associated with phone numbers. Fromthis we are able to both describe the origins of and au-thenticity of a phone number. In addition, our methodsprovide insights into the effectiveness of such generativemodels as effective instruments that can be used in classi-fying scam campaigns and phishing attacks.

9 ConclusionRobocalls act as a pivotal instrument for criminal activi-ties. In this paper, we propose an Online Variational LDAmodel, based on the probability theory of Latent Dirich-let Allocation, for describing and for classifying mali-cious scam-campaigns deployed through the telephonenetwork. The online variant can eliminate the sparse rep-resentation issues in document classification. It is moresuitable for the task of describing and for the task of clas-sifying robocall scam campaigns. When compared withthe existing state of the art techniques, namely LSA, weshow that it is more suitable for analysis, and for describ-ing the malicious content of robocall complaints, as re-ported on online forums. We will continue to build uponthis model towards a systems solution that incorporates aclient interface and keyword search capabilities. The de-mocratization of this system will empower both everydayusers, and law enforcement in identifying and preventingscam campaigns that are specific to identity theft, accountfraud, and phishing attacks.

References[1] https://bit.ly/2KlhJyt. [Online; accessed

12-March-2018].

[2] https://www.youmail.com/. [Online; ac-cessed 20-January-2018].

[3] https://www.truecaller.com/. [Online;accessed 19-January-2018].

[4] https://www.projecthoneypot.org/.[Online; accessed 21-March-2018].

[5] https://www.mongodb.com/. [Online; ac-cessed 17-September-2017].

[6] http://www.nltk.org/api/nltk.stem.html. [Online; accessed 21-November-2017].

[7] https://github.com/googlei18n/libphonenumber. [Online; accessed 21-November-2017].

[8] L. Alsumait, D. Barbara, J. Gentle, and C. Domeni-coni. Topic significance ranking of lda generativemodels. In Proceedings of the European Conferenceon Machine Learning and Knowledge Discovery inDatabases: Part I, ECML PKDD ’09, pages 67–82,Berlin, Heidelberg, 2009. Springer-Verlag.

[9] V. Balasubramaniyan, M. Ahamad, and H. Park.Callrank: Combating spit using call duration. In InCEAS (2007), CEAS ’07, 2007.

[10] M. Bidgoli and J. Grossklags. Hello. this is the irscalling.: A case study on scams, extortion, imper-sonation, and phone spoofing. In In Proceedingsof the Symposium on Electronic Crime Research(eCrime), 2017.

[11] D. M. Blei. Probabilistic topic models. Commun.ACM, 55(4):77–84, 2012.

[12] D. M. Blei, A. Y. Ng, and M. I. Jordan. Latentdirichlet allocation. Journal of Machine LearningResearch, 3:993–1022, 2003.

[13] J. Chang, J. L. Boyd-Graber, S. Gerrish, C. Wang,and D. M. Blei. Reading tea leaves: How humansinterpret topic models. In Advances in Neural Infor-mation Processing Systems 22: 23rd Annual Con-ference on Neural Information Processing Systems2009. Proceedings of a meeting held 7-10 December2009, Vancouver, British Columbia, Canada., pages288–296, 2009.

[14] S. Chiappetta, R. P. Claudio Mazzariello, and S. P.Romano. An anomaly-based approach to the analy-sis of the social behavior of voip users. Number 6,pages 1545–1559, 2013.

[15] A. Costin, J. Isacenkova, M. Balduzzi, A. Francil-lon, and D. Balzarotti. ”The role of phone numbersin understanding cyber-crime schemes.”. In Privacy,Security and Trust (PST), 2013 Eleventh Annual In-ternational Conference on, pp. 213-220., 2013.

[16] Y. Gao and G. Zhao. Knowledge-based informa-tion extraction: a case study of recognizing emailsof nigerian frauds. In In Proceedings of the 2005,NLDB conference, pages 161–171, 2005.

[17] P. Gupta, B. Srinivasan, V. Balasubramaniyan, andM. Ahamad. Phoneypot: Data-driven understandingof telephony threats. In In NDSS, 2015.

[18] G. Heinrich. Parameter estimation for text analysis.Web: http://www. arbylon. net/publications/text-est.pdf, 2005.

10

[19] M. Hoffman, F. R. Bach, and D. M. Blei. On-line learning for latent dirichlet allocation. In J. D.Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S.Zemel, and A. Culotta, editors, Advances in NeuralInformation Processing Systems 23, pages 856–864.Curran Associates, Inc., 2010.

[20] T. Hofmann. Probabilistic latent semantic indexing.In Proceedings of the 22Nd Annual InternationalACM SIGIR Conference on Research and Develop-ment in Information Retrieval, SIGIR ’99, pages 50–57, New York, NY, USA, 1999. ACM.

[21] T. K. Landauer, P. W. Foltz, and D. Laham. An intro-duction to latent semantic analysis. Discourse Pro-cesses, 25(2-3):259–284, 1998.

[22] J. B. Lovins. Development of a stemming algo-rithm. Mechanical Translation and ComputationalLinguistics, 11:22–31, 1968.

[23] J. Ma, Y. Zhang, J. Liu, K. Yu, and X. Wang. Intel-ligent sms spam filtering using topic model. In 2016International Conference on Intelligent Networkingand Collaborative Systems (INCoS), pages 380–383,Sept 2016.

[24] D. M. Mimno, H. M. Wallach, E. M. Talley,M. Leenders, and A. D. McCallum. Optimizing se-mantic coherence in topic models. In EMNLP, 2011.

[25] N. Miramirkhani, O. Starov, and N. Nikiforakis.Dial One for Scam: A Large-Scale Analysis of Tech-nical Support Scams. In Proceedings of the 24th Net-work and Distributed System Security Symposium(NDSS), 2017.

[26] I. Murynets and R. P. Jover. Crime scene investiga-tion: Sms spam data analysis. In In Proceedings ofthe 2012 ACM conference on Internet measurementconference, pages 441–442, 2012.

[27] S. Pandit, R. Perdisci, M. Ahamad, and P. Gupta.Towards measuring the effectiveness of telephonyblacklists. In In NDSS, 2018.

[28] M. Roder, A. Both, and A. Hinneburg. Exploring thespace of topic coherence measures. In Proceedingsof the Eighth ACM International Conference on WebSearch and Data Mining, WSDM ’15, pages 399–408, New York, NY, USA, 2015. ACM.

[29] M. Sahin and A. Francillon. Over-the-top bypass:Study of a recent telephony fraud. In Proceedings ofthe 23rd ACM conference on Computer and commu-nications security (CCS), CCS ’16. ACM, October2016.

[30] M. Sahin, A. Francillon, P. Gupta, and M. Ahamad.Sok: Fraud in telephony networks. In Proceedingsof the 2nd IEEE European Symposium on Securityand Privacy, April 2017.

[31] V. S. Tseng, J.-C. Ying, C.-W. Huang, Y. Kao, andK.-T. Chen. Fraudetector: A graph-mining-basedframework for fraudulent phone call detection. InIn Proceedings of the 21th ACM SIGKDD Inter-national Conference on Knowledge Discovery andData Mining, 2015.

[32] P. D. Turney and P. Pantel. From frequency to mean-ing: Vector space models of semantics. J. Artif. Int.Res., 37(1):141–188, Jan. 2010.

[33] K. Wedenberg and A. Sjoberg. Online inference oftopics : Implementation of the topic model latentdirichlet allocation using an online variational bayesinference algorithm to sort news articles, 2014.

[34] C. Wei, A. Sprague, G. Warner, and A. Skjel-lum. Mining spam email to identify common ori-gins for forensic application. In Proceedings of the2008 ACM Symposium on Applied Computing, SAC’08, pages 1433–1437, New York, NY, USA, 2008.ACM.

[35] S. Yardi, D. Romero, and G. Schoenebeck. Detect-ing spam in a twitter network. Number 1, 2009.

[36] R. ehek and P. Sojka. Software framework fortopic modelling with large corpora. In Proceedingsof LREC 2010 workshop New Challenges for NLPFrameworks, pages 46–50, Valletta, Malta, 2010.University of Malta.

11