Representing Documents via Latent Keyphrase Inference

Jialu Liu† Xiang Ren† Jingbo Shang†

Taylor Cassidy‡ Clare R. Voss‡ Jiawei Han††Department of Computer Science, University of Illinois at Urbana-Champaign‡Computational & Information Sciences Directorate, Army Research Laboratory

†jliu64, xren7, shang7, hanj@illinois.edu ‡taylor.cassidy, clare.r.voss.civ@mail.mil

ABSTRACTMany text mining approaches adopt bag-of-words or n-gramsmodels to represent documents. Looking beyond just thewords, i.e., the explicit surface forms, in a document canimprove a computer’s understanding of text. Being awareof this, researchers have proposed concept-based models thatrely on a human-curated knowledge base to incorporate otherrelated concepts in the document representation. But thesemethods are not desirable when applied to vertical domains(e.g., literature, enterprise, etc.) due to low coverage ofin-domain concepts in the general knowledge base and in-terference from out-of-domain concepts. In this paper, wepropose a data-driven model named Latent Keyphrase In-ference (LAKI ) that represents documents with a vector ofclosely related domain keyphrases instead of single wordsor existing concepts in the knowledge base. We show thatgiven a corpus of in-domain documents, topical content unitscan be learned for each domain keyphrase, which enables acomputer to do smart inference to discover latent documentkeyphrases, going beyond just explicit mentions. Comparedwith the state-of-art document representation approaches,LAKI fills the gap between bag-of-words and concept-basedmodels by using domain keyphrases as the basic representa-tion unit. It removes dependency on a knowledge base whileproviding, with keyphrases, readily interpretable representa-tions. When evaluated against 8 other methods on two textmining tasks over two corpora, LAKI outperformed all.

1. INTRODUCTIONText data (e.g., web queries, business reviews, product

manuals) are ubiquitous and tasks like document groupingand categorization play an essential role in big data appli-cations. At the heart of analysing text data is a unifiedrepresentation of text input.

Related Work: The most common representation for textsis the bag-of-words [1] due to its simplicity and efficiency.This method however typically fails to capture word-levelsynonymy (missing shared concepts in distinct words, such

Copyright is held by the International World Wide Web Conference Com-mittee (IW3C2). IW3C2 reserves the right to provide a hyperlink to theauthor’s site if the Material is used in electronic media.WWW 2016, April 11–15, 2016, Montréal, Québec, Canada.ACM 978-1-4503-4143-1/16/04.Include the http://dx.doi.org/10.1145/2872427.2883088 .

Categories RepresentationWords dbscan, method, clustering, process, ...Topics [k-means, clustering, clusters, dbscan, ...]

[clusters, density, dbscan, clustering, ...][machine, learning, knowledge, mining, ...]

KB Concepts data mining, clustering analysis, dbscan, ...Keyphrases dbscan: [dbscan, density, clustering, ...]

clustering: [clustering, clusters, partition, ...]data mining: [data mining, knowledge, ...]

Table 1: Representations for query “DBSCAN is amethod for clustering in process of knowledge discov-ery.” returned by various categories of methods.

as “doctor” and “physician”) and polysemy (missing distinctconcepts in same word, such as “Washington” can be ei-ther the city or the government). As a remedy, topic mod-els [7, 3] try to overcome this limitation by positing a setof latent topics which are distributions over words, and as-suming that each document can be described as a mixtureof these topics. Nevertheless, the interpretability of latentspace for topic models is not straightforward and perus-ing semantic meaning in inferred topics is difficult [4, 18].Concept-based models [10, 23, 21, 12, 14] were proposed toovercome these barriers. The intuition is to map a text queryinto a high-dimensional vectorial representation where eachdimension corresponds to a concept in a general KnowledgeBase (KB), like Wikipedia or Freebase, making them eas-ily interpretable. For example, the text sequence “DBSCANfor knowledge discovery”can be mapped to KB concepts like“KB: data mining”, “KB: density-based clustering”and“KB:dbscan”. Such methods take advantage of a vast amount ofhighly organized human knowledge. However, most of theexisting knowledge bases are manually maintained, and arelimited in coverage and freshness. Researchers have there-fore recently developed systems such as Probase [25] andDBpedia [2] to replace or enrich traditional KBs. Neverth-less, the rapid emergence of large, domain-specific text cor-pora (e.g., business reviews) poses significant challenges totraditional concept-based techniques and calls for methodsof representing texts by interpretable units without require-ment of a general KB.

In this paper, we study the problem of learning representa-tions for domain-specific texts: Given massive training textsin a specific domain or genre, we aim to learn a systematicway to derive interpretable representations for any new in-domain documents without relying on a KB. When existingapproaches are directly applied to this problem setting, theyencounter several limitations:

• Representation interpretability: A lot of data-drivenmethods (e.g., bag-of-words and topic models) lack straight-

1057

dbscan

kernel k-means dbscan data mining

0 01 0 2 1

knowledgediscovery

data

mining dbscankernel

k-means

clustering kernelk-means

…

……

dbscan

0.650.6

1 11

…

kdd

…

data

…

0.4 0.3

knowledgediscovery

data

mining dbscankernel

k-means

clustering kernelk-means

…

……

dbscan

0.650.6

1 11

…

kdd

…

data

…

0.4 0.3

Document Representation

DBSCAN / is / a /

method / for / clustering /

in / process / of / knowledge discovery.

DBSCAN / was /

proposed / by …

data mining

text mining

clustering

kernel k-means

dbscan

…

kernel kmeans 1

kernel k means 1

clustering 0.65

kernel 0.55

rbf kernel 0.5

dbscan 1

density 0.8

clustering 0.6

dense regions 0.3

shape 0.25

data mining 1

knowl. discov. 1

kdd 0.67

clustering 0.6

text mining 0.6

data

mining

0.6 0.8 0 0.8 0 0.7 0.9 ……

knowledge

discovery

density-based

clustering

Offline!

Online:

clustering

Domain Keyphrase Extraction

Segmentation

Domain Keyphrase Silhouetting

Document Keyphrase Inference

Figure 1: Overview of LAKI. White and grey nodes represent domain keyphrases and content units respectively.

forward interpretation for document representation, whichis critical for model verification and for ensuring that themodel is capturing user’s intuitions about the text input [4].

• Domain restriction: For concept-based methods relyingon a KB [10, 21, 23], the provided knowledge in KB usuallysuffers from limited coverage and freshness on specific, dy-namic or emerging domains.

• Cross-domain interference: Due to the wide range ofdomains covered in a general KB, many words will havemultiple possible KB referents even if they are unambiguousin the target domain, thus introducing noise and distortionin the text representation even when the vocabulary overlapbetween the target domain and knowledge base is small [12].

To address the above challenges, we exploit several intu-itive ideas as follows. We first extract domain keyphrasesfrom an in-domain corpus, which are both meaningful andinterpretable, and instantiate them as dimensions in ourtext representation. Thus, a document is represented asa vector of keyphrases, in contrast to traditional methodsusing words, topics or concepts (see Table 1). Note thatwithin a particular domain, a given keyphrase will likelyhave one meaning [11], making the representation relativelyunambiguous. On one hand, limiting phrase extraction tothe in-domain corpus helps eliminate cross-domain interfer-ence. However, many keyphrases relevant to a given doc-ument are not explicit document keyphrases, i.e., they arenot mentioned in the document. This presents a uniquechallenge, especially for short texts like paper abstracts andbusiness reviews. We therefore propose to associate each do-main keyphrase with a silhouette—a topically-cohesive set ofwords and phrases, which enables a computer to incorporatelatent document keyphrases into text representation.

Our solution, called Latent Keyphrase Inference (LAKI),is developed to systematically integrate the above ideas. Asshown in Fig. 1, it consists of an offline domain keyphraselearning phase and an online document keyphrase inferencephase. In the offline phase, corpus-wide domain keyphrasesare extracted and their silhouettes are learned through a hi-erarchical Bayesian network which optimizes the likelihoodwith respect to latent document keyphrases, given observedcontent units in the training corpus. The Bayesian networkis essentially a directed acyclic graph (DAG) for modelingthe dependency 1) between domain keyphrases and contentunits, and 2) between domain keyphrases themselves. In theonline process, LAKI identifies the top-ranked latent docu-ment keyphrases for the input text by statistical inference

on the Bayesian network. Consequently, LAKI is able totransform text string into a high-dimensional vector repre-sentation. Entries in the output vector quantify the related-ness between the document and the respective keyphrases.The major contributions of this paper are:

1. We propose to use latent keyphrases as document repre-sentation for domain-specific texts, which enhances rep-resentation interpretability, solves domain restriction andavoids cross-domain interference.

2. We develop a Bayesian network-based approach to modeldomain keyphrase silhouettes, which later helps infer la-tent document keyphrases and solves the rarity of explicitkeyphrase mentions in the document.

3. Experiments on corpora of different domains show boththe effectiveness and efficiency of the proposed solution.

2. BACKGROUNDThis paper deals with the problem of learning representa-

tion of domain-specific texts. Given a document corpus asinput, we aim to produce a set of keyphrases as the basisfor vectorial representation of any text queries posed to thiscorpus. The task is formally defined as follows.

Definition 1 (Problem Definition). Given a documentcorpus D with specific focus on certain genres of content, weaim to automatically extract semantically keyphrases K =K1, · · · ,KM fromD, and derive a model that can generate

high-dimensional vectorial representation [P(q)1 , · · · , P (q)

M ] forany text query q from the same domain, where each entryof the vector quantifies relatedness between query and thecorresponding keyphrase.

Ideally, these we would use domain-specific knowledge baseconcepts instead of keyphrases. But identifying and con-structing these concepts purely from text data with satis-factory performance remains an unsolved problem. Instead,we instantiate them as domain keyphrases since the prob-lem of extracting high-quality keyphrases is more tractable,and there exist many publicly available methods. A domainkeyphrase is a phrase (which may be one or more words) ofgreat significance in the domain extracted from a domain-specific corpus. They are meant to be natural, meaning-ful and likely unambiguous semantic units for representingdomain-specific texts. However, one needs to overcome theirlow rate of explicit mentions in documents to which theyare relevant. Our solution is to identify latent documentkeyphrases which are relevant to a given document, but are

1058

not explicitly mentioned therein. A document keyphraseis a domain keyphrase that is relevant to a specific docu-ment, i.e.it serves as an informative word or phrase to indi-cate the content of that specific document. Actual mentionof a document keyphrase is not necessarily required.

Following the above discussion, a good model should gen-erate representations in terms of document keyphrases, to-gether with numerical values indicating strength of related-ness. These representations could be used in various textmining tasks to attain outstanding performance in terms ofapplication-specific evaluation metrics.

Our propsed text representation method is called LatentKeyphrase Inference (LAKI). Its core technical contributionis domain keyphrase silhouetting—an unsupervised learningprocess to mine domain keyphrase silhouettes.

Definition 2 (Domain Keyphrase Silhouette). Givena domain keyphrase Km, its keyphrase silhouette Sm com-prises a topically-cohesive bag of content units (i.e., wordsand phrases) regardingKm. Let the total set of content unitsin the corpus is denoted as T = T1, · · · , TL. Then Sm isa non-negative vector [Sm1, · · · , SmL] where each entry Sml

refers to a relatedness score between domain keyphrase Km

and content unit Tl.

These silhouettes not only enable the LAKI to identify la-tent document keyphrases through statistical inference, butalso enhance the interpretability of corresponding domainkeyphrases. LAKI can be divided into two phases: (i) theoffline domain keyphrase learning phase, which extracts do-main keyphrases from the in-domain corpus and learns theirsilhouettes respectively, and (ii) the online document/querykeyphrase inference phase, which derives vectorial represen-tation for each query based on the domain keyphrase silhou-ettes, as outlined below.

• Offline Domain Keyphrase Learning:

1. Extract domain keyphrases from a domain-focused doc-ument corpus; and

2. Learn domain keyphrase silhouettes by iteratively opti-mizing a Bayesian network with respect to the unknownvalues, i.e., latent document keyphrases, given observedcontent units in the training corpus.”

• Online Document/Query Keyphrase Inference:1. Segment input query into content units; and2. Do inference for document keyphrases given the observed

content units, generating a high-dimensional vector whereeach entry quantifies relatedness between the input queryand corresponding keyphrase.

Fig. 1 illustrates the above two phases with examples foreach individual step.

3. DOMAIN KEYPHRASE EXTRACTIONAND SEGMENTATION

Domain Keyphrase Extraction refers to automatically dis-covering salient terms from a domain-focused document cor-pus, whereas segmentation is to partition text into continu-ous segments and is more context dependent. Both compo-nents serve as the building blocks for LAKI and examples oftheir functionality are shown on the left side of Fig. 1.

Solving these two tasks is not the objective of this workas there are various works proposed in the literature doingthem well. In general, they can be divided into two cat-egories: data-driven and linguistic-based. The former [24,

K5K1

K4K2

T5 T6T1 T2 T3 T4

K3

DomainKeyphrases

Content Units

Figure 2: An illustrative Bayesian network for domainkeyphrase silhouetting.

8, 17] explore frequent n-grams in document collections andleverage a variety of statistical measures derived from a cor-pus to estimate phrase quality and doing segmentation. Thelatter [9, 27, 15] rely on linguistic features that include part-of-speech (POS) tags or parse trees, and usually require largetraining sets labeled by humans to identify phrase bound-aries, which are very costly to obtain. As the rest of ourframework is fully data-driven, for the sake of consistency,we adopt an algorithm named SegPhrase which was recentlyproposed to combine domain keyphrase extraction with seg-mentation in an iterative fashion [17]. The algorithm is mo-tivated by the observation that domain keyphrase extractionand segmentation are mutually dependent and thus can ben-efit each other. More specifically, domain keyphrase extrac-tion relies on segmentation to locate candidate mentions,which later helps rectify their corpus-level statistics for as-sessing quality. Simultaneously, segmentation has access tostored extracted keyphrases to guide the partitioning of thedocument text into content units.

4. DOMAIN KEYPHRASE SILHOUETTINGIn this section, we present our model to domain keyphrase

silhouettes by optimizing a Bayesian network w.r.t. latentdocument keyphrases, given observed content units includ-ing both words and phrases in the training corpus. Recallthat the silhouette of a domain keyphrase Km consists ofa bag of related content units for capturing the topic ofKm. Besides modeling dependency between keyphrases andcontent units, we consider interactions between keyphrasesthemselves and make the network hierarchical with DAG-like structure shown in Fig. 2. Content units are located atthe bottom layer and domain keyphrases form the rest. Bothtypes of nodes act as binary variables1 and directional linksbetween nodes depict their dependency. Specifically, thelinks connecting domain keyphrases to content units formthe so-called domain keyphrase silhouettes.

Before diving into the technical details, we motivate ourmulti-layered Bayesian network approach to the silhouettingproblem. First, it is better to have our model infer more thanexplicit document keyphrases. For example, even if the textonly contains “html” and “css”, the word “web page” comesto mind. But more than that, a multi-layered network willactivate ancestor keyphrases like“world wide web”even theyare not directly linked to “html” or “css”, which are contentunits in the bottom layer.

Meanwhile, we expect to identify document keyphraseswith different strength scores. Reflected in this Bayesianmodel from a top-down view, when a parent keyphrase isactivated, it is more possible for its children with strongerconnection to get activated.

1For multiple mentions of a content unit, we choose to makeseveral copies of that node together with its links.

1059

Furthermore, this formulation is quite flexible. We allowa content unit to get activated by each connected domainkeyphrase as well as by a random noise factor (not shownin Fig. 2), which behaves like a Noisy-OR, i.e., a logical ORgate with some probability of having “noisy” output. Thisincreases robustness of the model especially when trainingdocuments are noisy.

We define the conditional distribution in our Bayesian net-work in line with the above motivations. Mathematically,we use K = K1,K2, . . . ,KM and T = T1, T2, . . . , TLto denote domain keyphrases and content units respectively.For notational convenience, we use a unified symbol Z todenote K and T such that K = Z1, . . . , ZM and T =ZM+1, . . . , ZM+L. A child node Zj is Noisy-OR [13] withits parent nodes Pa(Zj) = Pa1

j , Pa2j , . . . as:

p(Zj = 1∣∣Pa(Zj)) = 1− exp

(−W0j −

∑i

Wij1Paij

)(1)

where W denotes link weight and 1 is an indicator functionreturning 1 if its associated node state is true. In this way,larger weight of a link will make its child node more likelyto be activated. Note that the leak term W0j allows forthe possibility of a node to be true even if all parents arefalse. Intuitively, it can be explained as a prior for a domainkeyphrase node and a noise for a content unit. Meanwhile,leak terms W0∗ for all nodes can be naturally transformedto link weights by positing a latent factor Z0 with p(Z0 =1) = 1, where notationally convenient. An example of sucha notation is shown on the left side of Fig. 3 for a tiny family.

In the following subsections, we first discuss how to learnlink weights given the Bayesian network structure and thendiscuss how the initialization is done to decide this structureand to set initial link weights.

4.1 Model LearningTo effectively learn link weights during domain keyphrase

silhouetting, Maximum Likelihood Estimation is adopted.The intuition is to estimate parameters by maximizing thelikelihood of observed content units together with partially-observed document keyphrases2. Suppose we have N docu-ments in the corpus where each document d uses a binaryvector t(d) to represent the states of content units (i.e., ob-served or not), the log-likelihood of the corpus is:

L(D) =

N∑d=1

log∑

k∈Ω(d)

p(K = k, T = t(d)) (2)

where Ω(d) is the space of all possible combinations of doc-ument keyphrase states. This space changes for differentdocuments and will be discussed later in this subsection.

It is difficult to directly optimize Eq. 2 due to the la-tent states for the rest keyphrases. Instead we resort to theExpectation-Maximization (EM) algorithm which guaran-tees to give a local optimum solution. The EM algorithmstarts with some initial guess at the maximum likelihood pa-rameters and then proceeds to iteratively generate successiveestimates by repeatedly applying the E-step (Expectation-step) and M-step (Maximization-step). For general Bayesiannetworks, normally p(Zj , Pa(Zj)|T = t) must be computedfor all state combinations between Zj and Pa(Zj) in theE-step. In our case due to the presence of Noisy-OR as in

2Explicit document keyphrases can be identified by applyingexisting keyphrase extraction methods like [24].

w03 w13 w23

w13w03 w23

Z3

Z0 Z1 Z2 Z0 Z1 Z2

Z3

X03 X13 X23

w01

w02X02

X01

w01

w02

Figure 3: An alternative representation of Noisy-OrBayesian network. We assume parents Pa(Z3) of Z3 arePa1

3 = Z1 and Pa23 = Z2 respectively.

Eq. 1, we can dramatically reduce the complexity by divid-ing the whole family into parent-child pairs and computingeach pair separately. To demonstrate this, we show an alter-native representation of the left Noisy-OR network in Fig. 3by adding grey nodes for each link. A grey node Xij istrue with probability 1−exp (−Wij) only if the parent nodePaij is true. The original child white node on the left nowbecomes deterministic-OR of parent grey nodes. Based onthis representation, one can still derive the same probabilitydistribution as Eq. 1.

Expectation Step: Since the child white node always per-forms OR operation on its parents, we only need to focuson the grey nodes with single parent. This reduces a lot ofstorage consumption and transforms our task to computing

R(d)ij = P (Xij = 1, Paij = 1|T = t(d),Ω(d)) together with

P(d)m = P (Km = 1|T = t(d),Ω(d)) where:

• R(d)ij refers to the probability of a grey node Xij to be

activated as well as that both of its parent Paij and childnode Zj are present; and

• P (d)m refers to the probability of a domain keyphrase nodeKm to be activated, i.e., becoming a document keyphrase.

Unfortunately, to compute these two exactly, one still needsto enumerate all state combinations of domain keyphrases.This is NP-hard for a Bayesian network like ours [5]. We

therefore constrain the search space Ω(d) such that non-related keyphrases are directly excluded before applying EM.That is, we only allow keyphrase ancestors of observed con-tent units to change states during the inference. We areessentially forcing certain elements of K to be fixed in theirstates during the enumeration of Ω(d). The correspondinganalytical expressions are derived following Bayes rules:

R(d)ij =

∑c∈Ω(d) p

(Z = k, t(d)

) P (Xij=1|Paij)

p(Zj=zj |Pa(Zj))1zj 1Pai

j∑z∈Ω(d) p

(Z = k, t(d)

)P

(d)m =

∑z∈Ω(d) p

(Z = k, t(d)

)1km∑

z∈Ω(d) p(Z = k, t(d)

)For longer text, there will still be quite many keyphrases

left for inference following the above strategy, making in-ference intractable at a large scale. We therefore resortto approximate inference by applying a stochastic samplingtechnique for generating samples from the joint probabilitydistribution over Z. This type of approximation technique isalso used for online inference, introduced in the next section.

Maximization Step: Based on the sufficient statistics col-lected by the Expectation step, one can update each linkweight Wij between node Paij and Xij with the followingclosed-form solutions:

Wij = − log(1−

∑d R

(d)ij∑

d P(d)

Paij

), W0j = − log

(1−

∑d R

(d)0j

|N |

)

1060

Expectation and maximization steps are iterated until themodel changes minimally.

4.2 Model InitializationLike other EM frameworks, the parameter estimation will

suffer from the problem of local maximum, making the re-sults vary with different network structures and initializedlink weights. Therefore, to obtain a good initialization be-fore model training is important for our task.

Specifically, there are two sub-problems for the model ini-tialization:

1. How to decide the topological order among domainkeyphrase nodes and to build links among them?

2. How to build links between domain keyphrase and con-tent units?

For the former, a reasonable topological order of DAGshould be similar to that of a domain ontology. The linksamong domain keyphrase nodes should reflect IS-A relation-ships [26]. Ideally, documents and queries which are de-scribing specific topics will first imply some deep keyphrasenodes being activated. Then the ontology-like topologicalorder ensures these content units have the chance of beingjointly activated by general keyphrases via inter-keyphraselinks. Many techniques [26, 20, 6] have been previously de-veloped to induce an ontological structure over keyphrases.It is out of scope of our work to specifically address theseor evaluate their relative impact in our evaluation. Weinstead use a simple data-driven approach, where domainkeyphrases are sorted based on their counts in the corpus,assuming keyphrase generality is positively correlated withsits number of mentions [20]. Thus, keyphrases mentionedmore often are higher up in the graph. Links are addedfrom keyphrase Ki to Kj if Ki has more counts and theyare closely related and frequently co-occured:

p(Ki|Kj) ≥ α, sim(Ki,Kj) ≥ β

where α is a threshold reflecting the confidence about the IS-A relationship and β requires two domain keyphrases to berelated. And sim(Ki,Kj) is computed based on the cosinesimilarity between word2vec embeddings of keyphrases Ki

and Kj . In our work, we empirically set α and β to be 0.5and 0.3 respectively. Note that the latter score is also usedto detect equivalence between keyphrases (i.e., acronyms orinflectional variants) and we merge them to alleviate theduplication problem. We remark that some more sophis-ticated work can be applied here to help detect differentlexical semantic and syntactic relations. We leave this forfuture work.

Once the topological order among domain keyphrase nodeshas been decided, one can concatenate all content units rightafter the sorted keyphrases and link higher-ranked keyphrasenodes to lower-ranked content units when sim(Km, Tj) ≥ β.We initialize link weights between nodes to be their sim(·, ·)scores. As for the leak terms of nodes, they are simply setto be the probability of observing them in the corpus.

5. ONLINE INFERENCEThe online inference is designed to efficiently quantify the

relatedness between the text query and its potential docu-ment keyphrases. Inspired by the sufficient statistics col-lected in E-step, we are particularly interested in computing

P(q)m = p(Km|T = t(q)), i.e., the activation probability for

a certain keyphrase Km, as the non-negative weight for themth entry in our output vectorial representation.

Notice that in the online inference phase, efficiency isusually a big challenge and the inference previously men-tioned for the E-step will be intractable if the documentbecomes too long. In this regard, we resort to an approx-imate sampling approach. This technique can be applied

to compute both P(q)m and P

(d)m . At the same time, R

(d)ij =

p(Xij , Paij |T = t(d)) needed in the E-step can also be bene-

fited.In the last section, we discussed about exact inference

in the E-step where enumeration over potential documentkeyphrase states are necessary to help compute the aboveterms. In fact, they can be more efficiently approximatedby use of Monte Carlo sampling methods, which are a setof computational techniques for generating samples from atarget distribution like the joint probability p(K,T = t) inour setting. Among the Monte Carlo family, we apply Gibbssampling in this work to sample keyphrase variables duringeach inference procedure. Given content unit vector t rep-resenting a document d or query q, we proceed as follows:

1. Start with initial setting: only observed content units andexplicit document keyphrases are set to be true, denotedby k(0), t

2. For each s ∈ 1, . . . , S, sequentially sample all keyphrasenodes following conditional distribution p(Km|K−m =

k(s)1 , . . . , k

(s)m−1, k

(s−1)m+1 , . . . , k

(s−1)M , T = t), denoted as

k(s).

wherep(Km = 1|K−m = k−m, T = t) =

p(Km = 1, K−m = k−m, T = t)∑1i=0 p(Km = i,K−m = k−m, T = t)

(3)

To compute Eq. 3 efficiently, for every domain keyphrasenode, we maintain the following probability ratio:

p(Km = 1,K−m = k−m, T = t)

p(Km = 0,K−m = k−m, T = t)(4)

where K−m refers to all the keyphrase nodes except Km.Given the above ratio, one can easily compute Eq. 3 neededfor sampling Km.

Now the problem becomes how to maintain Eq. 4 for eachkeyphrase node during the sampling process. In fact, ac-cording to the chain rule in Bayesian network, we have

p(Km = 1,K−m = k−m, T = t)

p(Km = 0,K−m = k−m, T = t)=p(Km = 1|Pa(Km))

p(Km = 0|Pa(Km))

×∏

Zj∈Ch(Km)

p(Zj |Pa−Km (Zj),Km = 1)

p(Zj |Pa−Km (Zj),Km = 0)

where Ch(Km) refers Km’s children. From the above equa-tion, one can conclude that Eq. 4 for node Km should be up-dated whenever nodes in its Markov blanket3 change states.

The above Gibbs sampling process ensures that samplesapproximate the joint probability distribution between allkeyphrase variables and content units. Such sampling isperformed over all the original nodes in the network butdoes not include the grey nodes (see Fig. 3) in the alternativerepresentation for the sake of sampling efficiency. One caneasily compute the probability distribution over each of thegrey nodes given a domain keyphrase state combination.

3Markov blanket for a node in a Bayesian network composedof its parents, children and children’s other parents.

1061

Method Semantic Space Input Source ToolkitESA KB concepts KB ESAlibKBLink KB concepts KB WikiBrainBoW Words - scikit-learnESA-C Documents Corpus ESAlibLSA Topics Corpus scikit-learnLDA Topics Corpus MALLETWord2Vec - Corpus gensimEKM Explicit Keyphrases Corpus -LAKI Latent Keyphrases Corpus -

Table 2: Comparisons among different methods

By marginalizing over necessary domain keyphrase vari-ables in the Bayesian network, the following approximateequations can be derived:

Rij =

∑Ss=1

p(Xij=1|Paij)

p(Zj=z(s)j|Pa(Zj))

1z(s)j

1Pai

j

S, Pm =

∑Ss=1 1

k(s)m

S

For online inference, the ultimate representation for text

query q is a high-dimensional vector [P(q)1 , P

(q)2 , · · · , P (q)

M ]where non-zero entries indicate document keyphrases.

To further improve the efficiency of Gibbs sampling, onecan follow the idea of E-step to reduce the number of sam-pled nodes. Intuitively, only a small portion of domainkeyphrases are related to the text query. There is no needto sample all keyphrase nodes since most of them do nothave chance to get activated. That is to say, we can skipmajority of them based on a reasonable relatedness predic-tion before conducting Gibbs sampling. Suppose contentunit vector T ′ = T ′1, · · · , T ′l ⊆ T contains only observedcontent units. We pick the following scoring function:

p(T ′ = 1, · · · , 1|Zj = 1)

This score can be viewed as the probability of generatingthe observed content units when domain keyphrase Zj isactivated. Computing p(T ′ = 1, · · · , 1|Zj = 1) is stillchallenging because keyphrases are connected and enumer-ation over state combinations of connected keyphrases areunavoidable. Thus we adopt a local arborescence structureinspired by [22] to approximate the probability by keepingthe path from Zj to each content unit T ′r for which theactivation probability product along the path is maximumamong all paths from Zj to T ′r. In this way, the activationprobability for each content unit is independent given Zj isactivated. The associate probability of the approximate linkbetween keyphrase node Zj and content unit Tr is denotedas p(T ′r = 1|Zj = 1). We then have:

p(T′= 1, · · · , 1|Zj = 1) ≈

∏r

(1−(1− p(T

′r = 1|Z0)

)×

(1− p(T

′r = 1|Zj = 1)

))In addition, the score can be naturally and efficiently prop-agated from children to parents in the network recursively:

p(T ′r = 1|Zj = 1) = maxi

p(T ′r = 1|Chij = 1) p(Chij = 1|Zj = 1)

where Chij refers to the ith child node of Zj . The above

equation can be computed efficiently following reverse topo-logical order of domain keyphrase nodes.

6. EXPERIMENTAL STUDYIn this section, experiments were conducted to demon-

strate the effectiveness of the proposed LAKI in generatinghigh quality vectorial representations of text queries. Webegin with the description of datasets.

Dataset #Docs #Words Content typeAcademia 0.43M 28M title & abstractYelp 0.47M 98M review

Table 3: Dataset statistics

6.1 DatasetsTwo real-world data sets were used in the experiments

and detailed statistics are summarized in Table 3.

• The Academia dataset4 is a collection of major computerscience publications. We use both paper titles and ab-stracts in venues of database, data mining, machine learn-ing, natural language processing and computer vision.

• The Yelp dataset5 provides reviews of 250 businesses. Weextract all the reviews belong to the “restaurant” categoryand each individual review is considered as a document.

6.2 Compared MethodsThe proposed method is compared with an extensive set

of methods. They are briefly described as follows.

• Explicit Semantic Analysis (ESA) [10] views each textquery as a weighted vector of KB entries, where the valueson each dimension denote the similarity scores computedbetween the query and the associated KB entries

• KBLink [23] first detects related KB entries in the queryand then represents it using the hyperlink structures ofthe KB centered at the identified entries

• BoW stands for bag-of-words method where each dimen-sion reflects word frequency in the query

• ESA-C extends ESA by replacing the general KB with adomain-specific corpus where each document is consideredto be a KB entry in the original ESA framework

• Latent Semantic Analysis (LSA) [7] is a topic modelingtechnique learning word and document representations byapplying Singular Value Decomposition to the words-by-documents co-occurrence matrix

• Latent Dirichlet Allocation (LDA) [3] is a probabilistictopic model assuming words in each document were gen-erated by a mixture of topics, where a topic is representedas a multinomial probability distribution over words

• Word2Vec [19] computes continuous distributed repre-sentations of words by training a neural network, with thedesideratum that words with similar meanings will mapto similar vectors

• Explicit Keyphrase Mentiosn (EKM) uses bag of explicitdomain keyphrase mentions

• Latent Keyphrase Inference (LAKI) is proposed in thiswork to derive document representation via inferring la-tent keyphrases in the text.

Table 2 provides more details about the differences amongthe above methods. The first two methods utilize knowledgebase as the input source to train models in order to representany new text query. In contrast, the rest methods exceptBoW require a domain-specific corpus as the training source.

6.3 Experimental SettingFor domain keyphrase extraction, we set the minimum

keyphrase support as 10 and the maximum keyphrase lengthas 6, which are two parameters required by SegPhrase. The

4http://aminer.org/billboard/AMinerNetwork5https://www.yelp.com/academic_dataset

1062

process generates 33 and 25 thousand domain keyphrases onAcademia and Yelp respectively. Among them, only 6367and 4996 exist in Wikipedia, which implies its knowledgeincompleteness.

LAKI was initialized and trained following Sec. 4.2 onboth datasets, whose model information is listed in Table 4.EKM uses the same set of domain keyphrases. The Yelpmodel contains more content units but has fewer domainkeyphrases and links, indicating the knowledge underlyingYelp reviews is less structured. Regarding the inference pro-cess of LAKI, we run the Gibbs sampler for 5000 iterationsper query. The pruning strategy introduced in Sec. 5 is ap-plied to select at most 200 keyphrase nodes as candidates forsampling. Settings of these two parameters will be discussedlater in this section. EM steps are repeated until the changeon training perplexity is small enough.

Regarding other methods, ESA and KBLink need a gen-eral KB as the input source. Wikipedia is chosen since itis rich in both text content and structural links. For LSAand LDA, both require users to specify number of topicsbefore training. During our experiments, various numbersof topics ranging from 10 to 1000 have been tested and thebest got reported. Word2Vec has two main learning algo-rithms: continuous bag-of-words and continuous skip-gram.We compared both of them in our tasks and discovered theformer generally worked better. For the sake of convenience,results of the continuous bag-of-words algorithm with con-text window size of 5 are reported. As suggested in thepaper, negative sampling was activated and vector dimen-sionality was set to be 300.

In addition, TF-IDF is applied to re-weight terms forBoW, LSA and EKM. Stop words are removed for all themethods. Other parameters required by contrasting meth-ods were set as the default values according to the toolkits.

6.4 EvaluationThe goal of our experiments is to quantitatively evaluate

how well our method performs in generating vectorial rep-resentations of online queries. We introduce our empiricalevaluation in two problem domains: phrase relatedness anddocument classification.

Phrase Relatedness: For a set of phrase pairs, methodperformance is evaluated by how well the generated relat-edness scores correlate with the gold scores. The gold scorefor each phrase pair is based on human judgements of thepair’s semantic relatedness.

To create these phrase pairs, we first generate 100 pairs offrequently co-occurred and potentially realated phrases fromconcept mentions in Wiki articles. Then we expanded theset by randomly combining phrases from different pairs andthe size of final evaluation set is 300. Each pair was carefullyassigned a relatedness score from 0 to 3 where higher scoreindicates stronger similarity. We used Pearson’s linear cor-relation coefficient to compare computed relatedness scoreswith human judgements.

Document Classification: In this classification task, wewish to classify a document into several mutually exclusiveclasses. A challenging aspect of the document classifica-tion problem is the choice of features. Considering differentvectorial representations as document features, we expectthe classification accuracy can well reflect the discrimina-tive power of each method.

Dataset #Nodes #Links #Domain Keyphr. (in Wiki)Academia 237K 1.40M 33K (6,367)Yelp 292K 1.14M 25K (4,996)

Table 4: Model information of LAKI

For each compared method, we first derived vectorial rep-resentations for all the training and testing documents with-out reference to their true class label. Then a support vectormachine with both linear and radial basis function kernelswas trained on the training set (only the best was reported).Classification accuracy on the testing set is reported to mea-sure the performance of the method.

For the methods trained on the Academia dataset, wecreated a held-out set of 500 publications authored by fiveresearchers in different research areas. We sampled each au-thor’s publications to avoid the problem of class imbalance.Regarding the Yelp dataset, we followed the above proce-dure and extracted 1000 reviews for 10 chain restaurants.As for the classification details, we used LibSVM softwarepackage and created the training set with 70% of all docu-ments. Five-fold cross validation was conducted to decidethe proper parameter of the kernel. Five test runs were con-ducted on different randomly partitioned training and testsets. The average performance is reported.

6.5 ResultsThe correlations between computed relatedness scores and

human judgements are shown in Table 5. Let’s first ignorethe numbers in parenthesis and we can discover that thetrends on two datasets are similar.

Both BoW and EKM are not reported because queries(i.e., phrases) are too short and they tend to return 0 mostof the time, making it inappropriate for this task. Amongother existing work, Word2Vec has the best performancedue to its effective modeling of word representations in lowdimensional space. Meanwhile, Word2Vec is good at rep-resenting very short text by simply computing arithmeticmean of word embeddings. LDA performs the worst in thistask, suffering from the severe data sparsity in short textinference scenario. This phenomenon is consistent with re-cent studies on short text topic modeling. As for the twoKB-based methods, both behave relatively poor in spite oftheir extraordinary performance reported on open domain in[10, 23]. KBLink is noticeably lower on Academia datasetdue to the sparsity of hyperlinks between academic concepts.This is probably due to the reasons about domain restric-tion and out-of-domain noise. We can verify this assump-tion to some extent by comparing it with ESA-C. The lattermethod replaces KB with a domain corpus, which can beconsidered as a weak domain adaptation by viewing eachdocument as a KB entry. ESA-C thus outperforms ESAslightly but its performance is still not satisfactory comparedto LSA and Word2Vec. This implies enormous potential if amethod is able to handle domain adaptation well, and mean-while to bring KB-like semantics into each dimension. Ourproposed method, LAKI, first extracts domain keyphrasesfrom a corpus and then represents queries in the space ofthese keyphrases, fulfilling the above two targets simultane-ously. Consequently, it outperforms the second best method(i.e., Word2Vec) by 0.083 on Academia and 0.0466 on Yelpdataset, which is even more statistically significant than thegap between the second and the third best methods.

Table 6 shows evaluation results for the document classi-fication task. LAKI still achieves much improvement com-

1063

#Iterations1 3 5 7 9 11 13 15

Pe

rple

xity (

log

-sca

le)

101

102

103

AcademiaYelp

#Iterations1 3 5 7 9 11 13 15

Ph

rase

Re

lete

dn

ess C

orr

ela

tio

n

0.5

0.6

0.7

0.8

AcademiaYelp

#Iterations1 3 5 7 9 11 13 15

Do

cu

me

nt C

lassific

atio

n A

ccu

racy

55

65

75

85

95

AcademiaYelp

Figure 4: Training perplexity and performance of LAKI versus in-creasing iterations of EM

#Domain Keyphrases After Pruning0 100 200 300 400 500

Ph

rase

Re

lete

dn

ess C

orr

ela

tio

n

0.6

0.65

0.7

0.75

AcademiaYelp

#Domain Keyphrases After Pruning0 100 200 300 400 500D

ocu

me

nt C

lassific

atio

n A

ccu

racy

70

75

80

85

90

95

AcademiaYelp

Figure 5: Performance variations of LAKIwith increasing number of keyphrases

Method Academia (w/ phrase) Yelp (w/ phrase)ESA 0.4320 (-) 0.4567 (-)KBLink 0.1878 (-) 0.4179 (-)ESA-C 0.4905 (0.5243) 0.4655 (0.5029)LSA 0.5877 (0.6383) 0.6700 (0.7229)LDA 0.3610 (0.5391) 0.3928 (0.5405)Word2Vec 0.6674 (0.7281) 0.7143 (0.7419)LAKI 0.7504 0.7609

Table 5: Phrase relatedness correlation

pared with the competitors. The difference is especially no-ticeable on Yelp, where LAKI achieves 90.58% accuracy andthe second best is 75.55%. It supports our claim that theproposed LAKI method is quite useful in representing bothshort-text and long-text documents. Among the other meth-ods, LDA generates the best results since each documentnow contains more words than the previous short text sce-nario, making it easier to do inference by utilizing the wordco-occurrence patterns. BoW is not performing well mainlydue to the over-sparsity problem. Though documents arerelatively long (100 words for Academia and 200 words forYelp on average), some documents from the same categorystill share only a few words, which makes the classifier easyto misclassify. Similar to BoW, EMK performs much worsethan LAKI due to the sparsity of keyphrase mentions, in-dicating the superiority of using latent keyphrases over ex-plicit mentions. ESA-C beats ESA once again, reflectingthe shortcoming of using general KB on a topic-focused do-main. Word2Vec is omitted from the table due to its poorperformance when applied to long text. Simply computingarithmetic or geometric mean of word embeddings results invery poor performance (close to random guess). We havealso tried applying gradient descent to learn the documentrepresentations as introduced in [16]. But the performanceis still far below our expectation.

Another significant difference between LAKI and the ex-isting work is its support for multi-word phrases. By seg-menting queries into content units, LAKI views each inputquery as a bag of phrases instead of words. Some othermethods including BoW, ESA-C, LSA, LDA and Word2Veccan be modified to support such phrase-based input just likeLAKI. Therefore, we have conducted experiments to showwhether using the same input as LAKI can help boost theirperformance. Numbers within parenthesis in Tables 5 and6 indicate the corresponding performance after switching tothe phrase-based input. The highest correlation scores ob-tained by Word2Vec increase to 0.7218 and 0.7419 on thetwo datasets, which are still beated by LAKI. For the doc-ument classification task, LDA achieves the best accuracyamong all contrasting methods, which are still 3.9% and7.93% lower than LAKI for the two datasets respectively.It is interesting to see that BoW fails to utilize the phrase-based input. Our explanation is that BoW is lack of a train-ing process and it relies fully on the content units in the

Method Academia (w/ phrase) Yelp (w/ phrase)ESA 37.61 (-) 46.56 (-)KBLink 36.37 (-) 35.94 (-)BoW 48.05 (45.60) 51.26 (45.97)ESA-C 39.75 (42.20) 49.13 (54.51)LSA 72.50 (79.22) 66.55 (78.57)LDA 77.27 (80.52) 75.55 (82.65)EKM 45.46 40.57LAKI 84.42 90.58

Table 6: Document classification accuracy (%)

input. It usually becomes more difficult for queries to sharephrases than words due to the decrease in number of contentunits after segmentation.

6.6 Model SelectionIn this subsection, we study the model behavior under

different experimental settings. We begin with an empiri-cal convergence study of the EM algorithm. Fig. 4 presentsthe training perplexity of LAKI with its performance ver-sus iteration. Due to the good initialization discussed inSec. 4.2, the perplexity becomes quite stable after the fifthiteration. This help save a lot of training time in practice.The perplexity is not monotonically decreasing because ofthe approximations resulted from pruning and sampling.

There are two parameters in our LAKI method: num-ber of domain keyphrases after pruning and sample size forGibbs sampler. The former parameter decides the num-ber of keyphrase nodes involved in the later sampling pro-cess. A smaller value will make the final output sparserwhile a larger one has risk in incorporating more unrelatedkeyphrases to undermine the performance. Fig. 5 shows howthe performance varies with changes in this parameter. Wecan observe a peak around 150 keyphrases for the phraserelatedness task. In contrast, the curve is generally goingup for the document classification task and becomes sta-ble around 400. Our explanation is that the queries forthe latter task contain more observed content units andthere should exist more document keyphrases for longer textqueries. This suggests a way to dynamically decide the num-ber of pruned keyphrases. But in this work we simply fix itsvalue to 200 and plan to explore this idea in the future.

For the Gibbs sampling size, a larger value usually leadsto more accurate and stable estimation of the probabilitydistribution. We show the experiment in Fig. 8 where bothmean and standard deviation are reported. The curves areconsistent with our expectation and they become relativelyflat after 5000. Based on this observation we set the sam-pling size to 5000 for the sake of saving time consumption.Another experiment has been conducted to show the bene-fit by considering more domain keyphrases into the Bayesiannetwork. As reported in Table 4, there is a large portion ofkeyphrases not existing in Wikipedia. To justify the do-main restriction of Wiki, it is interesting to compare the

1064

#Samples1000 3000 5000 7000 9000

Ru

nin

g T

ime

(m

s)

100

200

300

400

500

AcademiaYelp

#Domain Keyphrases After Pruning10 100 200 300 400 500

Ru

nin

g T

ime

(m

s)

100

200

300

400

500

AcademiaYelp

#Words0 100 200 400 800

Ru

nin

g T

ime

(m

s)

0

500

1000

1500

AcademiaYelp

Figure 6: Impact of sample size, number of keyphrases, and wordcounts on query processing time

Sampling70.6% Pruning

29.2 %

Segmentation

0.2%

Sampling87.1%

Pruning12.7 %

Segmentation

0.2%

50-word Query 400-word Query

Figure 7: Breakdown of query processingtime

#Samples1000 3000 5000 7000 9000

Phra

se R

ele

tedness C

orr

ela

tion

0.7

0.72

0.74

0.76

0.78

AcademiaYelp

#Samples1000 3000 5000 7000 9000D

ocum

ent C

lassific

ation A

ccura

cy

82

84

86

88

90

92

AcademiaYelp

Figure 8: Performance with increasing sample size

standard LAKI model with the simplified version trainedonly on Wiki-covered keyphrases (see its definition in [17]).The bar plots in Fig. 9 demonstrate that with more domainkeyphrases involved, LAKI is able to achieve outstandingimprovement.

Academia Yelp

Phra

se R

ele

tedness C

orr

ela

tion

0.65

0.7

0.75

0.8Wiki-covered KeyphrasesAll Keyphrases

Academia YelpDocum

ent C

lassific

ation A

ccura

cy

70

80

90

Wiki-covered KeyphrasesAll Keyphrases

Figure 9: Performance with different keyphrase sets

6.7 Scalability StudyTo understand the run-time complexity of our framework,

we first analyse the execution time of online query inferencephase. The experiments were conducted on a machine withIntel Core i7-2600K and 16GB memory. The code is imple-mented in C++. As shown in Fig. 6, LAKI grows linearlyproportional to the sampling size, the number of domainkeyphrases after pruning and the length of the query. Be-sides this, the pies in Fig. 7 show ratios of different com-ponents of our framework. One can observe that the prun-ing and sampling steps occupy most of the runtime. More-over, as query size increases, the sampling part consumesrelatively more. Fortunately, almost all components of ourframeworks can be easily parallelized, because of the na-ture of independence between documents and queries. Forthe offline keyphrase learning phase, as the very first step,keyphrase extraction shares similar time complexity as re-ported in [17]. The most time consuming part is domainkeyphrase silhouetting, which is an EM algorithm and eachindividual inference in E-step has similar performance com-pared to Figs. 6 and 7. The only difference is the fewerpruning time for silhouetting because the search space isconstrained as discussed in Sec. 4.1.

6.8 Case StudyPrevious experiments are focused on evaluating represen-

tation quality and time complexity quantitatively. In thissubsection, we first present several queries with their top-

ranked document keyphrases in Table 7 generated from theonline phase of LAKI. Overall we see that LAKI can han-dle both short and long queries quite well. Most documentkeyphrases are successfully identified in the list. Related-ness between keyphrase and queries generally drops withranking lowers down. Meanwhile, both general and specificdocument keyphrases exist in the ranked list. This providesresults from LAKI with more discriminative power whensomeone is applying it to text mining applications like doc-ument clustering and classification. Moreover, LAKI has theability to process ambiguous queries like “lda” based on con-textual words“topic”. We attribute this to the well-modelleddomain keyphrase silhouettes and we show some examplesof them in Table 8. As a domain keyphrase silhouette mightcontain many content units, we only demonstrate ones withthe most significant link weights. For ease of presentation,link weights are omitted in the table.

7. CONCLUSIONSIn this paper, we have introduced a new research prob-

lem of learning representation for domain-specific texts. Wepropose a novel method called Latent Keyphrase Inferencewhich integrates domain keyphrase extraction and silhouet-ting to help infer latent document keyphrases and solves therarity of explicit keyphrase mentions in the query. The gen-erated high-dimensional representations of documents areshown to significantly boost performance in potential textmining tasks compared to state-of-art methods. Meanwhile,entries in the document vector are highly self-explanatorythrough automatically learned domain keyphrase silhouettes.

A number of open problems need to be solved to allowfurther development of LAKI. One direction is to simulta-neously model structured data such as meta information as-sociated with the documents, including named entity, au-thorship, publishers, etc.. An alternative is to improve themodel initialization by modeling more sophisticated rela-tionship between domain keyphrases and considering morerobust structured learning method. Regarding the scalabil-ity, it would be preferable if one can work out a more efficientinference algorithm. For example, we can use a determinis-tic module like neural network and train it using inferenceresults from our current framework.

AcknowledgementsResearch was sponsored in part by the U.S. Army ResearchLab. under Cooperative Agreement No. W911NF-09-2-0053(NSCTA), National Science Foundation IIS-1017362, IIS-1320617, and IIS-1354329, HDTRA1-10-1-0120, and grant1U54GM114838 awarded by NIGMS through funds providedby the trans-NIH Big Data to Knowledge (BD2K) initia-tive (www.bd2k.nih.gov), and MIAS, a DHS-IDS Center forMultimodal Information Access and Synthesis at UIUC.

1065

Query LDA BOA

DocumentKeyphrases

linear discriminant analysis, latent dirichletallocation, topic models, topic modeling, facerecognition, latent dirichlet, generative model,topic, subspace models, . . .

boa steakhouse, bank of america, stripsteak,agnolotti, credit card, santa monica, restaurants,wells fargo, steakhouse, prime rib, bank, vegas,las vegas, cash, cut, dinner, bank, money, . . .

Query LDA topic BOA steak

DocumentKeyphrases

latent dirichlet allocation, topic, topic models,topic modeling, probabilistic topic models, latenttopics, topic discovery, generative model,mixture, text mining, topic distribution, . . .

steak, stripsteak, boa steakhouse, steakhouse,ribeye, craftsteak, santa monica, medium rare,prime, vegas, entrees, potatoes, french fries, filetmignon, mashed potatoes, texas roadhouse, . . .

Query SVM deep dish pizza

DocumentKeyphrases

support vector machines, svm classifier, multiclass, training set, margin, knn, classificationproblems, kernel function, multi class svm, multiclass support vector machine, support vector, . . .

deep dish pizza, chicago, deep dish, amore tasteof chicago, amore, pizza, oregano, chicago style,chicago style deep dish pizza, thin crust, windycity, slice, pan, oven, pepperoni, hot dog, . . .

QueryMining Frequent Patterns without CandidateGeneration

I am a huge fan of the All You Can Eat Chinesefood buffet.

DocumentKeyphrases

mining frequent patterns, candidate generation,frequent pattern mining, candidate, prune, fpgrowth, frequent pattern tree, apriori, subtrees,frequent patterns, candidate sets, . . .

all you can eat, chinese food, buffet, chinesebuffet, dim sum, orange chicken, chineserestaurant, asian food, asian buffet, crab legs,lunch buffet, fan, salad bar, all you can drink, . . .

Query

Text mining, also referred to as text data mining,roughly equivalent to text analytics, refers to theprocess of deriving high-quality information fromtext. High-quality information is typically derivedthrough means such as statistical patternlearning.

It’s the perfect steakhouse for both meat and fishlovers. My table guest was completely deliriousabout his Kobe Beef and my lobster was perfectlycooked. Good wine list, they have a lovelySancerre! Professional staff, quick and smooth.

DocumentKeyphrases

text analytics, text mining, patterns, text,textual data, topic, information, text documents,information extraction, machine learning, datamining, knowledge discovery, . . .

kobe beef, fish lovers, steakhouse, sancerre, winelist, guests, perfectly cooked, lobster, staff, meat,fillet, fish, lover, seafood, ribeye, filet, sea bass,risotto, starter, scallops, steak, beef, . . .︸ ︷︷ ︸︸ ︷︷ ︸

Academia YelpTable 7: Examples of document representation by LAKI with top-ranked document keyphrases in the vector (related-ness scores are ommited due to the space limit).

Domain Keyphr. linear discriminant analysis boa steakhouse

Silhouettelinear discriminant analysis, lda, face recognition,feature extraction, principle component analysis,uncorrelated, between class scatter, . . .

boa steakhouse, boa, steakhouse, restaurant,dinner, strip steak, craftsteak, santa monica,vegas, filet, ribeye, new york strip, sushi roku, . . .

Domain Keyphr. latent dirichlet allocation ribeye

Silhouettelatent dirichlet allocation, lda, topics, perplexity,variants, subspace, mixture, baselines, topicmodels, text mining, bag of words, . . .

ribeye, steak, medium rare, medium, oz,marbled, new york strip, well done, prime rib,fatty, juicy, top sirloin, filet mignon, fillet, . . .

Domain Keyphr. support vector machines deep dish

Silhouettesupport vector machines, svm, classification,training, classifier, machine learning, prediction,hybrid, kernel, feature selection, . . .

deep dish, pizza, crust, thin crust pizza, chicago,slice, pepperoni, deep dish pizza, pan style, pizzajoints, oregano, stuffed crust, chicago style, . . .

Domain Keyphr. fp growth chinese food

Silhouettefp growth, algorithm, apriori like, mining,apriori, frequent patterns, mining associationrules, frequent pattern mining, fp tree, . . .

chinese food, food, chinese, restaurants,americanized, asian, orange chicken, chow mein,wok, dim sum, panda express, chinese cuisine, . . .

Domain Keyphr. text mining mcdonalds

Silhouettetext mining, text, information retrieval, machinelearning, topics, knowledge discovery, text datamining, text clustering, nlp, . . .

mcdonalds, drive through, fast food, mcnugget,mcflurry, fast food chain, sausage mcmuffin, bigbag, mcmuffin, burger king, . . .

Domain Keyphr. database sushi

Silhouettedatabase, information, set, objects, storing,retrieval, queries, accessing, relational, indexing,record, tables, query processing, transactions, . . .

sushi, rolls, japanese, sushi joint, seafood, ayce,sushi rolls, salmon sushi, tuna sushi, californiaroll, sashimi, sushi lovers, sushi fish, . . .︸ ︷︷ ︸︸ ︷︷ ︸

Academia YelpTable 8: Examples of domain keyphrase silhouettes (from offline domain keyphrase learning). Link weights areommited.

1066

8. REFERENCES[1] R. Baeza-Yates, B. Ribeiro-Neto, et al. Modern

information retrieval, volume 463. 1999.

[2] C. Bizer, J. Lehmann, G. Kobilarov, S. Auer,C. Becker, R. Cyganiak, and S. Hellmann. Dbpedia-acrystallization point for the web of data. WebSemantics: science, services and agents on the worldWide Web, 7(3):154–165, 2009.

[3] D. M. Blei, A. Y. Ng, and M. I. Jordan. Latentdirichlet allocation. Journal of Machine LearningResearch, 3:993–1022, 2003.

[4] J. Chang, J. Boyd-Graber, C. Wang, S. Gerrish, andD. M. Blei. Reading tea leaves: How humans interprettopic models. In Advances in Neural InformationProcessing Systems 22, 2009.

[5] G. F. Cooper. The computational complexity ofprobabilistic inference using bayesian belief networks.Artificial Intelligence, 42(2):393–405, 1990.

[6] M. Y. Dahab, H. A. Hassan, and A. Rafea.Textontoex: Automatic ontology construction fromnatural english text. Expert Systems with Applications,34(2):1474–1480, 2008.

[7] S. C. Deerwester, S. T. Dumais, T. K. Landauer,G. W. Furnas, and R. A. Harshman. Indexing bylatent semantic analysis. Journal of the AmericanSociety of Information Science, 41(6):391–407, 1990.

[8] A. El-Kishky, Y. Song, C. Wang, C. R. Voss, andJ. Han. Scalable topical phrase mining from textcorpora. Proceedings of the VLDB Endowment,8(3):305–316, 2014.

[9] K. Frantzi, S. Ananiadou, and H. Mima. Automaticrecognition of multi-word terms:. the c-value/nc-valuemethod. International Journal on Digital Libraries,3(2):115–130, 2000.

[10] E. Gabrilovich and S. Markovitch. Computingsemantic relatedness using wikipedia-based explicitsemantic analysis. In Proceedings of the 20thInternational Joint Conference on ArtificalIntelligence, pages 1606–1611, 2007.

[11] W. A. Gale, K. W. Church, and D. Yarowsky. Onesense per discourse. In Proceedings of the Workshop onSpeech and Natural Language, pages 233–237.Association for Computational Linguistics, 1992.

[12] T. Gottron, M. Anderka, and B. Stein. Insights intoexplicit semantic analysis. In Proceedings of the 20thACM International Conference on Information andKnowledge Management, pages 1961–1964, 2011.

[13] Y. Halpern and D. Sontag. Unsupervised learning ofnoisy-or bayesian networks. In Proceedings of theTwenty-Ninth Conference on Uncertainty in ArtificialIntelligence, pages 272–281, 2013.

[14] S. Hassan and R. Mihalcea. Semantic relatednessusing salient semantic analysis. In Proceedings of theTwenty-Fifth AAAI Conference on ArtificialIntelligence, 2011.

[15] A. Hulth. Improved automatic keyword extractiongiven more linguistic knowledge. In Proceedings of the

2003 Conference on Empirical Methods in NaturalLanguage Processing, pages 216–223, 2003.

[16] Q. Le and T. Mikolov. Distributed representations ofsentences and documents. In Proceedings of the 31thInternational Conference on Machine Learning, pages1188–1196, 2014.

[17] J. Liu, J. Shang, C. Wang, X. Ren, and J. Han.Mining quality phrases from massive text corpora. InProceedings of the 2015 ACM SIGMOD InternationalConference on Management of Data, pages 1729–1744,2015.

[18] Q. Mei, X. Shen, and C. Zhai. Automatic labeling ofmultinomial topic models. In Proceedings of the 13thACM SIGKDD International Conference onKnowledge Discovery and Data Mining, pages490–499. ACM, 2007.

[19] T. Mikolov, K. Chen, G. Corrado, and J. Dean.Efficient estimation of word representations in vectorspace. In Proceedings of Workshop at ICLR, 2013.

[20] M. Sanderson and B. Croft. Deriving concepthierarchies from text. In Proceedings of the 22ndAnnual International ACM SIGIR Conference onResearch and Development in Information Retrieval,pages 206–213, 1999.

[21] Y. Song, H. Wang, Z. Wang, H. Li, and W. Chen.Short text conceptualization using a probabilisticknowledgebase. In Proceedings of the Twenty-SecondInternational Joint Conference on ArtificialIntelligence, pages 2330–2336, 2011.

[22] C. Wang, W. Chen, and Y. Wang. Scalable influencemaximization for independent cascade model inlarge-scale social networks. Data Mining andKnowledge Discovery, 25(3):545–576, 2012.

[23] I. Witten and D. Milne. An effective, low-cost measureof semantic relatedness obtained from wikipedia links.In Proceedings of AAAI Workshop on Wikipedia andArtificial Intelligence: an Evolving Synergy, 2008.

[24] I. H. Witten, G. W. Paynter, E. Frank, C. Gutwin,and C. G. Nevill-Manning. Kea: Practical automatickeyphrase extraction. In Proceedings of the FourthACM Conference on Digital Libraries, pages 254–255.ACM, 1999.

[25] W. Wu, H. Li, H. Wang, and K. Q. Zhu. Probase: Aprobabilistic taxonomy for text understanding. InProceedings of the 2012 ACM SIGMOD InternationalConference on Management of Data, pages 481–492,2012.

[26] X. Yin and S. Shah. Building taxonomy of web searchintents for name entity queries. In Proceedings of the19th International Conference on World Wide Web,pages 1001–1010, 2010.

[27] Z. Zhang, J. Iria, C. A. Brewster, and F. Ciravegna. Acomparative evaluation of term recognition algorithms.Proceedings of the Sixth International Conference onLanguage Resources and Evaluation, 2008.

1067