What You Say and How You Say it: Joint Modeling ofTopics and Discourse in Microblog Conversations

Jichuan Zeng1∗ Jing Li2 ∗ Yulan He3 Cuiyun Gao1 Michael R. Lyu1 Irwin King1

1Department of Computer Science and EngineeringThe Chinese University of Hong Kong, HKSAR, China

2Tencent AI Lab, Shenzhen, China3Department of Computer Science, University of Warwick, UK

1{jczeng, cygao, lyu, king}@cse.cuhk.edu.hk2ameliajli@tencent.com, 3yulan.he@warwick.ac.uk

Abstract

This paper presents an unsupervised frame-work for jointly modeling topic content anddiscourse behavior in microblog conversa-tions. Concretely, we propose a neural modelto discover word clusters indicating what aconversation concerns (i.e., topics) and thosereflecting how participants voice their opin-ions (i.e., discourse).1 Extensive experimentsshow that our model can yield both coherenttopics and meaningful discourse behavior. Fur-ther study shows that our topic and discourserepresentations can benefit the classificationof microblog messages, especially when theyare jointly trained with the classifier.

1 Introduction

The last decade has witnessed the revolutionof communication, where the ‘‘kitchen tableconversations’’ have been expanded to public dis-cussions on online platforms. As a consequence,in our daily life, the exposure to new informationand the exchange of personal opinions have beenmediated through microblogs, one popular onlineplatform genre (Bakshy et al., 2015). The flourishof microblogs has also led to the sheer quantityof user-created conversations emerging everyday, exposing individuals to superfluous infor-mation. Facing such an unprecedented number ofconversations relative to limited attention of indi-viduals, how shall we automatically extract thecritical points and make sense of these microblogconversations?

∗This work was partially conducted in Jichuan Zeng’sinternship in Tencent AI Lab. Corresponding author: Jing Li.

1Our data sets and code are available at: http://github.com/zengjichuan/Topic_Disc.

Toward key focus understanding of a conver-sation, previous work has shown the benefits ofdiscourse structure (Li et al., 2016b; Qin et al.,2017; Li et al., 2018), which shapes how messagesinteract with each other, forming the discussionflow, and can usefully reflect salient topics raisedin the discussion process. After all, the topicalcontent of a message naturally occurs in contextof the conversation discourse and hence should notbe modeled in isolation. Conversely, the extractedtopics can reveal the purpose of participants andfurther facilitate the understanding of their dis-course behavior (Qin et al., 2017). Further, thejoint effects of topics and discourse will contributeto better understanding of social media conversa-tions, benefiting downstream tasks such as themanagement of discussion topics and discoursebehavior of social chatbots (Zhou et al., 2018) andthe prediction of user engagements for conversa-tion recommendation (Zeng et al., 2018b).

To illustrate how the topics and discourse inter-play in a conversation, Figure 1 displays a snip-pet of Twitter conversation. As can be seen, thecontent words reflecting the discussion topics(such as ‘‘supreme court’’ and ‘‘gun rights’’) ap-pear in context of the discourse flow, whereparticipants carry the conversation forward viamaking a statement, giving a comment, asking aquestion, and so forth. Motivated by such an ob-servation, we assume that a microblog conver-sation can be decomposed into two cruciallydifferent components: one for topical contentand the other for discourse behavior. Here, thetopic components indicate what a conversationis centered around and reflect the important dis-cussion points put forward in the conversationprocess. The discourse components signal thediscourse roles of messages, such as making

267

Transactions of the Association for Computational Linguistics, vol. 7, pp. 267–281, 2019. Action Editor: Jianfeng Gao.Submission batch: 12/2018; Revision batch: 2/2019; Published 5/2019.

c© 2019 Association for Computational Linguistics. Distributed under a CC-BY 4.0 license.

Figure 1: A Twitter conversation snippet about thegun control issue in U.S. Topic words reflecting theconversation focus are in boldface. The italic words in[ ] are our interpretations of the messages’ discourseroles.

a statement, asking a question, and other dia-logue acts (Ritter et al., 2010; Joty et al., 2011),which further shape the discourse structure of aconversation.2 To distinguish the above two com-ponents, we examine the conversation contextsand identify two types of words: topic words,indicating what a conversation focuses on, anddiscourse words, reflecting how the opinion isvoiced in each message. For example, in Figure 1,the topic words ‘‘gun’’ and ‘‘control’’ indicate theconversation topic while the discourse word ‘‘what’’and ‘‘?’’ signal the question in M3.

Concretely, we propose a neural frameworkbuilt upon topic models, enabling the joint explo-ration of word clusters to represent topic anddiscourse in microblog conversations. Differentfrom the prior models trained on annotated data(Li et al., 2016b; Qin et al., 2017), our model isfully unsupervised, not dependent on annotationsfor either topics or discourse, which ensures itsimmediate applicability in any domain or lan-guage. Moreover, taking advantages of the recentadvances in neural topic models (Srivastava andSutton, 2017; Miao et al., 2017), we are able toapproximate Bayesian variational inference with-out requiring model-specific derivations, whereasmost existing work (Ritter et al., 2010; Joty et al.,2011; Alvarez-Melis and Saveski, 2016; Zenget al., 2018b; Li et al., 2018) require expertiseinvolved to customize model inference algo-rithms. In addition, our neural nature enables

2In this paper, the discourse role refers to a certain type ofdialogue act (e.g., statement or question) for each message.And the discourse structure refers to some combination ofdiscourse roles in a conversation.

end-to-end training of topic and discourse repre-sentation learning with other neural models fordiverse tasks.

For model evaluation, we conduct an extensiveempirical study on two large-scale Twitter datasets. The intrinsic results show that our modelcan produce latent topics and discourse roleswith better interpretability than the state-of-the-art models from previous studies. The extrinsicevaluations on a tweet classification task exhibitthe model’s ability to capture useful representa-tions for microblog messages. Particularly, ourmodel enables an easy combination with existingneural models for end-to-end training, such asconvolutional neural networks, which is shown toperform better in classification than the pipelineapproach without joint training.

2 Related Work

Our work is in the line with previous studiesthat use non-neural models to leverage discoursestructure for extracting topical content from con-versations (Li et al., 2016b; Qin et al., 2017; Liet al., 2018). Zeng et al. (2018b) explore how dis-course and topics jointly affect user engagementsin microblog discussions. Different from them,we build our model in a neural network frame-work, where the joint effects of topic and discourserepresentations can be exploited for various down-stream deep learning tasks in an end-to-end man-ner. In addition, we are inspired by prior researchthat only models topics or conversation discourse.In the following, we discuss them in turn.

Topic Modeling. Our work is closely relatedwith the topic model studies. In this field, despitethe huge success achieved by the springboard topicmodels (e.g., pLSA [Hofmann, 1999] and LDA[Blei et al., 2001]), and their extensions (Blei et al.,2003; Rosen-Zvi et al., 2004), the applications ofthese models have been limited to formal and well-edited documents, such as news reports (Blei et al.,2003) and scientific articles (Rosen-Zvi et al.,2004), attributed to their reliance on document-level word collocations. When processing shorttexts, such as the messages on microblogs, it islikely that the performance of these models willbe inevitably compromised, due to the severe datasparsity issue.

To deal with such an issue, many previous ef-forts incorporate the external representations, suchas word embeddings (Nguyen et al., 2015; Li et al.,

268

2016a; Shi et al., 2017) and knowledge (Songet al., 2011; Yang et al., 2015; Hu et al., 2016),pre-trained on large-scale high-quality resources.Different from them, our model learns topic anddiscourse representations only with the internaldata and thus can be widely applied on scenarioswhere the specific external resource is unavailable.

In another line of the research, most prior workfocuses on how to enrich the context of shortmessages. To this end, biterm topic model (BTM)(Yan et al., 2013) extends a message into a bitermset with all combinations of any two distinct wordsappearing in the message. On the contrary, ourmodel allows the richer context in a conversationto be exploited, where word collocation patternscan be captured beyond a short message.

In addition, there are many methods usingsome heuristic rules to aggregate short messagesinto long pseudo-documents, such as those basedon authorship (Hong and Davison, 2010; Zhaoet al., 2011) and hashtags (Ramage et al., 2010;Mehrotra et al., 2013). Compared with thesemethods, we model messages in the context oftheir conversations, which has been demonstratedto be a more natural and effective text aggregationstrategy for topic modeling (Alvarez-Melis andSaveski, 2016).

Conversation Discourse. Our work is also inthe area of discourse analysis for conversa-tions, ranging from the prediction of the shallowdiscourse roles on utterance level (Stolcke et al.,2000; Ji et al., 2016; Zhao et al., 2018) to thediscourse parsing for a more complex conversationstructure (Elsner and Charniak, 2008, 2010;Afantenos et al., 2015). In this area, most existingmodels heavily rely on the data annotated withdiscourse labels for learning (Zhao et al., 2017).Different from them, our model, in a fullyunsupervised way, identifies distributional wordclusters to represent latent discourse factors inconversations. Although such latent discoursevariables have been studied in previous work(Ritter et al., 2010; Joty et al., 2011; Ji et al., 2016;Zhao et al., 2018), none of them explores theeffects of latent discourse on the identification ofconversation topic, which is a gap our work fills in.

3 Our Neural Model for Topics andDiscourse in Conversations

This section introduces our neural model thatjointly explores latent representations for topics

and discourse in conversations. We first presentan overview of our model in Section 3.1, followedby the model generative process and inferenceprocedure in Section 3.2 and 3.3, respectively.

3.1 Model Overview

In general, our model aims to learn coherent wordclusters that reflect the latent topics and discourseroles embedded in the microblog conversations.To this end, we distinguish two latent componentsin the given collection: topics and discourse, eachrepresented by a certain type of word distribution(distributional word cluster). Specifically, at thecorpus level, we assume that there are K topics,represented by φTk (k = 1, 2, . . . ,K), and D dis-course roles, captured with φDd (d = 1, 2, . . . , D).φT and φD are all multinomial word distributionsover the vocabulary size V . Inspired by the neuraltopic models in Miao et al. (2017), our modelencodes topic and discourse distributions (φT andφD) as latent variables in a neural network andlearns the parameters via back propagation.

Before touching the details of our model, wefirst describe how we formulate the input. Onmicroblogs, as a message might have multiplereplies, messages in an entire conversation canbe organized as a tree with replying relations (Liet al., 2016b, 2018). Though the recent progressin recursive models allows the representationlearning from the tree-structured data, previousstudies have pointed out that, in practice, sequencemodels serve as a more simple yet robust alterna-tive (Li et al., 2015). In this work, we follow thecommon practice in most conversation modelingresearch (Ritter et al., 2010; Joty et al., 2011;Zhao et al., 2018) to take a conversation as asequence of turns. To this end, each conversationtree is flattened into root-to-leaf paths. Each one ofsuch paths is hence considered as a conversationinstance, and a message on the path correspondsto a conversation turn (Zarisheva and Scheffler,2015; Cerisara et al., 2018; Jiao et al., 2018).

The overall architecture of our model is shownin Figure 2. Formally, we formulate a conversationc as a sequence of messages (x1, x2, . . . , xMc),where Mc denotes the number of messages inc. In the conversation, each message x, as thetarget message, is fed into our model sequentially.Here we process the target message x as thebag-of-words (BoW) term vector xBoW ∈ RV ,following the bag-of-words assumption in most

269

Figure 2: The architecture of our neural framework.The latent topics z and latent discourse d are jointlymodeled from conversation c and target message x,respectively. Mutual information penalty (MI) is usedto separate words clusters representing topics anddiscourse. Afterward, z and d are used to reconstructthe target message x′.

topic models (Blei et al., 2003; Miao et al., 2017).The conversation, c, where the target message xis involved, is considered as the context of x.It is also encoded in the BoW form (denoted ascBoW ∈ RV ) and fed into our model. In doingso, we ensure that the context of the targetmessage is incorporated while learning its latentrepresentations.

Following the previous practice in neural topicmodels (Miao et al., 2017; Srivastava and Sutton,2017), we utilize the variational auto-encoder(VAE) (Kingma and Welling, 2013) to resemblethe data generative process via two steps. First,given the target message x and its conversation c,our model converts them into two latent variables:topic variable z and discourse variable d. Then,using the intermediate representations captured byz and d, we reconstruct the target message, x′.

3.2 Generative Process

In this section, we first describe the two latentvariables in our model: the topic variable z and thediscourse variable d. Then, we present our datagenerative process from the latent variables.

Latent Topics. For latent topic learning, weexamine the main discussion points in the contextof a conversation. Our assumption is that messagesin the same conversation tend to focus on similar

topics (Li et al., 2018; Zeng et al., 2018b). Con-cretely, we define the latent topic variable z ∈ RKat the conversation level and generate the topicmixture of c, denoted as a K-dimensional distri-bution θ, via a softmax construction conditionedon z (Miao et al., 2017).

Latent Discourse. For modeling the discoursestructure of conversations, we capture the message-level discourse roles reflecting the dialogue actsof each message, as is done in Ritter et al. (2010).Concretely, given the target message x, we usea D-dimensional one-hot vector to represent thelatent discourse variable d, where the high bitindicates the index of a discourse word distribu-tion that can best express x’s discourse role. Inthe generative process, the latent discourse d isdrawn from a multinomial distribution with pa-rameters estimated from the input data.

Data Generative Process As mentioned pre-viously, our entire framework is based on VAE,which consists of an encoder and a decoder.The encoder maps a given input into latent topicand discourse representations and the decoderreconstructs the original input from the latentrepresentations. In the following, we first describethe decoder followed by the encoder.

In general, our decoder is learned to reconstructthe words in the target message x (in the BoWform) from the latent topic z and latent discoursed. We show the generative story that reflects thereconstruction process below:

• Draw the latent topic z ∼ N (µ,σ2)

• c’s topic mixture θ = softmax(fθ(z))

• Draw the latent discourse d ∼Multi(π)

• For the n-th word in x

– βn = softmax(fφT (θ) + fφD(d))– Draw the word wn ∼Multi(βn)

where f∗(·) is a neural perceptron, with a lineartransformation of inputs activated by a non-lineartransformation. Here we use rectified linear units(Nair and Hinton, 2010) as the activate func-tions. In particular, the weight matrix of fφT (·)(after the softmax normalization) is consideredas the topic-word distributions φT . The discourse-word distributions φD are similarly obtained fromfφD(·).

270

For the encoder, we learn the parameters µ, σ,and π from the input xBoW and cBoW (the BoWform of the target message and its conversation),following the following formula:

µ = fµ(fe(cBoW )), logσ = fσ(fe(cBoW ))

π = softmax(fπ(xBoW ))(1)

3.3 Model Inference

For the objective function of our entire framework,we take three aspects into account: the learning oflatent topics and discourse, the reconstruction ofthe target messages, and the separation of topic-associated words and discourse-related words.

Learning Latent Topics and Discourse. Forlearning the latent topics/discourse in our model,we utilize the variational inference (Blei et al.,2016) to approximate posterior distribution overthe latent topic z and the latent discourse d givenall the training data. To this end, we maximize thevariational lower bound Lz for z and Ld for d,each defined as following:

Lz = Eq(z | c)[p(c | z)]−DKL(q(z | c) || p(z))Ld = Eq(d | x)[p(x |d)]−DKL(q(d | x) || p(d))

(2)

q(z | c) and q(d | x) are approximated posteriorprobabilities describing how the latent topic zand the latent discourse d are generated fromthe data. p(c | z) and p(x |d) represent the corpuslikelihoods conditioned on the latent variables.Here, to facilitate coherent topic production, inp(c | z), we penalize the likelihood of stopwordsto be generated from latent topics following Liet al. (2018). p(z) follows the standard normalprior N (0, I) and p(d) is the uniform distribu-tion Unif(0, 1). DKL refers to the Kullback-Leibler divergence that ensures the approximatedposteriors to be close to the true ones. For morederivation details, we refer readers to Miao et al.(2017).

Reconstructing target messages. From the la-tent variables z and d, the goal of our model isto reconstruct the target message x. The corre-sponding learning objective is to maximize Lxdefined as:

Lx = Eq(z | x)q(d | c)[log p(x | z,d)] (3)

Here we design Lx to ensure that the learnedlatent topics and discourse can reconstruct x.

Distinguishing Topics and Discourse. Ourmodel aims to distinguish word distributions fortopics (φT ) and discourse (φD), which enables topicsand discourse to capture different informationin conversations. Concretely, we use the mutualinformation, given below, to measure the mutualdependency between the latent topics z and thelatent discourse d.3

Eq(z)q(d)[log

p(z,d)p(z)p(d)

](4)

Equation 4 can be further derived as the Kullback-Leibler divergence of the conditional distribution,p(d | z), and marginal distribution, p(d). The de-rived formula, defined as the mutual information(MI) loss (LMI ) and shown in Equation 5, is usedto map z and d into the separated semantic space.

LMI = Eq(z)[DKL(p(d | z)||p(d))] (5)

We can hence minimize LMI for guiding ourmodel to separate word distributions that representtopics and discourse.

The Final Objective. To capture the joint ef-fects of the learning objectives described above(Lz , Ld, Lx, and LMI ), we design the final ob-jective function for our entire framework as thefollowing:

L = Lz + Ld + Lx − λLMI (6)

where the hyperparameter λ is the trade-offparameter for balancing between the MI loss(LMI ) and the other learning objectives. Bymaximizing the final objective L via back pro-pagation, the word distributions of topics anddiscourse can be jointly learned from microblogconversations.4

4 Experimental Setup

Data Collection. For our experiments, we col-lected two microblog conversation data sets fromTwitter. One is released by the TREC 2011microblog track (henceforth TREC), containingconversations concerning a wide range of topics.5

3The distributions in Equation 4 are all conditionalprobability distributions given the target message x and itsconversation c. We omit the conditions for simplicity.

4To smooth the gradients in implementation, for z ∼N (µ,σ), we apply the reparameterization on z (Kingma andWelling, 2013; Rezende et al., 2014), and for d ∼Multi(π),we adopt the Gumbel-Softmax trick (Maddison et al., 2016;Jang et al., 2016).

5http://trec.nist.gov/data/tweets/.

271

Data sets # of Avg msgs Avg words|Vocab|

convs per conv per msgTREC 116,612 3.95 11.38 9,463TWT16 29,502 8.67 14.70 7,544

Table 1: Statistics of the two data sets containingTwitter conversations.

The other is crawled from January to June 2016with Twitter streaming API6 (henceforth TWT16,short for Twitter 2016), following the way ofbuilding the TREC data set. During this period,there are a large volume of discussions centeredaround the U.S. presidential election. In addition,for both data sets, we apply Twitter search API7

to retrieve the missing tweets in the conversationhistory, as the Twitter streaming API (used tocollect both data sets) only returns sampled tweetsfrom the entire pool.

The statistics of the two experiment data setsare shown in Table 1. For model training andevaluation, we randomly sampled 80%, 10%, and10% of the data to form the training, development,and test set, respectively.

Data Preprocessing. We preprocessed the datawith the following steps. First, non-Englishtweets were filtered out. Then, hashtags, men-tions (@username), and links were replaced withgeneric tags ‘‘HASH’’, ‘‘MENT’’, and ‘‘URL’’,respectively. Next, the natural languge toolkit wasapplied for tweet tokenization.8 After that, allletters were normalized to lower cases. Finally,words that occurred fewer than 20 times werefiltered out from the data.

Parameter Setting. To ensure comparableresults with Li et al. (2018) (the prior workfocusing on the same task as ours), in the topiccoherence evaluation, we follow their setup toreport the results under two sets ofK (the numberof topics): K = 50 and K = 100, and with thenumber of discourse roles (D) set to 10. Theanalysis for the effects of K and D will be furtherpresented in Section 5.5. For all the other hyper-parameters, we tuned them on development set bygrid search. The trade-off parameter λ (defined

6https://developer.twitter.com/en/docs/tweets/filter-realtime/api-reference/post-statuses-filter.html.

7https://developer.twitter.com/en/docs/tweets/search/api-reference/get-savedsearches-show-id.

8https://www.nltk.org/.

in Equation 6), balancing the MI loss and theother objective functions, is set to 0.01. In modeltraining, we use the Adam optimizer (Kingma andBa, 2014) and run 100 epochs with early stopstrategy adopted.

Baselines. In topic modeling experiments, weconsider the five topic model baselines treatingeach tweet as a document: LDA (Blei et al., 2003),BTM (Yan et al., 2013), LF-LDA, LF-DMM(Nguyen et al., 2015), and NTM (Miao et al.,2017). In particular, BTM and LF-DMM are thestate-of-the-art topic models for short texts. BTMexplores the topics of all word pairs (biterms) ineach message to alleviate data sparsity in shorttexts. LF-DMM incorporates word embeddingspre-trained on external data to expand semanticmeanings of words, so does LF-LDA. In Nguyenet al. (2015), LF-DMM, based on one-topic-per-document Dirichlet Multinomial Mixture (DMM)(Nigam et al., 2000), was reported to performbetter than LF-LDA, based on LDA. For LF-LDAand LF-DMM, we use GloVe Twitter embeddings(Pennington et al., 2014) as the pre-trained wordembeddings.9

For the discourse modeling experiments, wecompare our results with LAED (Zhao et al.,2018), a VAE-based representation learning modelfor conversation discourse. In addition, for bothtopic and discourse evaluation, we compare withLi et al. (2018), a recently proposed model formicroblog conversations, where topics and dis-course are jointly explored with a non-neuralframework. Besides the existing models from pre-vious studies, we also compare with the variantsof our model that only models topics (hence-forth TOPIC ONLY) or discourse (henceforth DISC

ONLY).10 Our joint model of topics and discourseis referred to as TOPIC+DISC.

In the preprocessing procedure for the baselines,we removed stop words and punctuation for topicmodels unable to learn discourse representationsfollowing the common practice in previous work(Yan et al., 2013; Miao et al., 2017). For the othermodels, stop words and punctuation were retainedin the vocabulary, considering their usefulnessas discourse indicators (Li et al., 2018).

9https://nlp.stanford.edu/projects/glove/.10In our ablation without mutual information loss (LMI

defined in Equation 4), topics and discourse are learned inde-pendently. Thus, its topic representation can be used for theoutput of TOPIC ONLY, so does its discourse one for DISC ONLY.

272

Models K = 50 K = 100TREC TWT16 TREC TWT16

BaselinesLDA 0.467 0.454 0.467 0.454BTM 0.460 0.461 0.466 0.463LF-DMM 0.456 0.448 0.463 0.466LF-LDA 0.470 0.456 0.467 0.453NTM 0.478 0.479 0.482 0.443Li et al. (2018) 0.463 0.433 0.464 0.435Our modelsTOPIC ONLY 0.478 0.482 0.481 0.471TOPIC+DISC 0.485 0.487 0.496 0.480

Table 2: Cv coherence scores for latent topicsproduced by different models. The best result ineach column is highlighted in bold. Our jointmodel TOPIC+DISC achieves significantly bettercoherence scores than all the baselines (p < 0.01,paired test).

5 Experimental Results

In this section, we first report the topic coher-ence results in Section 5.1, followed by a dis-cussion in Section 5.2 comparing the latentdiscourse roles discovered by our model with themanually annotated dialogue acts. Then, we studywhether we can capture useful representations formicroblog messages in a tweet classification task(in Section 5.3). A qualitative analysis, showingsome example topics and discourse roles, is furtherprovided in Section 5.4. Finally, in Section 5.5,we provide more discussions on our model.

5.1 Topic Coherence

For the topic coherence, we adopt the Cv scoresmeasured via the open-source Palmetto toolkit asour evaluation metric.11 Cv scores assume thatthe top N words in a coherent topics (ranked bylikelihood) tend to co-occur in the same documentand have shown comparable evaluation results tohuman judgments (Roder et al., 2015). Table 2shows the average Cv scores over the producedtopics given N = 5 and N = 10. The valuesrange from 0.0 to 1.0, and higher scores indicatebetter topic coherence. We can observe that:

• Models assuming a single topic for each mes-sage do not work well. It has long been pointedout that the one-topic-per-message assumption(each message contains only one topic) helps

11https://github.com/dice-group/Palmetto.

topic models alleviate the data sparsity issue inshort texts on microblogs (Zhao et al., 2011;Quan et al., 2015; Nguyen et al., 2015; Li et al.,2018). However, we observe contradictory resultsbecause both LF-DMM and Li et al. (2018),following this assumption, achieve generally worseperformance than the other models. This mightbe attributed to the large-scale data used in ourexperiments (each data set has over 250K mes-sages as shown in Table 1), which potentiallyprovide richer word co-occurrence patterns andthus partially alleviate the data sparsity issue.

• Pre-trained word embeddings do not bring ben-efits. Comparing LF-LDA with LDA, we foundthat they result in similar coherence scores.This shows that with sufficiently large trainingdata, with or without using the pre-trained wordembeddings do not make any difference in thetopic coherence results.

• Neural models perform better than non-neuralbaselines. When comparing the results of neuralmodels (NTM and our models) with the otherbaselines, we find the former yield topics withbetter coherence scores in most cases.

• Modeling topics in conversations is effective.Among neural models, we found our modelsoutperform NTM (without exploiting conversa-tion contexts). This shows that the conversationsprovide useful context and enables more coherenttopics to be extracted from the entire conversationthread instead of a single short message.

• Modeling topics together with discourse helpsproduce more coherent topics. We can observebetter results with the joint model TOPIC+DISC incomparison with the variant considering topicsonly. This shows that TOPIC+DISC, via the jointmodeling of topic- and discourse-word distribu-tions (reflecting non-topic information), can bet-ter separate topical words from non-topical ones,hence resulting in more coherent topics.

5.2 Discourse Interpretability

In this section, we evaluate whether our modelcan discover meaningful discourse representa-tions. To this end, we train the comparison modelsfor discourse modeling on the TREC data set andtest the learned latent discourse on a benchmarkdata set released by Cerisara et al. (2018). Thebenchmark data set consists of 2,217 microblogmessages forming 505 conversations collected

273

Models Purity Homogeneity VIBaselinesLAED 0.505 0.022 6.418Li et al. (2018) 0.511 0.096 5.540Our modelsDISC ONLY 0.510 0.112 5.532TOPIC+DISC 0.521 0.142 5.097

Table 3: The purity, homogeneity, and variationof information (VI) scores for the latent discourseroles measured against the human-annotated dia-logue acts. For purity and homogeneity, higherscores indicate better performance, while for VIscores, lower is better. In each column, the bestresults are in boldface. Our joint model TOPIC+DISC

significantly outperforms all the baselines (p <0.01, paired t-test).

from Mastodon,12 a microblog platform exhibit-ing Twitter-like user behavior (Cerisara et al.,2018). For each message, there is a human-assigned discourse label, selected from one ofthe 15 dialogue acts, such as question, answer,disagreement, and so forth.

For discourse evaluation, we measure whetherthe model-produced discourse assignments areconsistent with the human-annotated dialogueacts. Hence, following Zhao et al. (2018), weassume that an interpretable latent discourserole should cluster messages labeled with thesame dialogue act. Therefore, we adopt purity(Manning et al., 2008), homogeneity (Rosenbergand Hirschberg, 2007), and variation of information (VI) (Meila, 2003; Goldwater and Griffiths,2007) as our automatic evaluation metrics. Here,we set D = 15 to ensure the number of latentdiscourse roles to be the same as the number ofmanually labeled dialogue acts. Table 3 shows thecomparison results of the average scores over the15 latent discourse roles. Higher values indicatebetter performance for purity and homogeneity,while for VI, lower is better.

It can be observed that our models exhibitgenerally better performance, showing the effec-tiveness of our framework in inducing inter-pretable discourse roles. Particularly, we observethe best results achieved by our joint modelTOPIC+DISC, which is learned to distinguish topic-and discourse-words, important in recognizingindicative words to reflect latent discourse.

12https://mastodon.social.

Figure 3: A heatmap showing the alignments of thelatent discourse roles and human-annotated dialogueact labels. Each line visualizes the distribution ofmessages with the corresponding dialogue act labelover varying discourse roles (indexed from 1 to 15),where darker colors indicate higher values.

To further analyze the consistency of varyinglatent discourse roles (produced by our TOPIC+DISC

model) with the human-labeled dialogue acts,Figure 3 displays a heatmap, where each linevisualizes how the messages with a dialogue actdistribute over varying discourse roles. It is seenthat among all dialogue acts, our model discov-ers more interpretable latent discourse for ‘‘greet-ings’’, ‘‘thanking’’, ‘‘exclamation’’, and ‘‘offer’’,where most messages are clustered into one ortwo dominant discourse roles. It may be becausethese dialogue acts can be relatively easier todetect based on their associated indicative words,such as the word ‘‘thanks’’ for ‘‘thanking’’, andthe word ‘‘wow’’ for ‘‘exclamation’’.

5.3 Message Representations

To further evaluate our ability to capture effec-tive representations for microblog messages, wetake tweet classification as an example and testthe classification performance with the topic anddiscourse representations as features. Here, theuser-generated hashtags capturing the topics ofonline messages are used as the proxy class labels(Li et al., 2016b; Zeng et al., 2018a). We construct

274

Models TREC TWT16Acc Avg F1 Acc Avg F1

BaselinesBoW 0.120 0.026 0.132 0.030TF-IDF 0.116 0.024 0.153 0.041LDA 0.128 0.041 0.146 0.046BTM 0.123 0.035 0.167 0.054LF-DMM 0.158 0.072 0.162 0.052NTM 0.138 0.042 0.186 0.068Our model 0.259 0.180 0.341 0.269

Table 4: Evaluation of tweet classification re-sults in accuracy (Acc) and average F1 (Avg F1).Representations learned by various models serveas the classification features. For our model, boththe topic and discourse representations are fedinto the classifier.

the classification data set from TREC and TWT16with the following steps. First, we removed thetweets without hashtags. Second, we ranked hash-tags by their frequencies. Third, we manuallyremoved the hashtags that are not topic-related(e.g. ‘‘#fb’’ for indicating the source of tweetsfrom Facebook), and combined the hashtags refer-ring to the same topic (e.g., ‘‘#DonaldTrump’’and ‘‘#Trump’’). Finally, we selected the top 50frequent hashtags, and all tweets containing thesehashtags as our classification data set. Here, wesimply use the support vector machines as theclassifier, since our focus is to compare the rep-resentations learned by various models. Li et al.(2018) are unable to produce vector representationon tweet level, hence not considered here.

Table 4 shows the classification results of accu-racy and average F1 on the two data sets withthe representations learned by various modelsserving as the classification features. We observethat our model outperforms other models with alarge margin. The possible reasons are twofold.First, our model derives topics from conversa-tion threads and thus potentially yields bettermessage representations. Second, the discourserepresentations (only produced by our model) areindicative features for hashtags, because userswill exhibit various discourse behaviors in dis-cussing diverse topics (hashtags). For instance, weobserve prominent ‘‘argument’’ discourse fromtweets with ‘‘#Trump’’ and ‘‘#Hillary’’, attributedto the controversial opinions to the two candidatesin the 2016 U.S. presidential election.

5.4 Example Topics and Discourse RolesWe have shown that joint modeling of topicsand discourse presents superior performance on aquantitative measure. In this section, we qualita-tively analyze the interpretability of our outputsvia analyzing the word distributions of someexample topics and discourse roles.

Example Topics. Table 5 lists the top 10 wordsof some example latent topics discovered by var-ious models from the TWT16 data set. Accordingto the words shown, we can interpret the extractedtopics as ‘‘gun control’’ — discussion about gunlaw and the failure of gun control in Chicago. Weobserve that LDA wrongly includes off-topic word‘‘flag’’. From the outputs of BTM, LF-DMM, Liet al. (2018), and our TOPIC ONLY variant, thoughwe do not find off-topic words, there are some non-topic words, such as ‘‘said’’ and ‘‘understand’’.13

The output of our TOPIC+DISC model appears to bethe most coherent, with words such as ‘‘firearm’’and ‘‘criminals’’ included, which are clearly rel-evant to ‘‘gun control’’. Such results indicate thebenefit of examining the conversation contexts andjointly exploring topics and discourse in them.

Example Discourse Roles. To qualitativelyanalyze whether our TOPIC+DISC model can dis-cover interpretable discourse roles, we select thetop 10 words from the distributions of some exam-ple discourse roles and list them in Table 6. It canbe observed that there are some meaningful wordclusters reflecting varying discourse roles foundwithout any supervision. Interestingly, we observethat the latent discourse roles from TREC andTWT16, though learned separately, exhibit somenotable overlap in their associated top 10 words,particularly for ‘‘question’’ and ‘‘statement’’. Wealso note that ‘‘argument’’ is represented by verydifferent words. The reason is that TWT16 con-tains a large volume of arguments centered aroundcandidates Clinton and Trump, resulting in the fre-quent appearance of words like ‘‘he’’ and ‘‘she’’.

5.5 Further DiscussionsIn this section, we further present more discussionson our joint model: TOPIC+DISC.

Parameter Analysis. Here, we study the twoimportant hyper-parameters in our model, the

13Non-topic words do not clearly indicate the correspond-ing topic, whereas off-topic words are more likely to appearin other topics.

275

LDA:::::people trump police violence gun death protest guns flag shot

BTM gun guns:::::people police wrong right

::::think law agree black

LF-DMM gun police black:::said

::::::people guns killing ppl amendment laws

Li et al. (2018) wrong don trump gun:::::::::understand laws agree guns

:::::doesn

::::make

NTM gun:::::::::understand

::yes guns world dead

:::real discrimination trump silence

TOPIC ONLY shootings gun guns cops charges control::::mass commit

::::know agreed

TOPIC+DISC guns gun shootings chicago shooting cops firearm criminals commit laws

Table 5: Top 10 representative words of example latent topics discovered from the TWT16 data set. Weinterpret the topics as ‘‘gun control’’ by the displayed words.

:::::::::Non-topic

::::::words are wave-underlined

and in blue, and off-topic words are underlined and in red.

Table 6: Top 10 representative words of example discourse roles learned from TREC and TWT16. Thediscourse roles of the word clusters are manually assigned according to their associated words.

number of topics (K) and the number of discourseroles (D). In Figure 4, we show the Cv topiccoherence given varying K in (a) and the homo-geneity measure given varyingD in (b). As can beseen, the curves corresponding to the performanceon topics and discourse are not monotonic. In par-ticular, better topic coherence scores are achievedgiven relatively larger topic numbers for TRECwith the best result observed at K = 80. On thecontrary, the optimum topic number for TWT16is K = 20, and increasing the number of topicsresults in worse Cv scores in general. This may beattributed to the relatively centralized topic con-cerning U.S. election in the TWT16 corpus. Fordiscourse homogeneity, the best result is achievedgivenD = 15, with same the number of manuallyannotated dialogue acts in the benchmark.

Case Study. To further understand why ourmodel learns meaningful representations for topics

and discourse, we present a case study basedon the example conversation shown in Figure 1.Specifically, we visualize the topic words (withp(w | z) > p(w |d)) in red and the rest of thewords in blue to indicate discourse. Darker redindicates the higher topic likelihood (p(w | z)) anddarker blue shows the higher discourse likelihood(p(w |d)). The results are shown in Figure 5. Wecan observe that topic and discourse words arewell separated by our model, which explains whyit can generate high-quality representations forboth topics and discourse.

Model Extensibility. Recall that in the Intro-duction, we mentioned that our neural-basedmodel has an advantage to be easily combinedwith other neural network architectures and allowsfor the joint training of both models. Here, wetake message classification (with the setup inSection 5.3) as an example, and study whether

276

Figure 4: (a) The impact of topic numbers. Thehorizontal axis shows the number of topics; the verticalaxis shows the Cv topic coherence. (b) The impact ofdiscourse numbers. The horizontal axis represents thenumber of discourse; the vertical axis represents thehomogeneity measure.

joint training our model with convolutional neuralnetwork (CNN) (Kim, 2014), the widely usedmodel on short text classification, can bring ben-efits to the classification performance. We set theembedding dimension to 200, with random initial-ization. The results are shown in Table 7, wherewe observe that joint training our model and theclassifier can successfully boost the classificationperformance.

Error Analysis. We further analyze the errorsin our outputs. For topics, taking a closer lookat their word distributions, we found that ourmodel sometimes mixes sentiment words withtopic words. For example, among the top 10 wordsof a topic ‘‘win people illegal americans hate ltracism social tax wrong’’, there are words ‘‘hate’’and ‘‘wrong’’, expressing sentiment rather thanconveying topic-related information. This is dueto the prominent co-occurrences of topic wordsand sentiment words in our data, which results inthe similar distributions for topics and sentiment.Future work could focus on the further separationof sentiment and topic words.

For discourse, we found that our model caninduce some discourse roles beyond the 15 man-ually defined dialogue acts in the Mastodon dataset (Cerisara et al., 2018). For example, as shownin Table 6, our model discovers the ‘‘quotation’’discourse from both TREC and TWT16, whichis, however, not defined in the Mastodon data set.This perhaps should not be considered as an error.We argue that it is not sensible to pre-define afixed set of dialogue acts for diverse microblogconversations due to the rapid change and awide variety of user behaviors in social media.Therefore, future work should involve a betteralternative to evaluate the latent discourse without

Figure 5: Visualization of the topic-discourse assign-ment of a twitter conversion from TWT16. The anno-tated blue words are prone to be discourse words,and the red are topic words. The shade indicates theconfidence of the current assignment.

relying on manually defined dialogue acts. Wealso notice that our model sometimes fails to iden-tify discourse behaviors requiring more in-depthsemantic understanding, such as sarcasm, irony,and humor. This is because our model detectslatent discourse purely based on the observedwords, whereas the detection of sarcasm, irony, orhumor requires deeper language understanding,which is beyond the capacity of our model.

6 Conclusion and Future Work

We have presented a neural framework that jointlyexplores topic and discourse from microblogconversations. Our model, in an unsupervisedmanner, examines the conversation contexts anddiscovers word distributions that reflect latenttopics and discourse roles. Results from extensiveexperiments show that our model can generatecoherent topics and meaningful discourse roles.In addition, our model can be easily combinedwith other neural network architectures (suchas CNN) and allows for joint training, whichhas presented better message classification resultscompared with the pipeline approach without jointtraining.

Our model captures topic and discourse repre-sentations embedded in conversations. They arepotentially useful for a broad range of down-stream applications, worthy to be explored infuture research. For example, our model is usefulfor developing social chatbots (Zhou et al., 2018).By explicitly modeling ‘‘what you say’’ and ‘‘howyou say’’, our model can be adapted to track the

277

Models TREC TWT16Acc Avg F1 Acc Avg F1

CNN only 0.199 0.167 0.334 0.311Separate-Train 0.284 0.270 0.391 0.390Joint-Train 0.297 0.286 0.428 0.413

Table 7: Accuracy (Acc) and average F1 (Avg F1)on tweet classification (hashtags as labels). CNNonly: CNN without using our representations.Seperate-Train: CNN fed with our pre-trainedrepresentations. Joint-Train: Joint training CNNand our model.

change of topics in conversation context, helpfulto determine ‘‘what to say and how to say’’ inthe next turn. Also, it would be interesting tostudy how our learned latent topics and discourseaffect recommendation (Zeng et al., 2018b) andsummarization of microblog conversations (Liet al., 2018).

Acknowledgments

This work is partially supported by the ResearchGrants Council of the Hong Kong Special Admin-istrative Region, China (No. CUHK 14208815 andNo.CUHK14210717of theGeneralResearchFund),Innovate UK (grant no. 103652), and MicrosoftResearch Asia (2018 Microsoft Research Asia Col-laborative Research Award). We thank ShumingShi, Dong Yu, and TACL reviewers and editorsfor the insightful suggestions on various aspectsof this work.

References

Stergos D. Afantenos, Eric Kow, Nicholas Asher,and Jeremy Perret. 2015. Discourse parsing formulti-party chat dialogues. In Proceedings ofthe 2015 Conference on Empirical Methods inNatural Language Processing, EMNLP 2015,pages 928–937. Lisbon.

David Alvarez-Melis and Martin Saveski. 2016.Topic modeling in twitter: Aggregating tweetsby conversations. In Proceedings of the TenthInternational Conference on Web and SocialMedia, pages 519–522. Cologne.

Eytan Bakshy, Solomon Messing, and Lada A.Adamic. 2015. Exposure to ideologically di-verse news and opinion on facebook. Science,348(6239):1130–1132.

David M. Blei, Thomas L. Griffiths, Michael I.Jordan, and Joshua B. Tenenbaum. 2003. Hier-archical topic models and the nested chineserestaurant process. In Advances in Neural In-formation Processing Systems, NIPS 2003,pages 17–24. Vancouver and Whistler.

David M. Blei, Alp Kucukelbir, and Jon D.McAuliffe. 2016. Variational inference: A re-view for statisticians. CoRR, abs/1601.00670.

David M. Blei, Andrew Y. Ng, and Michael I.Jordan. 2001. Latent Dirichlet allocation. InAdvances in Neural Information ProcessingSystems 14, pages 601–608. Vancouver.

Christophe Cerisara, Somayeh Jafaritazehjani,Adedayo Oluokun, and Hoa T. Le. 2018. Multi-task dialog act and sentiment recognition onmastodon. In Proceedings of the 27th Interna-tional Conference on Computational Linguis-tics, COLING 2018, pages 745–754. Santa Fe,NM.

Micha Elsner and Eugene Charniak. 2008. Youtalking to me? A corpus and algorithm forconversation disentanglement. In Proceed-ings of the 46th Annual Meeting of the Asso-ciation for Computational Linguistics, ACL 2008,pages 834–842. Columbus, OH.

Micha Elsner and Eugene Charniak. 2010. Dis-entangling chat. Computational Linguistics, 36(3):389–409.

Sharon Goldwater and Thomas L. Griffiths. 2007.A fully Bayesian approach to unsupervisedpart-of-speech tagging. In Proceedings of the45th Annual Meeting of the Association forComputational Linguistics. Prague.

Thomas Hofmann. 1999. Probabilistic latent se-mantic indexing. In Proceedings of the 22ndAnnual International ACM SIGIR Conferenceon Research and Development in InformationRetrieval, pages 50–57. Berkeley, CA.

Liangjie Hong and Brian D. Davison. 2010. Em-pirical study of topic modeling in Twitter. InProceedings of the 3rd Workshop on SocialNetwork Mining and Analysis, SNAKDD 2009,pages 80–88. Paris.

Zhiting Hu, Gang Luo, Mrinmaya Sachan, Eric P.Xing, and Zaiqing Nie. 2016. Grounding topic

278

models with knowledge bases. In Proceedingsof the Twenty-Fifth International Joint Con-ference on Artificial Intelligence, IJCAI 2016,pages 1578–1584. New York, NY.

Eric Jang, Shixiang Gu, and Ben Poole. 2016.Categorical reparameterization with gumbel-softmax. CoRR, abs/1611.01144.

Yangfeng Ji, Gholamreza Haffari, and JacobEisenstein. 2016. A latent variable recurrentneural network for discourse-driven languagemodels. In Proceedings of the 2016 Conferenceof the North American Chapter of the Asso-ciation for Computational Linguistics: HumanLanguage Technologies, NAACL HLT 2016,pages 332–342. San Diego, CA.

Yunhao Jiao, Cheng Li, Fei Wu, and Qiaozhu Mei.2018. Find the conversation killers: A predictivestudy of thread-ending posts. In Proceedingsof the 2018 World Wide Web Conference onWorld Wide Web, WWW 2018, pages 1145–1154.Lyon.

Shafiq R. Joty, Giuseppe Carenini, and Chin-YewLin. 2011. Unsupervised modeling of dialog actsin asynchronous conversations. In Proceedingsof the 22nd International Joint Conference on Arti-ficial Intelligence, IJCAI 2011, pages 1807–1813.Barcelona.

Yoon Kim. 2014. Convolutional neural networksfor sentence classification. In Proceedings ofthe 2014 Conference on Empirical Methods inNatural Language Processing, EMNLP 2014,A meeting of SIGDAT, a Special Interest Groupof the ACL, pages 1746–1751. Doha.

Diederik P. Kingma and Jimmy Ba. 2014. Adam:A method for stochastic optimization. CoRR,abs/1412.6980.

Diederik P. Kingma and Max Welling. 2013. Auto-encoding variational Bayes. CoRR, abs/1312.6114.

Chenliang Li, Haoran Wang, Zhiqian Zhang,Aixin Sun, and Zongyang Ma. 2016a. Topicmodeling for short texts with auxiliary wordembeddings. In Proceedings of the 39th Inter-national ACM SIGIR conference on Researchand Development in Information Retrieval,SIGIR 2016, pages 165–174. Pisa.

Jing Li, Ming Liao, Wei Gao, Yulan He, andKam-Fai Wong. 2016b. Topic extraction frommicroblog posts using conversation structures.In Proceedings of the 54th Annual Meeting ofthe Association for Computational Linguistics,ACL 2016, Volume 1: Long Papers. Berlin.

Jing Li, Yan Song, Zhongyu Wei, and Kam-FaiWong. 2018. A joint model of conversationaldiscourse and latent topics on microblogs.Computational Linguistics, 44(4):719–754.

Jiwei Li, Thang Luong, Dan Jurafsky, andEduard H. Hovy. 2015. When are tree structuresnecessary for deep learning of representations?In Proceedings of the 2015 Conference on Em-pirical Methods in Natural Language Process-ing, EMNLP 2015, pages 2304–2314. Lisbon.

Chris J. Maddison, Andriy Mnih, and Yee WhyeTeh. 2016. The concrete distribution: A contin-uous relaxation of discrete random variables.CoRR, abs/1611.00712.

Christopher D. Manning, Prabhakar Raghavan,and Hinrich Schutze. 2008. Introduction toInformation Retrieval. Cambridge UniversityPress.

Rishabh Mehrotra, Scott Sanner, Wray L. Buntine,and Lexing Xie. 2013. Improving LDA topicmodels for microblogs via tweet pooling andautomatic labeling. In The 36th InternationalACM SIGIR Conference on Research andDevelopment in Information Retrieval, SIGIR’13, pages 889–892. Dublin.

Marina Meila. 2003. Comparing clusterings by thevariation of information. In Computational Learn-ing Theory and Kernel Machines, 16th AnnualConference on Computational Learning Theoryand 7th Kernel Workshop, COLT/Kernel,Proceedings, pages 173–187. Washington, DC.

Yishu Miao, Edward Grefenstette, and PhilBlunsom. 2017. Discovering discrete latenttopics with neural variational inference. InProceedings of the 34th International Con-ference on Machine Learning, ICML 2017,pages 2410–2419. Sydney.

Vinod Nair and Geoffrey E. Hinton. 2010. Rectifiedlinear units improve restricted Boltzmann ma-chines. In Proceedings of the 27th International

279

Conference on Machine Learning (ICML 2010),pages 807–814. Haifa.

Dat Quoc Nguyen, Richard Billingsley, Lan Du,and Mark Johnson. 2015. Improving topicmodels with latent feature word represen-tations. Transactions of the Association forComputational Linguistics, TACL, 3:299–313.

Kamal Nigam, Andrew McCallum, Sebastian Thrun,and Tom M. Mitchell. 2000. Text classificationfrom labeled and unlabeled documents usingEM. Machine Learning, 39(2/3):103–134.

Jeffrey Pennington, Richard Socher, and ChristopherD. Manning. 2014. GloVe: Global vectors forword representation. In Proceedings of the2014 Conference on Empirical Methods inNatural Language Processing, EMNLP 2014,pages 1532–1543. Doha.

Kechen Qin, Lu Wang, and Joseph Kim. 2017.Joint modeling of content and discourse rela-tions in dialogues. In Proceedings of the 55thAnnual Meeting of the Association for Com-putational Linguistics, ACL 2017, Volume 1:Long Papers, pages 974–984. Vancouver.

Xiaojun Quan, Chunyu Kit, Yong Ge, and SinnoJialin Pan. 2015. Short and sparse text topicmodeling via self-aggregation. In Proceedingsof the Twenty-Fourth International Joint Con-ference on Artificial Intelligence, IJCAI 2015,pages 2270–2276. Buenos Aires.

Daniel Ramage, Susan T. Dumais, and Daniel J.Liebling. 2010. Characterizing microblogs withtopic models. In Proceedings of the FourthInternational Conference on Weblogs and So-cial Media, ICWSM 2010, Washington, DC.

Danilo Jimenez Rezende, Shakir Mohamed,and Daan Wierstra. 2014. Stochastic back-propagation and approximate inference in deepgenerative models. In Proceedings of the 31thInternational Conference on Machine Learn-ing, ICML 2014, pages 1278–1286. Beijing.

Alan Ritter, Colin Cherry, and Bill Dolan. 2010.Unsupervised modeling of twitter conversa-tions. In Human Language Technologies: Con-ference of the North American Chapter of theAssociation of Computational Linguistics, Pro-ceedings, pages 172–180. Los Angeles, CA.

Michael Roder, Andreas Both, and AlexanderHinneburg. 2015. Exploring the Space ofTopic Coherence Measures. In Proceedings ofthe Eighth ACM International Conference onWeb Search and Data Mining, WSDM 2015,pages 399–408. Shanghai.

Michal Rosen-Zvi, Thomas L. Griffiths, MarkSteyvers, and Padhraic Smyth. 2004. Theauthor-topic model for authors and documents.In UAI ’04, Proceedings of the 20th Confer-ence Uncertainty in Artificial Intelligence,pages 487–494. Banff.

Andrew Rosenberg and Julia Hirschberg. 2007.V-measure: A conditional entropy-based exter-nal cluster evaluation measure. In Proceed-ings of the 2007 Joint Conference on EmpiricalMethods in Natural Language Processing andComputational Natural Language Learning,EMNLP-CoNLL 2007, pages 410–420. Prague.

Bei Shi, Wai Lam, Shoaib Jameel, StevenSchockaert, and Kwun Ping Lai. 2017. Jointlylearning word embeddings and latent topics.In Proceedings of the 40th International ACMSIGIR Conference on Research and Develop-ment in Information Retrieval, pages 375–384.Tokyo.

Yangqiu Song, Haixun Wang, Zhongyuan Wang,Hongsong Li, and Weizhu Chen. 2011. Shorttext conceptualization using a probabilisticknowledgebase. In Proceedings of the 22ndInternational Joint Conference on ArtificialIntelligence, IJCAI 2011, pages 2330–2336.Barcelona.

Akash Srivastava and Charles Sutton. 2017. Auto-encoding variational inference for topic models.In Proceedings of the Fifth InternationalConference on Learning Representations, ICLR2017. Toulon.

Andreas Stolcke, Noah Coccaro, Rebecca Bates,Paul Taylor, Carol Van Ess-Dykema, KlausRies, Elizabeth Shriberg, Daniel Jurafsky,Rachel Martin, and Marie Meteer. 2000. Dia-logue act modeling for automatic tagging andrecognition of conversational speech. Compu-tational Linguistics, 26(3).

Xiaohui Yan, Jiafeng Guo, Yanyan Lan, andXueqi Cheng. 2013. A biterm topic model for

280

short texts. In 22nd International World WideWeb Conference, WWW ’13, pages 1445–1456.Rio de Janeiro.

Yi Yang, Doug Downey, and Jordan L. Boyd-Graber. 2015. Efficient methods for incor-porating knowledge into topic models. InProceedings of the 2015 Conference on Em-pirical Methods in Natural Language Pro-cessing, EMNLP 2015, pages 308–317. Lisbon.

Elina Zarisheva and Tatjana Scheffler. 2015. Dia-log act annotation for twitter conversations. InProceedings of the SIGDIAL 2015 Conference,pages 114–123. Prague.

Jichuan Zeng, Jing Li, Yan Song, Cuiyun Gao,Michael R. Lyu, and Irwin King. 2018a. Topicmemory networks for short text classification.In Proceedings of the 2018 Conference onEmpirical Methods in Natural LanguageProcessing, EMNLP 2018. Brussels.

Xingshan Zeng, Jing Li, Lu Wang, NicholasBeauchamp, Sarah Shugars, and Kam-FaiWong. 2018b. Microblog conversation recom-mendation via joint modeling of topics anddiscourse. In Proceedings of the 2018 Con-ference of the North American Chapter ofthe Association for Computational Linguistics:Human Language Technologies, NAACL-HLT

2018, Volume 1 (Long Papers), pages 375–385.New Orleans, LA.

Tiancheng Zhao, Kyusong Lee, and MaxineEskenazi. 2018. Unsupervised discrete sentencerepresentation learning for interpretable neuraldialog generation. In Proceedings of the 56thAnnual Meeting of the Association for Com-putational Linguistics, ACL 2018, Volume 1:Long Papers, pages 1098–1107. Melbourne.

Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi.2017. Learning discourse-level diversity forneural dialog models using conditional varia-tional autoencoders. In Proceedings of the 55thAnnual Meeting of the Association for Com-putational Linguistics, ACL 2017, Volume 1:Long Papers, pages 654–664. Vancouver.

Wayne Xin Zhao, Jing Jiang, Jianshu Weng, JingHe, Ee-Peng Lim, Hongfei Yan, and XiaomingLi. 2011. Comparing Twitter and traditionalmedia using topic models. In Proceedingsof Advances in Information Retrieval - 33rdEuropean Conference on IR Research, ECIR2011, pages 338–349. Dublin.

Li Zhou, Jianfeng Gao, Di Li, and Heung-YeungShum. 2018. The design and implementation ofXiaoice, an empathetic social chatbot. CoRR,abs/1812.08989.

281