Multi-Task Learning with Multi-View Attention for Answer Selectionand Knowledge Base Question Answering

Yang Deng1, Yuexiang Xie1, Yaliang Li2, Min Yang3, Nan Du2, Wei Fan2, Kai Lei1∗, Ying Shen1∗∗Corresponding Author

1Shenzhen Key Lab for Information Centric Networking & Blockchain Technology (ICNLAB),School of Electronics and Computer Engineering, Peking University Shenzhen Graduate School

2Tencent Medical AI Lab 3Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences{ydeng,xieyx}@pku.edu.cn, yaliangli@tencent.com, min.yang@siat.ac.cn,

{ndu,davidwfan}@tencent.com, {leik,shenying}@pkusz.edu.cn

AbstractAnswer selection and knowledge base question answering(KBQA) are two important tasks of question answering (QA)systems. Existing methods solve these two tasks separately,which requires large number of repetitive work and neglectsthe rich correlation information between tasks. In this paper,we tackle answer selection and KBQA tasks simultaneouslyvia multi-task learning (MTL), motivated by the followingmotivations. First, both answer selection and KBQA can beregarded as a ranking problem, with one at text-level whilethe other at knowledge-level. Second, these two tasks canbenefit each other: answer selection can incorporate the ex-ternal knowledge from knowledge base (KB), while KBQAcan be improved by learning contextual information from an-swer selection. To fulfill the goal of jointly learning these twotasks, we propose a novel multi-task learning scheme that uti-lizes multi-view attention learned from various perspectivesto enable these tasks to interact with each other as well aslearn more comprehensive sentence representations. The ex-periments conducted on several real-world datasets demon-strate the effectiveness of the proposed method, and the per-formance of answer selection and KBQA is improved. Also,the multi-view attention scheme is proved to be effective inassembling attentive information from different representa-tional perspectives.

IntroductionQuestion answering (QA) is an important but challeng-ing NLP application. Nowadays, there are many possiblesources of data for the QA systems, such as web docu-ments, QA communities, knowledge bases and so on. Ac-cording to these data sources, question answering can be di-vided into several different tasks, including machine read-ing comprehension (Rajpurkar et al. 2016), answer selec-tion (Wang, Smith, and Mitamura 2007), knowledge basequestion answering (Bordes et al. 2015) and more. Recentyears have witnessed many successes in applying deep neu-ral networks on these QA tasks (dos Santos et al. 2016;Wenpeng et al. 2016; Hao et al. 2017).

Despite the advancement of these models, different QAtasks are still solved separately. Designing and training var-ious models for specific tasks are time-consuming and ex-pensive. Recently, as seen in many other NLP tasks, multi-

Copyright c© 2019, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

task learning has been extensively studied to learn multiplerelated tasks simultaneously. The applications of MTL areextensive in NLP domain, such as text classification (Zheng,Chen, and Qiu 2018), sequence labeling (Chen et al. 2018)and text summarization (Guo, Pasunuru, and Bansal 2018),etc. However, applying MTL on QA has received little at-tention.

In this work, we explore multi-task learning approachesto tackle answer selection and knowledge base question an-swering at the same time, with the intuition that these tasksboth can be regarded as a ranking problem, but one in text-level and the other in knowledge-level. Specifically, the taskof answer selection aims to pick out the correct answersfor the given question from a set of candidate answer sen-tences, while the task of KBQA focuses on extracting thecorresponding facts from KB, such as Freebase (Bollackeret al. 2008), to answer the given question. Besides, previ-ous works prove that answer selection task can be benefitedfrom external knowledge (Savenkov and Agichtein 2017;Deng et al. 2018; Ying et al. 2018), while incorporatingtext information also enhances the performance of KBQAtask (Yu et al. 2017; Sorokin and Gurevych 2018).

Most existing multi-task learning schemes divide the lay-ers of a model into task-specific and shared layers (Abhishekand Hal 2012; Guo, Pasunuru, and Bansal 2018). The sharedlayers are shared across all tasks, while the task-specific lay-ers are separate for each task. However, these methods ne-glect the interrelation between the task-specific layers andthe shared layers, and the interaction among different tasks.Thus, we present a novel multi-task learning scheme to learnmulti-view attentions from different aspects, which enablesdifferent tasks to interact with each other. Concretely, we as-semble the attentive information from the task-specific lay-ers to learn more comprehensive sentence representationsin the shared layers. In addition, the multi-view attentionmechanism enhances the sentence representational learningby combining word-level and knowledge-level information.That is to say, the attentive information in both word-leveland knowledge-level are shared and transferred among dif-ferent tasks by using the multi-view attention scheme.

To demonstrate the effectiveness of the proposed method,we conduct experiments on both answer selection andKBQA datasets. Empirically, joint learning of answer se-lection and KBQA tasks significantly improves the perfor-

arX

iv:1

812.

0235

4v1

[cs

.CL

] 6

Dec

201

8

Answer Selection KBQA

word Q what was johnny appleseed ’s real name ? what is the name of a track created by katy perry ?A john chapman , aka american folk hero johnny appleseed . katy perry music artist track witness

knowledge Q johnny appleseed katy perryA john chapman, johnny appleseed katy perry, music.artist.track, witness

Table 1: Examples of Answer Selection and KBQA Data

mance of each task compared to learning them indepen-dently. The experimental results also indicate the effective-ness of the multi-view attention scheme and each views at-tention contributes.

In summary, our main contributions are as follows:• We explore multi-task learning approaches for answer

selection and knowledge base question answering. An-swer selection task can be improved by KBQA task inknowledge-level, while KBQA task can be enhanced byanswer selection task in word-level.

• We propose a novel multi-task learning scheme that lever-ages multi-view attention mechanism to bridge differ-ent tasks, which integrates the important information ofthe task-specific layers into the shared layers as well asenables the model to interactively learn word-level andknowledge-level representations.

• Experimental results show that multi-task learning of an-swer selection and KBQA outperforms state-of-the-artsingle-task learning methods. Besides, the multi-view at-tention based MTL scheme further enhance the perfor-mance.

Multi-Task Learning for Question AnsweringIn this section, we introduce the multi-task learning of an-swer selection and knowledge base question answering.

Problem DefinitionThe tasks of answer selection and KBQA can be typicallyregarded as a ranking problem. Given a question qi ∈ Q,the task is to rank a set of candidate answer sentences orfacts ai ∈ A. Specifically, a function f(q, a) that computesa relevancy score f(q, a) ∈ [0, 1] for each question-answerpair.

The multi-task learning of answer selection and KBQAstarts with the entity linking results. As the example shownin Table 1, a word sequence W = {w1, w2, ..., wL} and aknowledge sequence K = {k1, k2, ..., kL} are prepared foreach question and each candidate answer. For the questionand answer in answer selection and the question in KBQA,we derive the knowledge of the sentence by entity link-ing (Savenkov and Agichtein 2017). For the answer fact inKBQA, we obtain the word sequence from the tokenized en-tity name and relation name (Yu et al. 2017).

In the multi-task scenario, we aim to rank the candidateanswers for each question from T related tasks. We refer Dt

as the t-th preprocessed task dataset with N samples:

Dt = {(W (t)qi ,K

(t)qi ,W

(t)ai,K(t)

ai, Y

(t)i )}Nt

i=1, (1)

where Y (t)i denotes the label of i-th QA pair in t-th task.

KBQA

Encoder

KBQA

Encoder

AS

Encoder

AS

Encoder

𝑥

KBQA

Softmax

AS

Softmax

Ta

sk

-Sp

ec

ific

En

co

der

La

yer

Sh

are

d

Rep

rese

nta

tio

n

Le

arn

ing

La

yer

Sh

are

d

Jo

int

Lay

er

Ta

sk

-Sp

ec

ific

So

ftm

ax

La

ye

r

Figure 1: Basic Multi-Task QA Network

Multi-Task Question Answering NetworkThe basic multi-task learning model is a deep neural net-work that adopts layer-specific sharing mechanism (Guo,Pasunuru, and Bansal 2018) which shares some high-levelinformation across different tasks and the remained layersare parallel and independent to learn task-specific low-levelinformation. Figure 1 illustrates the overall architecture ofmulti-task QA network (MTQA-net) for answer selection(AS) and knowledge base question answering (KBQA).

Task-specific Encoder Layer The preprocessed sen-tences are first encoded into distributed vector representa-tions. Different QA tasks are supposed to be diverse in datadistributions and low-level representations. Therefore, eachtask is equipped with a task-specific siamese encoder forboth questions and answers, and each task-specific encodercontains a word encoder and a knowledge encoder to learnthe integral sentence representations, as shown in Figure 2.

Word Encoder. The input of the word encoder module isa sequence of word embeddings EW = {ew1

, ew2, ..., ewL

}.We employ Bi-LSTM (Bidirectional Long Short-TermMemory Networks) to capture the context information fromboth the head-to-tail and the tail-to-head context. The outputat l-th word is represented by hl = [

−→hl :

←−hl ], in which

−→hl

is the output of the forward network and←−hl is that of the

backward network. Given a pair of word sequences of thequestion q and the answer a, we generate the word-basedsentence representation HW ∈ RL×dh for both the questionand the answer, where L and dh are the length of sentencesand the size of hidden units:

HWq= Bi-LSTM(EWq

);HWa= Bi-LSTM(EWa

). (2)

cat

E

mb

ed

din

g

La

yer

En

co

der

La

yer

Word-Level Knowledge-Level

En

co

ded

Sen

ten

ce

Rep

rese

nta

tio

n

𝑤1 𝑤2 𝑤𝑙

𝑘1 𝑘2 𝑘𝑙

Figure 2: Task-specific Encoder Layer

Knowledge Encoder. Different from word encoder, a se-quence of knowledge embeddings EK = {ek1

, ek2, ..., ekL

}is input into knowledge encoder. As the knowledge sequenceis composed by a series of tokenized entity or relation name,the high-level knowledge-based representations are desiredfor the latter learning procedure. We deal with this problemby applying CNN on the knowledge sequences, in which fil-ters of size n slide over the knowledge embedding matrixto capture the local n-gram features. Each move computes ahidden layer vector as

xl =[ekl−n−1

2

, . . . , ekl, . . . , ek

l+n−12

], (3)

hl = tanh (Wcxl + bc) , (4)

where Wc and bc are the convolution kernel and the biasvector to be learned.

Due to the uncertainty of the length of entities, a coupleof filters of various sizes are employed to obtain differentoutput vectors

{H(1), H(2), . . . ,H(n)

}, whereH(i) denotes

the output vector obtained by the i-th filter. We pass theseoutput vectors through a fully-connected layer to get theknowledge-based sentence representation HK ∈ RL×df ,where L is the length of the sentence and df is the totalfilter sizes of CNN. Given the question q and the answer a,the knowledge-based sentence representations are:

HKq= [H

(1)Kq

: H(2)Kq

: . . . : H(n)Kq

], (5)

HKa= [H

(1)Ka

: H(2)Ka

: . . . : H(n)Ka

]. (6)

After obtaining the word-based and the knowledge-basedsentence representations, HKq

and HKaare still vectors in

the order of words in the sentence, since the fully-connectedlayer concatenates the outputs from all the filters in the di-mension of the feature instead of the sequence. Thus, theorder of HKq

and HKais consistent with HWq

and HWa.

Then we concatenate them into the encoded sentence repre-sentations, Hq = [HWq : HKq ] and Ha = [HWa : HKa ].

Shared Representation Learning Layer After encodingsentence into vector representations with task-specific en-coder, we share high-level information across different tasks

via a shared representation learning layer. Compared withthe input of task-specific encoder layer, the integral sentencerepresentation contains richer semantic meaning and sharemore similar distributions with other tasks. Therefore, weintegrate the encoded vectors from all the tasks and passthrough a high-level shared Siamese Bi-LSTM to generatethe final QA representations:

Sq = Bi-LSTM(Hq); Sa = Bi-LSTM(Ha). (7)We apply average pooling over the Bi-LSTM output,

sq = Average(Sq), sa = Average(Sa). Inspired by pre-vious works (Severyn and Moschitti 2015) and (Tay et al.2017), we incorporate some word and knowledge overlapfeatures xol ∈ R6 to form the final feature space for bi-nary classification, including word overlap score, non-stopword overlap score, weighted word overlap score, non-stopweighted word overlap score, knowledge overlap score andweighted knowledge overlap score. Thus, the final featurespace will be x = [sq, sa, xol].

Task-specific Softmax Layer For a question-answer pair,q(t)i and a(t)i , and its label y(t)i in k-th task, the final feature

representations is fed into the task-specific softmax layer forbinary classification:

p(t) = softmax(W (t)

s x+ b(t)s

), (8)

where p(t) is the predicted probability, W (t)s ∈ Rdx×2 and

b(t)s ∈ R2 are the task-specific weight matrix and bias vector

in the hidden layer.

Multi-Task LearningThe overall multi-task learning model is trained to minimizethe cross-entropy loss function:

L = −T∑

t=1

λt

Nt∑i=1

[y(t)i log p

(t)i +

(1− y

(t)i

)log(1− p

(t)i

)],

(9)where λt is a parameter that determines the weight of t-thtask, and y(t)i is the ground-truth label of question-answerpair (q(t)i , a

(t)i ). In practice, the same weight is assigned to

all tasks.

Multi-Task Model with Multi-View AttentionIn order to enhance the interaction between different QAtasks in latent representation space, we propose a multi-viewattention mechanism to fetch the important information fromtask-specific layers as well as the shared layers.

Multi-View Attention SchemeAs shown in the Figure 3, different with other attentionsharing scheme, we not only draw attention from the task-specific layers, but also combine the information from theshared layers. In addition, we obtain the attentive infor-mation from both word and knowledge perspectives, sinceword-level and knowledge-level information may make ajoint contribution to representational learning. Specifically,we compute five views of attention, including the views ofword, knowledge, semantic, knowledge semantic and co-attention.

𝑘1 𝑘2 𝑘𝑙

𝑤1 𝑤2 𝑤𝑙

𝑘1 𝑘2 𝑘𝑙

𝑤1 𝑤2 𝑤𝑙

Att4

Att2

Att3

Att1

Multi-View

Attention

Att5

𝑥

Multi-View

Attention

Word View Knowledge View

Semantic View Knowledge Semantic View

Co-attention View

Figure 3: Multi-View Attention

Word View & Knowledge View We first adopt a two-wayattention mechanism (dos Santos et al. 2016) to interactivelymodeling a pair of QA sentences, by attending correspond-ingly important information in two sentences. We collect theattention weights from both the word and the knowledge em-beddings:

MW = tanh(ET

WqUWEWa

);MK = tanh

(ET

KqUKEKa

),

(10)where UW ∈ Rdew×dew and UK ∈ Rdek

×dek are parametermatrices to be learned; dew and dek are the dimensions ofword embeddings and knowledge embeddings.

Then we apply max-pooling over MW and MK row-wiseand column-wise to obtain the word view and the knowledgeview attention weights for the question and the answer:α(1)q = softmax(Max(MW ));α(1)

a = softmax(Max(MWT )),

(11)

α(2)q = softmax(Max(MK));α(2)

a = softmax(Max(MKT )),

(12)

where α(1)q and α(1)

a are the attention weights for the ques-tion and the answer from word view; α(2)

q and α(2)a are the

attention weights from knowledge view.

Semantic View & Knowledge Semantic View As thecontext semantic information is of great importance in sen-tence representational learning, we exploit the overall con-text semantic information as the attention source over all theelements in the sentence. Thus, we apply max/mean poolingover the output of the task-specific encoder layer to obtainthe overall semantic information of a sentence. We experi-mented on using either max or mean pooling to generate thesemantic information. The result shows that the followingpooling methods achieve the best performance:owq

= Average(HWq

); owa

= Average (HWa) , (13)

okq= Max

(HKq

); oka

= Max (HKa) , (14)

Conceptually, the attention mechanism takes into consid-eration the semantic information, which is expected to cap-ture the correlations between question words and answerwords:

α(3)q = softmax(wT

wqtanh(Wwaowa +WwqHWq )), (15)

α(3)a = softmax(wT

watanh(Wwqowq +WwaHWa)), (16)

α(4)q = softmax(wT

kqtanh(Wkaoka +WkqHKq )), (17)

α(4)a = softmax(wT

katanh(Wkqokq +WkaHKa)), (18)

where Wwq ,Wwa ∈ Rdh×dh , wwq , wwa ∈ Rdh ,Wkq ,Wka ∈ Rdf×df , wkq , wka ∈ Rdf are attention param-eters to be learned; α(3)

q and α(3)a are the attention weights

for the question and the answer from semantic view; α(4)q

and α(4)a are the attention weights from knowledge semantic

view.

Co-attention View Similar to word view and knowledgeview attention, we employ a two-way attention mechanismto generate the co-attention between final question and an-swer representations:

Mco = tanh(STq USSa

), (19)

α(5)q = softmax(Max(Mco)), (20)

α(5)a = softmax(Max(Mco

T )), (21)

where UW ∈ Rds×ds is the attention parameter matrix tobe learned; ds is the dimension of final QA representations;α(5)q and α(5)

a are the co-attention weights for the questionand the answer.

Multi-View Attentive Representation We define themulti-view attention fusion from word, knowledge, seman-tic, knowledge semantic and co-attention views as:

αq = softmax5∑

i=1

λ(i)q α(i)

q ;αa = softmax5∑

i=1

λ(i)a α(i)

a , (22)

where λ(i)q and λ(i)a are hyper-parameters that determines theweights of the five kinds of attentions. In order to observe thecontribution of each view of attention, we assign the sameweight to all views of attention in the experiment. Finally,the attentive QA representations will be:

sq = Sqαq; sa = Saαa. (23)

Multi-View Attention SharingAs the multi-view attention is applied over hidden states inthe shared representation layer, the parameters to computethe attention weights are supposed to be shared across tasksas well. Meanwhile, different tasks are connected by themulti-view attention, since the multi-view attention schemegathers the information from both task-specific layers andthe shared layers.

ExperimentDatasets & PreprocessingWe use YahooQA (Tay et al. 2017) and TREC QA (Wang,Smith, and Mitamura 2007) for answer selection task, andSimpleQuestions (Bordes et al. 2015) and WebQSP (Yih etal. 2016) for knowledge base question answering task. Thestatistics of these datasets are described in Table 2.

Dataset #Question (train/dev/test) #QA Pair (train/dev/test)

Yahoo QA 50098/6289/6283 253K/31K/31KTREC QA 1229/82/100 53417/1148/1517

SimpleQuestions 71038/10252/20464 571K/80K/164KWebQSP 3067/-/1632 302K/-/160K

Table 2: Summary statistics of datasets.

YahooQA1 An open-domain community-based datasetcollected from Yahoo Answers. Tay et al. (2017) filters outquestions and answers whose length is out of 5-50, and gen-erates 4 negative samples for each question. The same met-rics as (Tay, Tuan, and Hui 2018a; Deng et al. 2018) areadopted for evaluation, including Precision@1 and MRR.

TREC QA A widely-adopted factoid question answer-ing dataset. Following previous works (Tay, Tuan, and Hui2018a; Deng et al. 2018), we experiment on the raw TRECQA dataset and use Mean Average Precision (MAP) andMean Reciprocal Rank (MRR) as evaluation metrics.

SimpleQuestions A single-relation KBQA dataset. Thisdataset consists of questions annotated with a correspondingfact from Freebase that provides the answer. We report Ac-curacy as previous studies (Yih et al. 2016; Yu et al. 2017).A question is considered answered correctly only when thepredicted positive answers match one of the ground-truths.

WebQSP A multi-relation KBQA dataset. Yih et al. (2016)created this dataset by extracting the questions that are an-swerable using Freebase from WebQuestions (Berant et al.2014). We adopt Accuracy as the evaluation metric as theSimpleQuestions dataset.

In our setting, we assume that the data are preprocessedby entity linking, which means entities in sentences have al-ready been extracted and linked to certain entities in the KB.

Entity Linking We use FB5M2 as the knowledge base,which contains 4,904,397 entities, 7,523 relations, and22,441,880 facts. For YahooQA and TREC QA3, we labelall the sentences with the entity mentioned in themselves.We apply the TagMe4 entity linker to extract entity men-tions from sentences, and only keep the entity mentions withthe confidence score above the 0.2 threshold. For each en-tity mention, we retrieve one certain entity from FB5M. ForSimpleQuestions and WebQSP, we start with the entity link-ing results5 as Yu et al. (2017).

1https://github.com/vanzytay/YahooQA Splits2https://research.facebook.com/researchers/15439345391893483https://github.com/dengyang17/MTQA4https://github.com/marcocor/tagme-python5https://github.com/Gorov/KBQA RE data

Experiment SettingsThe word embeddings for all the models are initializedby pre-trained GloVE embeddings6 of 300 dimensions.TransE (Bordes et al. 2013) is adopted as the knowledge em-bedding method to generate the knowledge embeddings forall the models. OpenKE7 is employed to implement TransEwith the default settings.

For all the implemented models, we apply the same pa-rameter settings. The LSTM hidden layer size and the finalhidden layer size are both set to 200. The width of the convo-lution filters is set to be 2 and 3, and the number of convolu-tional feature maps is set to be 100. The learning rate and thedropout rate are set to 0.0005 and 0.5 respectively. We trainour models in batches with size of 128. All other param-eters are randomly initialized from [-0.1, 0.1]. The modelparameters are regularized with a L2 regularization strengthof 0.0001. The maximum length of sentence is set to be 40.

Multi-Task Learning ResultsTable 3 summarizes the experimental results of differentmethods on answer selection and knowledge base questionanswering. For answer selection task, four methods listed inTable 3 achieve the state-of-the-art results in Yahoo QA andTREC QA datasets. The first three methods (Tay et al. 2017;Tay, Tuan, and Hui 2018a; Tay, Tuan, and Hui 2018b)are traditional single-task learning methods, while Deng etal. (2018) employs transfer learning method to pre-trainthe model with a large-scale dataset and leverages exter-nal knowledge from KB to improve the sentence represen-tational learning. For KBQA task, we compare the proposedmethod to four single-task learning state-of-the-art methods.Note that we start with the same entity linking results as Yuet al. (2017).

In general, the proposed multi-view attention based MTLmethod, MVA-MTQA-net (MTL), outperforms the state-of-the-art results by a noticeable margin on all the datasets. Forinstance, on the YahooQA and SimpleQuestions dataset, theproposed method improves about 8% and 2% on the metricsover these baselines.

In both MVA-MTQA-net and its basic model (MTQA-net), multi-task learning (MTL) methods can significantlyimprove the performance of all four datasets compared withsingle-task learning (STL), which demonstrates the effec-tiveness of combining answer selection and knowledge basequestion answering to conduct multi-task learning.

Ablation Analysis of Multi-View AttentionIn this section, we conduct ablation experiments to illustratethe effect of multi-view attention scheme in the proposedmethod. We exclude the five kinds of views from MVA-MTQA-Net one by one and report the results in Table 4, in-cluding single-task learning and multi-task learning results.

From the results, we can observe that all kinds of viewcontribute more or less performance boost to the model. Ap-parently, co-attention view attention makes the most contri-

6http://nlp.stanford.edu/data/glove.6B.zip7https://github.com/thunlp/OpenKE

ModelYahoo QA TREC QA SimpleQuestions WebQSP

P@1 MRR MAP MRR Accuracy Accuracy

HD-LSTM (Tay et al. 2017) 0.557 0.735 0.750 0.815 - -CTRN (Tay, Tuan, and Hui 2018a) 0.601 0.755 0.771 0.838 - -

HyperQA (Tay, Tuan, and Hui 2018b) 0.683 0.801 0.770 0.825 - -KAN(AP-LSTM) (Deng et al. 2018) 0.744 0.840 0.797 0.850 - -

BiCNN (Yih et al. 2015) - - - - 0.900 0.777AMPCNN (Wenpeng et al. 2016) - - - - 0.913 -

HR-BiLSTM (Yu et al. 2017) - - - - 0.933 0.825Multiple View Matching (Yu et al., 2018) - - - - 0.937 0.854

MTQA-net (STL) 0.737 0.818 0.763 0.832 0.913 0.808MTQA-net (MTL) 0.752 0.839 0.779 0.841 0.922 0.820

MVA-MTQA-net (STL) 0.806 0.878 0.783 0.838 0.931 0.823MVA-MTQA-net (MTL) 0.833 0.909 0.811 0.862 0.957 0.858

Table 3: Multi-Task Learning Results

ModelYahoo QA TREC QA SimpleQuestions WebQSP

P@1 MRR MAP MRR Accuracy Accuracy

STL MTQA-net 0.737 0.818 0.763 0.832 0.913 0.808MTL MTQA-net 0.752 0.839 0.779 0.841 0.922 0.820

STL

MVA-MTQA-net 0.806 0.878 0.783 0.838 0.931 0.823w/o word view 0.792 0.863 0.769 0.834 0.926 0.809

w/o knowledge view 0.781 0.854 0.761 0.827 0.930 0.818w/o semantic view 0793 0.862 0.773 0.837 0.921 0.813

w/o knowledge semantic view 0.788 0.859 0.762 0.822 0.928 0.814w/o co-attention view 0.775 0.850 0.761 0.824 0.917 0.803

MTL

MVA-MTQA-net 0.833 0.909 0.811 0.862 0.957 0.858w/o word view 0.824 0.894 0.792 0.854 0.947 0.835

w/o knowledge view 0.826 0.893 0.796 0.861 0.944 0.844w/o semantic view 0.822 0.886 0.789 0.856 0.945 0.836

w/o knowledge semantic view 0.822 0.890 0.793 0.856 0.944 0.840w/o co-attention view 0.811 0.882 0.792 0.847 0.937 0.829

Table 4: Ablation Analysis of Multi-View Attention

bution to the improvement, which brings about 2-3% incre-ment on both STL and MTL of four tasks.

For STL, knowledge and knowledge semantic view at-tentions are more distinguishable than word view and se-mantic view in two answer selection tasks, Yahoo QA andTREC QA, which indicates that attending more informa-tive knowledge in the sentence is important to measure thecorrelation between question and answer sentence. Simi-larly, slight difference exists between two word-level viewand two knowledge-level view attentions in SimpleQues-tions and WebQSP tasks. However, the word view and se-mantic attentions contribute more.

For MTL, we observe that each view of attention makes asimilar contribution to the improvement in four tasks, whichdemonstrates that the multi-view attention scheme enableseach task to interact with each other in multi-task learning.

Case Study of Multi-View AttentionMulti-view attention scheme provides an intuitive way to in-spect the importance of each word in the question and theanswer by visualizing the attention weight from each kindof view. Due to the limited space, we randomly choose onequestion-answer pair from TREC QA dataset and one from

SimpleQuestions, and visualize the attention weights pre-dicted by MVA-MTQA-net. To be more differentiated, eachview of the attention weights is normalized to [0, 1].

Figure 4(a)(c) shows the predicted attention weights ofan example from TREC QA. For the attention weights fromthe word view, we observe that words with rich information(e.g., “johnny appleseed” and “real name”) in both questionand answer receive high weights. On the other hand, entitywith valuable information, such as “johnny appleseed” and“john chapman”, are assigned with high weights in knowl-edge view. In two semantic views, not only similar wordsare attended, but also the related sentence elements, e.g.,“aka american folk hero”. In co-attention view, there is amore synthetical attention distribution over all the words. Ingeneral, the distinction in word-level attentions is not as ef-fective as that of knowledge-level. The results indicate thatthe incorporation of knowledge-level information aids in at-tending more valuable information in answer selection task.

Figure 4(b)(d) provides an example from SimpleQues-tions. For the question, we observe a similar distributionwith the attention weights of the question example in Fig-ure 4(a), since the form of questions in KBQA task is in ac-cord with that in answer selection task. However, for the an-

what wasjohnny

appleseed ’s realname

(a) Question - An Example from Answer Selection

0.0

0.2

0.4

0.6

0.8

1.0

Att

enti

onW

eigh

t

sept 26 :

the

birthday of

john

chapman ,

aka

america

nfol

khero

johnny

applesee

d

(c) Answer - An Example from Answer Selection

0.0

0.2

0.4

0.6

0.8

1.0

Att

enti

onW

eigh

t

burdurprovince

traveltravel

destinationguidebooks

travel toturkey

(d) Answer - An Example from KBQA

0.0

0.2

0.4

0.6

0.8

1.0

Att

enti

onW

eigh

t

whatguidebook is for the

burdurprovince

(b) Question - An Example from KBQA

0.0

0.2

0.4

0.6

0.8

1.0

Att

enti

onW

eigh

t

word viewknowledge viewsemantic viewknowledge semantic viewco-attention view

Figure 4: Case Study of Multi-View Attention

swer, the word-level attentions compensate the insufficiencyof the knowledge-level attentions, e.g., “guidebooks”, whichis neglected in the knowledge-level attentions due to the in-completeness of the KG. This result demonstrates the effec-tiveness of combining word-level and knowledge-level in-formation in KBQA task.

Related WorkAnswer Selection Neural networks based models has beenproven effective in answer selection task, e.g., convolu-tional neural network (CNN) (Severyn and Moschitti 2015)and recurrent models like the long short-term memory(LSTM) (Wang and Nyberg 2015). Instead of learning therepresentations of the question and the answer separately,most recent studies utilize attention mechanisms to learnthe interaction information between questions and answers,which can better focus on relevant parts of the input (Tanet al. 2016; dos Santos et al. 2016; Lei et al. 2018). Withthe emergence of large-scale knowledge bases, several ef-forts have been made on incorporating external knowledgefrom KBs to improve answer selection models. Savenkovand Agichtein (2017) scores candidate answers by com-bining the supporting evidence from structured KBs andunstructured text documents. Ying et al. (2018) proposeda knowledge-aware attentive neural network to attend theknowledge information in QA sentences, based on the enti-ties in sentences. Deng et al. (2018) leverage external knowl-edge from KB as a bridge to transfer more valuable informa-tion between cross-domain answer selection tasks.Knowledge Base Question Answering Most approaches ofKBQA are based on semantic parsing, which transform nat-ural language questions into structured query to extract theanswer from the knowledge base (Yao and Durme 2014;Yih et al. 2015). Besides, most of state-of-the-art results

are achieved by deep learning models. The core idea is tolearn semantic representations of both the question and theknowledge. Based on the relevancy score between the rep-resentations of the question and the knowledge, the modelultimately re-rank a list of candidate facts from KB (Bordeset al. 2015; Wenpeng et al. 2016; Hao et al. 2017). Contraryto answer selection, some recent studies integrate word orcontext information to aid in knowledge base question an-swering (Yu et al. 2017; Sorokin and Gurevych 2018).Multi-Task Learning Recent years have witnessed greatsuccesses of applying multi-task learning with neural net-works on many NLP problems (Liu, Qiu, and Huang 2017;Guo, Pasunuru, and Bansal 2018). Improved from the hardparameter sharing (Caruana 1997), most of these modelsconsist of some task-specific private layers separated fromthe shared layer (Abhishek and Hal 2012). In addition, thekey innovation in most recent studies in multi-task learningis the technique used to connect the private and shared parts.Chen et al. (2018) employs a meta-network to capture themeta-knowledge from the shared representational learningand generate the parameters of the task-specific representa-tional learning. Zheng, Chen, and Qiu (2018) exploits atten-tion mechanisms to select the task-specific information fromthe shared representation layer. Different from this multi-task learning scheme, we propose a multi-view attentionscheme to gather the valuable information from both task-specific and shared perspective, which enhance the interac-tion between different tasks.

ConclusionIn this paper, we study multi-task learning approaches tosolve answer selection and knowledge base question answer-ing simultaneously. We propose a novel multi-task learning

scheme that utilizes multi-view attention learned from vari-ous perspectives to enable these tasks to interact with eachother as well as learn more comprehensive sentence repre-sentations, including word view, knowledge view, semanticview, knowledge semantic view and co-attention view. Theexperiments conducted on several widely-used benchmarkQA datasets demonstrate that joint learning of answer selec-tion and knowledge base question answering significantlyoutperforms single-task learning methods. Also, the multi-view attention scheme is effective in assembling attentiveinformation from different representational perspectives toimprove overall representational learning.

AcknowledgmentsThis work was financially supported by the National Natu-ral Science Foundation of China (No.61602013), the NaturalScience Foundation of Guangdong (No.2018A030313017),the Shenzhen Fundamental Research Project (Grant No.JCYJ20170818091546869), and the project ”PCL FutureRegional Network Facilities for Large-scale Experimentsand Applications”. Min Yang was sponsored by CCF-Tencent Open Research Fund.

References[Abhishek and Hal] Abhishek, K., and Hal, D. I. 2012.Learning task grouping and overlap in multi-task learning.In ICML.

[Berant et al.] Berant, J.; Chou, A.; Frostig, R.; and Liang, P.2014. Semantic parsing on freebase from question-answerpairs. In EMNLP.

[Bollacker et al.] Bollacker, K.; Evans, C.; Paritosh, P.;Sturge, T.; and Taylor, J. 2008. Freebase: a collaborativelycreated graph database for structuring human knowledge. InSIGMOD, 1247–1250.

[Bordes et al.] Bordes, A.; Usunier, N.; Weston, J.; andYakhnenko, O. 2013. Translating embeddings for model-ing multi-relational data. In NIPS, 2787–2795.

[Bordes et al.] Bordes, A.; Usunier, N.; Chopra, S.; and We-ston, J. 2015. Large-scale simple question answering withmemory networks. Computer Science.

[Caruana] Caruana, R. 1997. Multitask learning. MachineLearning 28(1):41–75.

[Chen et al.] Chen, J.; Qiu, X.; Liu, P.; and Huang, X. 2018.Meta multi-task learning for sequence modeling. In AAAI.

[Deng et al.] Deng, Y.; Shen, Y.; Yang, M.; Li, Y.; Du, N.;Fan, W.; and Lei, K. 2018. Knowledge as A bridge: Improv-ing cross-domain answer selection with external knowledge.In COLING, 3295–3305.

[dos Santos et al.] dos Santos, C. N.; Tan, M.; Xiang, B.;and Zhou, B. 2016. Attentive pooling networks. CoRR,abs/1602.03609.

[Guo, Pasunuru, and Bansal] Guo, H.; Pasunuru, R.; andBansal, M. 2018. Soft layer-specific multi-task summariza-tion with entailment and question generation. In ACL.

[Hao et al.] Hao, Y.; Zhang, Y.; Liu, K.; He, S.; Liu, Z.; Wu,H.; Zhao, J.; Hao, Y.; Zhang, Y.; and Liu, K. 2017. An end-to-end model for question answering over knowledge basewith cross-attention combining global knowledge. In ACL.

[Lei et al.] Lei, S.; Xiaodong, Z.; Feng, Q.; Baobao, C.; andZhifang, S. 2018. A multi-view fusion neural network foranswer selection. In AAAI.

[Liu, Qiu, and Huang] Liu, P.; Qiu, X.; and Huang, X. 2017.Adversarial multi-task learning for text classification. InACL, 1–10.

[Rajpurkar et al.] Rajpurkar, P.; Zhang, J.; Lopyrev, K.; andLiang, P. 2016. Squad: 100, 000+ questions for machinecomprehension of text. In EMNLP, 2383–2392.

[Savenkov and Agichtein] Savenkov, D., and Agichtein, E.2017. Evinets: Neural networks for combining evidence sig-nals for factoid question answering. In ACL, 299–304.

[Severyn and Moschitti] Severyn, A., and Moschitti, A.2015. Learning to rank short text pairs with convolutionaldeep neural networks. In SIGIR, 373–382.

[Sorokin and Gurevych] Sorokin, D., and Gurevych, I. 2018.Modeling semantics with gated graph neural networks forknowledge base question answering. In COLING, 3306–3317.

[Tan et al.] Tan, M.; Santos, C. D.; Xiang, B.; and Zhou, B.2016. Improved representation learning for question answermatching. In ACL, 464–473.

[Tay et al.] Tay, Y.; Phan, M. C.; Tuan, L. A.; and Hui, S. C.2017. Learning to rank question answer pairs with holo-graphic dual lstm architecture. In SIGIR.

[Tay, Tuan, and Hui] Tay, Y.; Tuan, L. A.; and Hui, S. C.2018a. Cross temporal recurrent networks for ranking ques-tion answer pairs. In AAAI.

[Tay, Tuan, and Hui] Tay, Y.; Tuan, L. A.; and Hui, S. C.2018b. Hyperbolic representation learning for fast and ef-ficient neural question answering. In WSDM.

[Wang and Nyberg] Wang, D., and Nyberg, E. 2015. A longshort-term memory model for answer sentence selection inquestion answering. In ACL, 707–712.

[Wang, Smith, and Mitamura] Wang, M.; Smith, N. A.; andMitamura, T. 2007. What is the jeopardy model? a quasi-synchronous grammar for qa. In EMNLP, 22–32.

[Wenpeng et al.] Wenpeng, Y.; Mo, Y.; Bing, X.; Bowen, Z.;and Hinrich, S. 2016. Simple question answering by atten-tive convolutional neural network. In COLING, 1746–1756.

[Yao and Durme] Yao, X., and Durme, B. V. 2014. Infor-mation extraction over structured data: Question answeringwith freebase. In ACL, 956–966.

[Yih et al.] Yih, W. T.; Chang, M. W.; He, X.; and Gao, J.2015. Semantic parsing via staged query graph generation:Question answering with knowledge base. In ACL, 1321–1331.

[Yih et al.] Yih, W. T.; Richardson, M.; Meek, C.; Chang,M. W.; and Suh, J. 2016. The value of semantic parselabeling for knowledge base question answering. In ACL,201–206.

[Ying et al.] Ying, S.; Yang, D.; Min, Y.; Yaliang, L.; Nan,D.; Wei, F.; and Kai, L. 2018. Knowledge-aware attentiveneural networks for ranking question answer pairs. In SIGIR.

[Yu et al.] Yu, M.; Yin, W.; Hasan, K. S.; dos Santos, C.; Xi-ang, B.; and Zhou, B. 2017. Improved neural relation detec-tion for knowledge base question answering. In ACL, 571–581.

[Zheng, Chen, and Qiu] Zheng, R.; Chen, J.; and Qiu, X.2018. Same representation, different attentions: Shareablesentence representation learning from multiple tasks. In IJ-CAI, 4616–4622.