Making Neural QA as Simple as Possible but not Simpler

Dirk Weissenborn Georg WieseLanguage Technology Lab, DFKI

Alt-Moabit 91cBerlin, Germany

{dirk.weissenborn, georg.wiese, laura.seiffe}@dfki.de

Laura Seiffe

Abstract

Recent development of large-scale ques-tion answering (QA) datasets triggered asubstantial amount of research into end-to-end neural architectures for QA. Increas-ingly complex systems have been con-ceived without comparison to simpler neu-ral baseline systems that would justifytheir complexity. In this work, we proposea simple heuristic that guides the develop-ment of neural baseline systems for the ex-tractive QA task. We find that there aretwo ingredients necessary for building ahigh-performing neural QA system: first,the awareness of question words whileprocessing the context and second, a com-position function that goes beyond simplebag-of-words modeling, such as recurrentneural networks. Our results show thatFastQA, a system that meets these two re-quirements, can achieve very competitiveperformance compared with existing mod-els. We argue that this surprising findingputs results of previous systems and thecomplexity of recent QA datasets into per-spective.

1 Introduction

Question answering is an important end-user taskat the intersection of natural language processing(NLP) and information retrieval (IR). QA systemscan bridge the gap between IR-based search en-gines and sophisticated intelligent assistants thatenable a more directed information retrieval pro-cess. Such systems aim at finding precisely thepiece of information sought by the user instead ofdocuments or snippets containing the answer. Aspecial form of QA, namely extractive QA, dealswith the extraction of a direct answer to a question

from a given textual context.The creation of large-scale, extractive QA

datasets (Rajpurkar et al., 2016; Trischler et al.,2017; Nguyen et al., 2016) sparked research in-terest into the development of end-to-end neuralQA systems. A typical neural architecture consistsof an embedding-, encoding-, interaction- and an-swer layer (Wang and Jiang, 2017; Yu et al., 2017;Xiong et al., 2017; Seo et al., 2017; Yang et al.,2017; Wang et al., 2017). Most such systems de-scribe several innovations for the different layersof the architecture with a special focus on devel-oping powerful interaction layer that aims at mod-eling word-by-word interaction between questionand context.

Although a variety of extractive QA systemshave been proposed, there is no competitive neu-ral baseline. Most systems were built in what wecall a top-down process that proposes a complexarchitecture and validates design decisions by anablation study. Most ablation studies, however, re-move only a single part of an overall complex ar-chitecture and therefore lack comparison to a rea-sonable neural baseline. This gap raises the ques-tion whether the complexity of current systems isjustified solely by their empirical results.

Another important observation is the fact thatseemingly complex questions might be answer-able by simple heuristics. Let’s consider the fol-lowing example:

When did building activity occur on St. KazimierzChurch?

Building activity occurred in numerous noble palacesand churches [...]. One of the best examples [..] areKrasinski Palace (1677-1683), Wilanow Palace(1677-1696) and St. Kazimierz Church (1688-1692)

Although it seems that evidence synthesis of mul-tiple sentences is necessary to fully understand the

arX

iv:1

703.

0481

6v3

[cs

.CL

] 8

Jun

201

7

relation between the answer and the question, an-swering this question is easily possible by apply-ing a simple context/type matching heuristic. Theheuristic aims at selecting answer spans that a)match the expected answer type (a time as indi-cated by “When”) and b) are close to importantquestion words (“St. Kazimierz Church”). Theactual answer “1688-1692” would easily be ex-tracted by such a heuristic.

In this work, we propose to use the aforemen-tioned context/type matching heuristic as a guide-line to derive simple neural baseline architecturesfor the extractive QA task. In particular, we de-velop a simple neural, bag-of-words (BoW)- and arecurrent neural network (RNN) baseline, namelyFastQA. Crucially, both models do not make useof a complex interaction layer but model interac-tion between question and context only throughcomputable features on the word level. FastQA’sstrong performance questions the necessity of ad-ditional complexity, especially in the interactionlayer, which is exhibited by recently developedmodels. We address this question by evaluatingthe impact of extending FastQA with an addi-tional interaction layer (FastQAExt) and find thatit doesn’t lead to systematic improvements. Fi-nally, our contributions are the following: i) def-inition and evaluation of a BoW- and RNN-basedneural QA baselines guided by a simple heuris-tic; ii) bottom-up evaluation of our FastQA systemwith increasing architectural complexity, reveal-ing that the awareness of question words and theapplication of a RNN are enough to reach state-of-the-art results; iii) a complexity comparison be-tween FastQA and more complex architectures aswell as an in-depth discussion of usefulness of aninteraction layer; iv) a qualitative analysis indi-cating that FastQA mostly follows our heuristicwhich thus constitutes a strong baseline for extrac-tive QA.

2 A Bag-of-Words Neural QA System

We begin by motivating our architectures by defin-ing our proposed context/type matching heuristic:a) the type of the answer span should correspondto the expected answer type given by the ques-tion, and b) the correct answer should further besurrounded by a context that fits the question, or,more precisely, it should be surrounded by manyquestion words. Similar heuristics were frequentlyimplemented explicitly in traditional QA systems,

e.g., in the answer extraction step of Moldovanet al. (1999), however, in this work our heuristicis merely used as a guideline for the constructionof neural QA systems. In the following, we de-note the hidden dimensionality of the model by n,the question tokens by Q = (q1, ..., qLQ

), and thecontext tokens by X = (x1, ..., xLX

).

2.1 EmbeddingThe embedding layer is responsible for mappingtokens x to their corresponding n-dimensionalrepresentation x. Typically this is done by map-ping each word x to its corresponding word em-bedding xw (lookup-embedding) using an embed-ding matrix E, s.t. xw = Ex. Another approachis to embed each word by encoding their corre-sponding character sequence xc = (c1, ..., cLX

)with C, s.t. xc = C(xc) (char-embedding). Inthis work, we use a convolutional neural networkfor C of filter width 5 with max-pooling over timeas explored by Seo et al. (2017), to which we referthe reader for additional details. Both approachesare combined via concatenation, s.t. the final em-bedding becomes x = [xw;xc] ∈ Rd.

2.2 Type MatchingFor the BoW baseline, we extract the span in thequestion that refers to the expected, lexical an-swer type (LAT) by extracting either the questionword(s) (e.g., who, when, why, how, how many,etc.) or the first noun phrase of the question afterthe question words “what” or “which” (e.g., “whatyear did...”).1 This leads to a correct LAT formost questions. We encode the LAT by concate-nating the embedding of the first- and last wordtogether with the average embedding of all wordswithin the LAT. The concatenated representationsare further transformed by a fully-connected layerfollowed by a tanh non-linearity into z ∈ Rn.Note that we refer to a fully-connected layer inthe following by FC, s.t. FC(u) = Wu + b,W ∈ Rn×m, b ∈ Rn,u ∈ Rm.

We similarly encode each potential answer span(s, e) in the context, i.e., all spans with a specified,maximum number of words (10 in this work), byconcatenating the embedding of the first- and lastword together with the average embedding of allwords within the span. Because the surroundingcontext of a potential answer span can give im-portant clues towards the type of an answer span,

1More complex heuristics can be employed here but forthe sake of simplicity we chose a very simple approach.

for instance, through nominal modifiers left of thespan (e.g., “... president obama ...”) or through anapposition right of the span (e.g., “... obama, pres-ident of...”), we additionally concatenate the aver-age embeddings of the 5 words to the left and tothe right of a span, respectively. The concatenatedspan representation, which comprises in total fivedifferent embeddings, is further transformed bya fully-connected layer with a tanh non-linearityinto xs,e ∈ Rn.

Finally, the concatenation of the LAT represen-tation, the span representation and their element-wise product, i.e., [z; xs,e; z�xs,e], serve as inputto a feed-forward neural network with one hiddenlayer which computes the type score gtype(s, e) foreach span (s, e).

2.3 Context Matching

In order to account for the number of surroundingwords of an answer span as a measure for ques-tion to answer span match (context match), we in-troduce two word-in-question features. They arecomputed for each context word xj and explainedin the following

binary The binary word-in-question (wiqb) fea-ture is 1 for tokens that are part of the question andelse 0. The following equation formally definesthis feature where I denotes the indicator function:

wiqbj = I(∃i : xj = qi) (1)

weighted The wiqwj feature for context word xjis defined in Eq. 3, where Eq. 2 defines a ba-sic similarity score between qi and xj based ontheir word-embeddings. It is motivated on the onehand by the intuition that question tokens whichrarely appear in the context are more likely to beimportant for answering the question, and on theother hand by the fact that question words mightoccur as morphological variants, synonyms or re-lated words in the context. The latter can be cap-tured (softly) by using word embeddings insteadof the words themselves whereas the former iscaptured by the application of the softmax oper-ation in Eq. 3 which ensures that infrequent occur-rences of words are weighted more heavily.

simi,j = vwiq(xj � qi) , vwiq ∈ Rn (2)

wiqwj =∑i

softmax(simi,·)j (3)

A derivation that connects wiqw with the term-frequencies (a prominent information retrievalmeasure) of a word in the question and the con-text, respectively, is provided in Appendix A.

Finally, for each answer span (s, e) we computethe average wiqb and wiqw scores of the 5, 10 and20 token-windows to the left and to the right of therespective (s, e)-span. This results in a total of 2(kinds of features)×3 (windows)×2 (left/right) =12 scores which are weighted by trainable scalarparameters and summed to compute the context-matching score gctxt(s, e).

2.4 Answer Span Scoring

The final score g for each span (s, e) is thesum of the type- and the context matching score:g(s, e) = gtype(s, e) + gctxt(s, e). The modelis trained to minimize the softmax-cross-entropyloss given the scores for all spans.

3 FastQA

Although our BoW baseline closely models ourintended heuristic, it has several shortcomings.First of all, it cannot capture the compositional-ity of language making the detection of sensibleanswer spans harder. Furthermore, the semanticsof a question is dramatically reduced to a BoWrepresentation of its expected answer-type and thescalar word-in-question features. Finally, answerspans are restricted to a certain length.

To account for these shortcomings we introduceanother baseline which relies on the application ofa single bi-directional recurrent neural networks(BiRNN) followed by a answer layer that sepa-rates the prediction of the start and end of the an-swer span. Lample et al. (2016) demonstrated thatBiRNNs are powerful at recognizing named en-tities which makes them sensible choice for con-text encoding to allow for improved type match-ing. Context matching can similarly be achievedwith a BiRNN by informing it of the locations ofquestion tokens appearing in the context throughour wiq-features. It is important to recognize thatour model should implicitly learn to capture theheuristic, but is not limited by it.

On an abstract level, our RNN-based model,called FastQA, consists of three basic layers,namely the embedding-, encoding- and answerlayer. Embeddings are computed as explained in§2.1. The other two layers are described in detailin the following. An illustration of the basic archi-

Wha

t is

the

term fo

r a

mee

ting of

mon

gol

chie

fs ?

Q = [...]

At 5

5

a 56

Khuruldai

57

, 58

a 59

coun

cil 6

0

of6

1

Mon

gol 6

2

chie

fs6

3

, [...

]

X =

wiqw

wiqb

start= 57end = 57

Ans

wer

Laye

rEn

code

r

Figure 1: Illustration of FastQA system on example question from SQuAD. The two word-in-questionfeatures (wiqb, wiqw) are presented with varying degrees of activation.

tecture is provided in Figure 1.

3.1 Encoding

In the following, we describe the encoding of thecontext which is analogous to that of the question.

To allow for interaction between the two em-beddings described in §2.1, they are first projectedjointly to a n-dimensional representation (Eq. 4))and further transformed by a single highway layer(Eq. 5) similar to Seo et al. (2017).

x′ = Px , P ∈ Rn×d (4)

ge = σ(FC(x′)) , x′′ = tanh

(FC(x′))

x = gex′ + (1− ge)x′′ (5)

Because we want the encoder to be aware ofthe question words we feed the binary- and theweighted word-in-question feature of §2.3 in addi-tion to the embedded context words as input. Thecomplete input X ∈ Rn+2×LX to the encoder istherefore defined as follows:

X = ([x1; wiqb1; wiq

w1 ], ..., [xLX

; wiqbLX; wiqwLX

])

X is fed to a bidirectional RNN and its outputis again projected to allow for interaction betweenthe features accumulated in the forward and back-ward RNN (Eq. 6). In preliminary experimentswe found LSTMs (Hochreiter and Schmidhuber,1997) to perform best.

H ′ = Bi-LSTM(X) , H ′ ∈ R2n×LX

H = tanh(BH ′>) , B ∈ Rn×2n (6)

We initialize the projection matrix B with[In; In], where In denotes the n-dimensional iden-tity matrix. It follows that H is the sum of theoutputs from the forward- and backward-LSTM atthe beginning of training.

As mentioned before, we utilize the same en-coder parameters for both question and context,except the projection matrix B which is notshared. However, they are initialized the sameway, s.t. the context and question encoding isidentical at the beginning of training. Finally, tobe able to use the same encoder for both questionand context we fix the two wiq features to 1 for thequestion.

3.2 Answer LayerAfter encoding context X to H = [h1, ...,hLX

]and the question Q to Z = [z1, ...,zLQ

], we firstcompute a weighted, n-dimensional question rep-resentation z of Q (Eq. 7). Note that this repre-sentation is context-independent and as such onlycomputed once, i.e., there is no additional word-by-word interaction between context and question.

α = softmax(vqZ) , vq ∈ Rn

z =∑i

αizi (7)

The probability distribution ps for the start loca-tion of the answer is computed by a 2-layer feed-forward neural network with a rectified-linear(ReLU) activated, hidden layer sj as follows:

sj = ReLU (FC ([hj ; z;hj � z]))ps(j) ∝ exp(vssj) , vs ∈ Rn (8)

Encoder (e.g., RNN)

Word-Embedder

Wha

t isth

ete

rm for a

mee

ting of

mon

gol

chie

fs ?Q =

Encoder (e.g., RNN)

Word-Embedder

[...]

At 5

5

a 56

Khuruldai

57

, 58

a 59

coun

cil 6

0

of6

1

Mon

gol 6

2

chie

fs6

3

, [...

]

X =

Interaction-Layer

Answer Layer

start=57 end=57

Optional

Information flow

Figure 2: Illustration of the basic architecturewhich underlies most existing neural QA systems.

The conditional probability distribution pe forthe end location conditioned on the start locations is computed similarly by a feed-forward neuralnetwork with hidden layer ej as follows:

ej = ReLU (FC ([hj ;hs; z;hj � z;hj � hs]))pe(j|s) ∝ exp(veej) , ve ∈ Rn (9)

The overall probability p of predicting an an-swer span (s, e) is p(s, e) = ps(s) · pe(e|s). Themodel is trained to minimize the cross-entropyloss of the predicted span probability p(s, e).

Beam-search During prediction time, we com-pute the answer span with the highest probabilityby employing beam-search using a beam-size of k.This means that ends for the top-k starts are pre-dicted and the span with the highest overall prob-ability is predicted as final answer.

4 Comparison to Prior Architectures

Many neural architectures for extractive QA havebeen conceived very recently. Most of these sys-tems can be broken down into four basic layers forwhich individual innovations were proposed. Ahigh-level illustration of these systems is show inFigure 2. In the following, we compare our systemin more detail with existing models.

Embedder The embedder is responsible for em-bedding a sequence of tokens into a sequence of n-dimensional states. Our proposed embedder (§2.1)is very similar to existing ones used for example inSeo et al. (2017); Yang et al. (2017).

Encoder Embedded tokens are further encodedby some form of composition function. A promi-nent type of encoder is the (bi-directional) recur-rent neural network (RNN) which is also used inthis work. Feeding additional word-level featuressimilar to ours is rarely done with the exception ofWang et al. (2017); Li et al. (2016).

Interaction Layer Most research focused on theinteraction layer which is responsible for word-by-word interaction between context and ques-tion. Different ideas were explored such as atten-tion (Wang and Jiang, 2017; Yu et al., 2017), co-attention (Xiong et al., 2017), bi-directional atten-tion flow (Seo et al., 2017), multi-perspective con-text matching (Wang et al., 2017) or fine-grainedgating (Yang et al., 2017). All of these ideas aimat enriching the encoded context with weightedstates from the question and in some cases alsofrom the context. These are gathered individuallyfor each context state, concatenated with it andserve as input to an additional RNN. Note that thislayer is omitted completely in FastQA and there-fore constitutes the main simplification over previ-ous work.

Answer Layer Finally, most systems divide theprediction the start and the end by another net-work. Their complexity ranges from using a sin-gle fully-connected layer (Seo et al., 2017; Wanget al., 2017) to employing convolutional neuralnetworks (Trischler et al., 2017) or recurrent, deepHighway-Maxout-Networks(Xiong et al., 2017).We further introduce beam-search to extract themost likely answer span with a simple 2-layerfeed-forward network.

5 FastQA Extended

To explore the necessity of the interaction layerand to be architecturally comparable to existingmodels we extend FastQA with an additional inter-action layer (FastQAExt). In particular, we intro-duce representation fusion to enable the exchangeof information in between passages of the con-text (intra-fusion), and between the question andthe context (inter-fusion). Representation fusion isdefined as the weighted addition between a state,i.e., its n-dimensional representation, and its re-spective co-representation. For each context stateits corresponding co-representation is retrieved viaattention from the rest of the context (intra) orthe question (inter), respectively, and “fused” into

its own representation. For the sake of brevitywe describe technical details of this layer in Ap-pendix B, because this extension is not the focusof this work but merely serves as a representativeof the more complex architectures described in §4.

6 Experimental Setup

We conduct experiments on the following datasets.

SQuAD The Stanford Question AnsweringDataset (Rajpurkar et al., 2016)2 comprisesover 100k questions about paragraphs of 536Wikipedia articles.

NewsQA The NewsQA dataset (Trischler et al.,2017)3 contains 100k answerable questions froma total of 120k questions. The dataset is built fromCNN news stories that were originally collectedby Hermann et al. (2015).

Performance on the SQuAD and NewsQAdatasets is measured in terms of exact match (ac-curacy) and a mean, per answer token-based F1measure which was originally proposed by Ra-jpurkar et al. (2016) to also account for partialmatches.

6.1 Implementation DetailsBoW Model The BoW model is trained onspans up to length 10 to keep the computationtractable. This leads to an upper bound of about95% accuracy on SQuAD and 87% on NewsQA.As pre-processing steps we lowercase all inputsand tokenize it using spacy4. The binary word inquestion feature is computed on lemmas providedby spacy and restricted to alphanumeric words thatare not stopwords. Throughout all experiments weuse a hidden dimensionality of n = 150, dropoutat the input embeddings with the same mask for allwords (Gal and Ghahramani, 2015) and a rate of0.2 and 300-dimensional fixed word-embeddingsfrom Glove (Pennington et al., 2014). We em-ployed ADAM (Kingma and Ba, 2015) for op-timization with an initial learning-rate of 10−3

which was halved whenever the F1 measure onthe development set dropped between epochs. Weused mini-batches of size 32.

FastQA The pre-processing of FastQA isslightly simpler than that of the BoW model. We

2https://rajpurkar.github.io/SQuAD-explorer/

3https://datasets.maluuba.com/NewsQA/4http://spacy.io

tokenize the input on whitespaces (exclusive) andnon-alphanumeric characters (inclusive). Thebinary word in question feature is computed onthe words as they appear in context. Throughoutall experiments we use a hidden dimensionalityof n = 300, variational dropout at the input em-beddings with the same mask for all words (Galand Ghahramani, 2015) and a rate of 0.5 and 300-dimensional fixed word-embeddings from Glove(Pennington et al., 2014). We employed ADAM(Kingma and Ba, 2015) for optimization with aninitial learning-rate of 10−3 which was halvedwhenever the F1 measure on the developmentset dropped between checkpoints. Checkpointsoccurred after every 1000 mini-batches eachcontaining 64 examples.

Cutting Context Length Because NewsQAcontains examples with very large contexts (up tomore than 1500 tokens) we cut contexts larger than400 tokens in order to efficiently train our models.We ensure that at least one, but at best all answersare still present in the remaining 400 tokens. Notethat this restriction is only employed during train-ing.

7 Results

7.1 Model Component Analysis

Model DevF1 Exact

Logistic Regression1 51.0 40.0

Neural BoW Baseline 56.2 43.8

BiLSTM 58.2 48.7BiLSTM + wiqb 71.8 62.3BiLSTM + wiqw 73.8 64.3BiLSTM + wiqb+w (FastQA∗) 74.9 65.5

FastQA∗ + intrafusion 76.2 67.2FastQA∗ + intra + inter (FastQAExt∗) 77.5 68.4

FastQA∗ + char-emb. (FastQA) 76.3 67.6FastQAExt∗ + char-emb. (FastQAExt) 78.3 69.9

FastQA w/ beam-size 5 76.3 67.8FastQAExt w/ beam-size 5 78.5 70.3

Table 1: SQuAD results on development set forincreasingly complex architectures. 1Rajpurkaret al. (2016)

Table 1 shows the individual contributions ofeach model component that was incrementallyadded to a plain BiLSTM model without features,character embeddings and beam-search. We seethat the most crucial performance boost stems

from the introduction of either one of our features(≈ 15% F1). However, all other extensions alsoachieve notable improvements typically between1 and 2% F1. Beam-search slightly improves re-sults which shows that the most probable start isnot necessarily the start of the best answer span.

In general, these results are interesting in manyways. For instance, it is surprising that a simplebinary feature like wiqb can have such a dramaticeffect on the overall performance. We believe thatthe reason for this is the fact that an encoder with-out any knowledge of the actual question has toaccount for every possible question that might beasked, i.e., it has to keep track of the entire con-text around each token in its recurrent state. Aninformed encoder, on the other hand, can selec-tively keep track of question related information.It can further abstract over concrete entities to theirrespective types because it is rarely the case thatmany entities of the same type occur in the ques-tion. For example, if a person is mentioned in thequestion the context encoder only needs to remem-ber that the “question-person” was mentioned butnot the concrete name of the person.

Another interesting finding is the fact that ad-ditional character based embeddings have a no-table effect on the overall performance which wasalready observed by Seo et al. (2017); Yu et al.(2017). We see further improvements when em-ploying representation fusion to allow for more in-teraction. This shows that a more sophisticated in-teraction layer can help. However, the differencesare not substantial, indicating that this extensiondoes not offer any systematic advantage.

7.2 Comparing to State-of-the-Art

Our neural BoW baseline achieves good resultson both datasets (Tables 3 and 1)5. For instance,it outperforms a feature rich logistic-regressionbaseline on the SQuAD development set (Table 1)and nearly reaches the BiLSTM baseline sys-tem (i.e., FastQA without character embeddingsand features). It shows that more than half ormore than a third of all questions in SQuAD orNewsQA, respectively, are (partially) answerableby a very simple neural BoW baseline. How-ever, the gap to state-of-the-art systems is quitelarge (≈ 20%F1) which indicates that employing

5We did not evaluate the BoW baseline on the SQuADtest set because it requires submitting the model to Rajpurkaret al. (2016) and we find that comparisons on NewsQA andthe SQuAD development set give us enough insights.

Model TestF1 Exact

Logistic Regression1 51.0 40.4Match-LSTM2 73.7 64.7Dynamic Chunk Reader3 71.0 62.5Fine-grained Gating4 73.3 62.5Multi-Perspective Matching5 75.1 65.5Dynamic Coattention Networks6 75.9 66.2Bidirectional Attention Flow7 77.3 68.0r-net8 77.9 69.5

FastQA w/ beam-size k = 5 77.1 68.4FastQAExt k = 5 78.9 70.8

Table 2: Official SQuAD leaderboard of single-model systems on test set from 2016/12/29, thedate of submitting our model. 1Rajpurkar et al.(2016), 2Wang and Jiang (2017), 3Yu et al. (2017),4Yang et al. (2017), 5Wang et al. (2017), 6Xionget al. (2017), 7Seo et al. (2017), 8 not published.Note that systems are regularly uploaded and im-proved on SQuAD.

Model Dev TestF1 Exact F1 Exact

Match-LSTM1 48.9 35.2 48.0 33.4BARB2 49.6 36.1 48.3 34.1

Neural BoW Baseline 37.6 25.8 36.6 24.1FastQA k = 5 56.4 43.7 55.7 41.9FastQAExt k = 5 56.1 43.7 56.1 42.8

Table 3: Results on the NewsQA dataset.1Wang and Jiang (2017) was re-implemented by2Trischler et al. (2017).

more complex composition functions than averag-ing, such as RNNs in FastQA, are indeed neces-sary to achieve good performance.

Results presented in Tables 2 and 3 clearlydemonstrate the strength of the FastQA system. Itis very competitive to previously established state-of-the-art results on the two datasets and even im-proves those for NewsQA. This is quite surpris-ing when considering the simplicity of FastQAputting existing systems and the complexity ofthe datasets, especially SQuAD, into perspective.Our extended version FastQAExt achieves evenslightly better results outperforming all reportedresults prior to submitting our model on the verycompetitive SQuAD benchmark.

In parallel to this work Chen et al. (2017) in-troduced a very similar model to FastQA, whichrelies on a few more hand-crafted features and a3-layer encoder instead of a single layer in this

work. These changes result in slightly better per-formance which is in line with the observations inthis work.

7.3 Do we need additional interaction?

In order to answer this question we compareFastQA, a system without a complex word-by-word interaction layer, to representative modelsthat have an interaction layer, namely FastQAExtand the Dynamic Coattention Network (DCN,Xiong et al. (2017)). We measured both time-and space-complexity of FastQAExt and a reim-plementation of the DCN in relation to FastQAand found that FastQA is about twice as fast asthe other two systems and requires 2 − 4× lessmemory compared to FastQAExt and DCN, re-spectively6.

In addition, we looked for systematic advan-tages of FastQAExt over FastQA by comparingSQuAD examples from the development set thatwere answered correctly by FastQAExt and incor-rectly by FastQA (589 FastQAExt wins) againstFastQA wins (415). We studied the averagequestion- and answer length as well as the ques-tion types for these two sets but could not findany systematic difference. The same observationwas made when manually comparing the kind ofreasoning that is needed to answer a certain ques-tion. This finding aligns with the marginal em-pirical improvements, especially for NewsQA, be-tween the two systems indicating that FastQAExtseems to generalize slightly better but does notoffer a particular, systematic advantage. There-fore, we argue that the additional complexity in-troduced by the interaction layer is not necessarilyjustified by the incremental performance improve-ments presented in §7.2, especially when memoryor run-time constraints exist.

7.4 Qualitative Analysis

Besides our empirical evaluations this section pro-vides a qualitative error inspection of predictionsfor the SQuAD development dataset. We analyse55 errors made by the FastQA system in detail andhighlight basic abilities that are missing to reachhuman level performance.

We found that most errors are based on a lackof either syntactic understanding or a fine-grainedsemantic distinction between lexemes with similar

6We implemented all models in TensorFlow (Abadi et al.,2015).

meanings. Other error types are mostly related toannotation preferences, e.g., answer is good butthere is a better, more specific one, or ambiguitieswithin the question or context.

Example FastQA errors. Predicted answers are under-lined while correct answers are presented in boldface.

Ex. 1: What religion did the Yuan discourage, tosupport Buddhism?

Buddhism (especially Tibetan Buddhism) flourished,although Taoism endured ... persecutions... from theYuan government

Ex. 2: Kurt Debus was appointed what position for theLaunch Operations Center?

Launch Operations Center (LOC) ... Kurt Debus,a member of Dr. Wernher von Braun’s ... team. Debuswas named the LOC’s first Director .

Ex. 3: On what date was the record low temperature inFresno?

high temperature for Fresno ... set on July 8, 1905,while the official record low ... set on January 6, 1913

A prominent type of mistake is a lack of fine-grained understanding of certain answer types (Ex.1). Another error is the lack of co-reference reso-lution and context sensitive binding of abbrevia-tions (Ex. 2). We also find that the model some-times struggles to capture basic syntactic struc-ture, especially with respect to nested sentenceswhere important separators like punctuation andconjunctions are being ignored (Ex. 3).

A manual examination of errors reveals thatabout 35 out of 55 mistakes (64%) can directlybe attributed to the plain application of our heuris-tic. A similar analysis reveals that about 44 out of50 (88%) analyzed positive cases are covered byour heuristic as well. We therefore believe that ourmodel and, wrt. empirical results, other models aswell mostly learn a simple context/type matchingheuristic.

This finding is important because it reveals thatan extractive QA system does not have to solve thecomplex reasoning types of Chen et al. (2016) thatwere used to classify SQuAD instances (Rajpurkaret al., 2016), in order to achieve current state-of-the-art results.

8 Related Work

The creation of large scale cloze datasets suchthe DailyMail/CNN dataset (Hermann et al., 2015)

or the Children’s Book Corpus (Hill et al., 2016)paved the way for the construction of end-to-endneural architectures for reading comprehension. Athorough analysis by Chen et al. (2016), however,revealed that the DailyMail/CNN was too easy andstill quite noisy. New datasets were constructed toeliminate these problems including SQuAD (Ra-jpurkar et al., 2016), NewsQA (Trischler et al.,2017) and MsMARCO (Nguyen et al., 2016).

Previous question answering datasets such asMCTest (Richardson et al., 2013) and TREC-QA(Dang et al., 2007) were too small to success-fully train end-to-end neural architectures such asthe models discussed in §4 and required differ-ent approaches. Traditional statistical QA sys-tems (e.g., Ferrucci (2012)) relied on linguisticpre-processing pipelines and extensive exploita-tion of external resources, such as knowledgebases for feature-engineering. Other paradigmsinclude template matching or passage retrieval(Andrenucci and Sneiders, 2005).

9 Conclusion

In this work, we introduced a simple, context/typematching heuristic for extractive question answer-ing which serves as guideline for the developmentof two neural baseline system. Especially FastQA,our RNN-based system turns out to be an efficientneural baseline architecture for extractive questionanswering. It combines two simple ingredientsnecessary for building a currently competitive QAsystem: a) the awareness of question words whileprocessing the context and b) a composition func-tion that goes beyond simple bag-of-words mod-eling. We argue that this important finding putsresults of previous, more complex architectures aswell as the complexity of recent QA datasets intoperspective. In the future we want to extend theFastQA model to address linguistically motivatederror types of §7.4.

Acknowledgments

We thank Sebastian Riedel, Philippe Thomas,Leonhard Hennig and Omer Levy for commentson an early draft of this work as well as theanonymous reviewers for their insightful com-ments. This research was supported by theGerman Federal Ministry of Education and Re-search (BMBF) through the projects ALL SIDES(01IW14002), BBDC (01IS14013E), and Soft-ware Campus (01IS12050, sub-project GeNIE).

ReferencesMartin Abadi, Ashish Agarwal, Paul Barham, Eugene

Brevdo, Zhifeng Chen, Craig Citro, Greg Corrado,Andy Davis, Jeffrey Dean, Matthieu Devin, San-jay Ghemawat, Ian Goodfellow, Andrew Harp, Ge-offrey Irving, Michael Isard, Yangqing Jia, LukaszKaiser, Manjunath Kudlur, Josh Levenberg, DanMan, Rajat Monga, Sherry Moore, Derek Mur-ray, Jonathon Shlens, Benoit Steiner, Ilya Sutskever,Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan,Oriol Vinyals, Pete Warden, Martin Wicke, Yuan Yu,and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Dis-tributed Systems .

Andrea Andrenucci and Eriks Sneiders. 2005. Auto-mated question answering: Review of the main ap-proaches. In ICITA.

Danqi Chen, Jason Bolton, and Christopher D Man-ning. 2016. A Thorough Examination of the CNN /Daily Mail Reading Comprehension Task. ACL .

Danqi Chen, Adam Fisch, Jason Weston, and AntoineBordes. 2017. Reading Wikipedia to Answer Open-Domain Questions. ACL .

Hoa Trang Dang, Diane Kelly, and Jimmy J Lin. 2007.Overview of the TREC 2007 Question AnsweringTrack. TREC .

D. A. Ferrucci. 2012. Introduction to ”This is Watson”.IBM Journal of Research and Development .

Yarin Gal and Zoubin Ghahramani. 2015. Dropout as aBayesian Approximation : Representing Model Un-certainty in Deep Learning. ICML .

Karl Moritz Hermann, Tomas Kocisky, EdwardGrefenstette, Lasse Espeholt, Will Kay, Mustafa Su-leyman, and Phil Blunsom. 2015. Teaching Ma-chines to Read and Comprehend. NIPS .

Felix Hill, Antoine Bordes, Sumit Chopra, and JasonWeston. 2016. The Goldilocks Principle: ReadingChildren’s Books with Explicit Memory Represen-tations. ICLR .

Sepp Hochreiter and Jurgen Schmidhuber. 1997.LONG SHORT-TERM MEMORY. Neural Compu-tation .

Diederik Kingma and Jimmy Ba. 2015. Adam: AMethod for Stochastic Optimization. ICLR .

Guillaume Lample, Miguel Ballesteros, Sandeep Sub-ramanian, Kazuya Kawakami, and Chris Dyer. 2016.Neural Architectures for Named Entity Recognition.NAACL .

Peng Li, Wei Li, Zhengyan He, Xuguang Wang,Ying Cao, Jie Zhou, and Wei Xu. 2016. Datasetand Neural Recurrent Sequence Labeling Modelfor Open-Domain Factoid Question Answering.arXiv:1607.06275v1 [cs.CL] .

Dan I. Moldovan, Sanda M. Harabagiu, Marius Pasca,Rada Mihalcea, Richard Goodrum, Roxana Girju,and Vasile Rus. 1999. Lasso: A tool for surfing theanswer net. In TREC.

Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao,Saurabh Tiwary, Rangan Majumder, and Li Deng.2016. Ms Marco: a Human Generated MachineReading Comprehension Dataset. NIPS .

Jeffrey Pennington, Richard Socher, and Christopher DManning. 2014. GloVe: Global Vectors for WordRepresentation. EMNLP .

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, andPercy Liang. 2016. SQuAD: 100,000+ Questionsfor Machine Comprehension of Text. EMNLP .

Matthew Richardson, Christopher J C Burges, and ErinRenshaw. 2013. MCTest: A Challenge Dataset forthe Open-Domain Machine Comprehension of Text.EMNLP .

Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, andHananneh Hajishirzi. 2017. Bi-Directional Atten-tion Flow for Machine Comprehension. In ICLR.

Yelong Shen, Po-Sen Huang, Jianfeng Gao, andWeizhu Chen. 2016. ReasoNet: Learningto Stop Reading in Machine Comprehension.arXiv:1609.05284 .

Adam Trischler, Tong Wang, Xingdi Yuan, Justin Har-ris, Alessandro Sordoni, Philip Bachman, and Ka-heer Suleman. 2017. NewsQA: A Machine Com-prehension Dataset. arXiv:1611.09830 .

Shuohang Wang and Jing Jiang. 2017. MachineComprehension Using Match-LSTM and AnswerPointer. In ICLR.

Zhiguo Wang, Haitao Mi, Wael Hamza, and Radu Flo-rian. 2017. Multi-Perspective Context Matching forMachine Comprehension. arXiv:1612.04211 .

Caiming Xiong, Victor Zhong, and Richard Socher.2017. Dynamic Coattention Networks for QuestionAnswering. ICLR .

Zhilin Yang, Bhuwan Dhingra, Ye Yuan, Junjie Hu,William W Cohen, and Ruslan Salakhutdinov. 2017.Words or Characters? Fine-grained Gating for Read-ing Comprehension. ICLR .

Yang Yu, Wei Zhang, Kazi Hasan, Mo Yu, Bing Xiang,and Bowen Zhou. 2017. End-to-End Reading Com-prehension with Dynamic Answer Chunk Ranking.In ArXiv.

A Weighted Word-in-Question to TermFrequency

In this section we explain the connection betweenthe weighted word-in-question feature (§2.3) de-fined in Eq. 3 and the term frequency (tf) of aword occurring in the question Q = (q1, ..., qLQ

)and context X = (x1, ..., xLX

), respectively. Tofacilitate this analysis, we repeat the equations atthis point:

simi,j = vwiq(xj � qi) , vwiq ∈ Rn (2)

wiqwj =∑i

softmax(simi,·)j (3)

Let us assume that we re-define the similarityscore simi,j of Eq. 2 as follows:

simi,j =

{0 if qi = xj

− inf else(10)

Given the new (discrete) similarity score we canderive the following equation for the wiqw featurefor context word xj . Note that we refer to the termfrequency of a word z in the context and questionby tf(z|C) and tf(z|Q), respectively.

wiqwj =∑i

softmax(simi,·)j

=∑i

exp(simi,j)∑j′ exp(simi,j′)

=∑i

I(xj = qi)∑j′ I(xj′ = qi)

=∑i

I(xj = qi)

tf(qi|C)

=∑

i, qi=xj

1

tf(qi|C)=

tf(xj |Q)

tf(xj |C)

Our derived formula shows that wiqw of con-text word xj would become a simple combinationof the term frequencies of xj within the contextand question if our similarity score is redefined asin Eq. 10. Note that this holds true for any finitevalue chosen in Eq. 10 and not just 0.

B Representation Fusion

B.1 Intra-FusionIt is well known that recurrent neural networkshave a limited ability to model long-term depen-

dencies. This limitation is mainly due to the infor-mation bottleneck that is posed by the fixed sizeof the internal RNN state. Hence, it is hard forour proposed baseline model to answer questionsthat require synthesizing evidence from differenttext passages. Such passages are typically con-nected via co-referent entities or events. Considerthe following example from the NewsQA dataset(Trischler et al., 2017):

Where is Brittanee Drexel from?

The mother of a 17-year-old Rochester, New Yorkhigh school student ... says she did not give herdaughter permission to go on the trip. Brittanee MarieDrexel’s mom says

To correctly answer this question the represen-tations of Rochester, New York should containthe information that it refers to Brittanee Drexel.This connection can, for example, be establishedthrough the mention of mother and its co-referentmention mom. Fusing information from the con-text representation hmom into hmother allows cru-cial information about the mentioning of BrittaneeDrexel to flow close to the correct answer. We en-able the model to find co-referring mentions viaa normalized similarity measure β (Eq. 11). Foreach context state we retrieve its co-state usingβ (Eq. 12) and finally fuse the representations ofeach state with their respective co-state represen-tations via a gated addition (Eq. 13). We call thisprocedure associative representation fusion.

βj,k = I(j 6= k)vβ (hj � hk)βj = softmax(βj,·) (11)

hcoj =∑k

βj,khk (12)

h∗j = FUSE(hj ,hcoj )

= gβhj + (1− gβ)hcoj (13)

gβ = σ(FC([hj ;hcoj ]))

We initialize vβ with 1, s.t. βj,k is initially iden-tical to the dot-product between hidden states.

We further introduce recurrent representationfusion to sequentially propagate information gath-ered by associative fusion between neighbouringtokens, e.g., between the representation of mothercontaining additional information about Britta-nee Drexel and those representations of Rochester,New York. This is achieved via a recurrent

backward- (Eq. 14) followed by a recurrent for-ward fusion (Eq. 15)

hbwj = FUSE(h∗j , h

bwj+1) (14)

hj = FUSE(hbwj , hj−1) (15)

Note, that during representation fusion no newfeatures are computed but simply combined witheach other.

B.2 Inter-FusionThe representation fusion between question andcontext states is very similar to the intra-fusionprocedure. It is applied on top of the contextrepresentations after intra-fusion has been em-ployed. Associative fusion is performed via atten-tion weights γ (Eq. 16) between question states ziand context states hj). The co-state is computedfor each context state via γ (Eq. 17).

γi,j = vγ (zi � hj)γi = softmax(γi,·) (16)

hcoj =

∑i

γi,jzi (17)

Note, because the softmax normalization is ap-plied over the all context tokens for each questionword, γi will be close to zero for most context po-sitions and therefore, its co-state will be close to azero-vector. Therefore, only question related con-text states will receive a non-empty co-state. Therest of inter-fusion follows the same procedure asfor intra-fusion and the resulting context represen-tations serve as input to the answer layer.

In contrast to existing interaction layerswhich typically combine representations retrievedthrough attention by concatenation and feed themas input to an additional RNN (e.g., LSTM), ourapproach can be considered a more light-weightversion of interaction.

C Generative Question Answering

Although our system was designed to answerquestion by extracting answers from a given con-text it can also be employed for generative ques-tion answering datasets such as the MicrosoftMachine Reading Comprehension (MsMARCO,Nguyen et al. (2016))7. MsMARCO contains

7http://www.msmarco.org/

Model Dev TestBleu Rouge Bleu Rouge

ReasoNet1 - - 14.8 19.2Match-LSTM2 - - 37.3 40.7

FastQA k = 5 34.9 33.0 34.0 32.1FastQAExt k = 5 35.0 34.4 33.9 33.7

Table 4: MsMARCO leaderboard of single-modelsystems on test set from 2017/02/07, the date ofsubmitting our model. 1Shen et al. (2016)- ex-tractive model trained out-of-domain on SQuAD,2Wang and Jiang (2017).

100k real world queries from the Bing search en-gine and human generated answers that are basedon relevant web documents. Because we focuson extractive question answering in this work, welimit the queries for training to queries whose an-swers are directly extractable from the given webdocuments. We found that 67.2% of all queriesfall into this category. Evaluation, however, is per-formed on the entire development and test set, re-spectively, which makes it impossible to answerthe subset of Yes/No questions (≈ 7%) properly.For the sake of simplicity, we concatenate all givenparagraphs and treat them as a single document.Since most queries in MsMARCO are lower-casedwe also lower-cased the context. The official scor-ing measure of MsMARCO for generative mod-els is ROUGE-L and BLEU-1. Even though ourmodel is extractive we use our extracted answersas if they were generated.

The results are shown in Table 4. The strongperformance of our purely extractive system onthe generative MsMARCO dataset is notable. Itshows that answers to Bing queries can mostlybe extracted directly from web documents withoutthe need for a more complex generative approach.Since this was only an initial experiment on gen-erative QA using extractive QA and the method-ology used for training, pre- and post-processingon this dataset for the other models, especially forWang and Jiang (2017), is unclear, the compara-bility to the other QA systems is limited.