Quentin Grail, Arnaud Sors, Morgan Funtowicz, Julien Perez

Machine Learning and Optimization group

6th September, 2018

Machine Reading, Models and Applications

Content

1. Machine reading tasks

2. Models of reading

3. Applications

4. Open Questions

2

Courtesy of Phil Blunsom

3

4

5

7

Machine Readingmotivations

8

Human knowledge is (mainly) stored in natural language

Natural Language is an efficient support of knowledge transcription

Languages assume apriori knowledgeof the world a.k.a common sense

Language is efficient because of itscontextuallity that leads to ambiguity

The Library of Trinity College Dublin

9

“A machine comprehends a passage of text if, for any question regarding that text, it can be answered correctly by a majority of native speakers.

The machine needs to provide a string which human readers would agree both 1. Answers that question2. Does not contain information irrelevant to that question.” (Burges, 2013)

Applications

• Collection of documents as KB• Social media mining• Dialog understanding • Fact checking – Fake news detection

Machine ReadingDefinition

Target Ontology

Reading

Machine Reasoning

Machine

Structured Query

Answer

Textual Documents

Knowledge Base approach

[1] Machine Reading, Peñas et al, 2011

Structureddata

10

“A system that produces machine operable representations of texts”

“ A system that produces machine operable representations of texts ”

… but we have 3 problems here

1. Fixed/Predefined ontologies

2. Fixed/Predefined lexical domain

3. Data duplication by structuration

Knowledge Base approach

Information retrieval approach

[2] Introduction to information Retrieval, Manning et al, 2008 12

“Information Retrieval (IR) is finding material, usually documents, of an unstructured nature, usually text, that satisfies an information need from within large collections usually stored on computers.” Manning,Introduction to IR.

“Machine reading, yet another (Deep) NLP task ? ”

… but we have 3 problems here

1. Is (Language dependant) syntax a requirement to semantics ?

2. Additional (unnecessary) requirement • Annotations• Priors

3. Not end-to-end machine comprehension

Classic Deep NLP approach

Machine Readingas Multi-choice question task

MCTest

• 500 passages

• 2000 questions about simple stories

RACE

• 28,000 passages

• 100,000 questions from English comprehension

tests

14

[5] MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text, Richardson et al, 2013[6] RACE: Large-scale ReAding Comprehension Dataset From Examinations, Lai et al, 2017

Machine Readingas Span selection

SQuAD• 500 passages• 100,000 questions on Wikipedia text• Human annotated

TriviaQA• 95k questions• 650k evidence documents• distant supervision

15[6] SQuAD: 100,000+ Questions for Machine Comprehension of Text, Liang et al, 2016[7] TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension, Zottlemoyer et al, 2017

Machine readingReasoning over knowledge extraction

– Textual data can specify reasoning capabilities

– Goal: build machines that can "understand" textual information, i.e. converting it into interpretable structured knowledge to be leveraged by humans and other machines alike.

– Optimized with categorical cross-entropyloss

[9] Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks, Weston and al 16

Machine ReadingDatasets

17DSTC6-T1

Building blocksRecurrent Neural Network

[10] Long Short Term Memory, Schmidhuber et al, 1997 18

19

Elements:

• Input sentence:

• Output local feature:

• Feature map:

• Max-pooling layer

• Fully connected layer with softmax output for classification tasks

… Trivial to parallelize

Building blocksConvolutional Network

[11] Convolutional Neural Networks for Sentence Classification, Kim et al, 2017

20

Building blocksAttention mechanism In Neural Machine Translation

• Encode each work in the input and output sentence into a vector

• Perform a linear combination of these vectors, weighted by « attention score »

• Use this combination as support to pick the next word

[12] Neural Machine Translation by Jointly Learning to Align and Translate, Badhanau et al, 2015

21

Building blocksSelf-Attention mechanism

[13] Attention Is All You Need, Polosukhin et al, 2017

22[14] Pointer Networks, Vinyals et al, 2015

Building blocksPointer Networks

Extractive modelsAttention Sum Reader Network

[15] Text Understanding with the Attention Sum Reader Network, Kadlec et al, 2016 23

24

Extractive modelsBidirectional Attention Flow for Machine Comprehension

[16] Bidirectional Attention Flow for Machine Comprehension, Seo et al, 2016

Extractive modelsR-Net

[17] R-Net, technical report, Microsoft Asia, 201725

Extractive modelsGoogle QANet

[18] Combining Local Convolution with Global Self-Attention for Reading Comprehension, Google Research, 2017 26

• Extractive model

• Fully differentiable

• Non-autoregressive model

• Language independant

• « Attention is All you Need »

27

Extractive modelsError analysis

Properties• End-to-End memory access regulation• Close to Highway Network and Residual Network

[19] Gated End-to-End Memory Network, Liu and Perez, EACL 2017

gated controller update

28

Reasoning modelsGated End-to-end memory networks

29

20 bAbi tasks: Benchmark results

30

Dialog State trackingExamples & Definition

[20] The third Dialog State Tracking Challenge, Henderson et al, 2016

Dialogue State TrackingState of the art

31

[21] A generalized rule based tracker for dialogue state tracking, Yu et al, 2014

[22] Deep Neural Network Approach for the Dialog State Tracking Challenge, Henderson et al, 2014

Generative

• {Factorial} HMM• Particle Filter

Discriminative

• Rule-based• CRF/Max Entropy• Deep Neural Network

Dialog State TrackingOpen Challenges

1. Longer context

2. Looser supervision schema

3. Reasoning capability

4. Minimize intermediary reps

– Fixed Ontology

– Fixed KB

32

Good Morning, how can I help you

I need a car for March 10th to go to Paris

Ok, I’m checking this

and find me a cheap hotel for the day after

(-_-) “

[23] Dialog State Tracking, a machine reading approach using deep memory networks, Perez and Liu, EACL 2017

Dialog State TrackingMachine reading approach

33

Table 1. State tracking as machine reading task

[23] Dialog State Tracking, a machine reading approach using deep memory networks, Perez and Liu, EACL 2017

Question

What is the

Pricerange?

Input story

Memory Module

Contro

ller

CheapAnswer

Dot product + softmax

Weighted Sum

Dialog State tracking with End-to-End Memory Network

2: I’m looking for

A cheap restaurant in

The north of town

1: Hi, how can I

Help you ?

3: do you have

a preference

for the type ?

[23] Dialog State Tracking, a machine reading approach using deep memory networks, Perez and Liu, EACL 2017

End-to-End Memory NetworkResults on DSTC-2 – Goal Tracking and Reasoning

35

[24] Dialog State Tracking, a machine reading approach using deep memory networks, Perez et Liu, EACL 2017

Dialog state trackingMachine reading approach

On “one supporting fact” task (DSTC-2 dataset): 83% acc vs 79% for the sota.

[24] Dialog State Tracking, a machine reading approach, Perez and Liu, 2017 36

37

Review readingInspiration from relational visual question answering [Johnson et al, 2017]

38

Review readingReviewQA: a relational aspect-based opinion reading dataset

[25] ReviewQA: a relational aspect-based opinion reading dataset, Grail and Perez, 2018

39

• Given a claim, retrieve evidence documents for and against it

• Given evidence documents, find relevant paragraphsand sentences in it

• For claim and each evidence paragraph and sentence: detect stance of paragraph sentence towards a claim/target

Fact checking

[26] http://www.fakenewschallenge.org/ , 2017

Open QuestionsMulti-document Open-Domain Question answering

[27] Reading Wikipedia to Answer Open-Domain Questions, Chen et al, 2017 40

Open QuestionsMulti document reasoning

[28] Constructing Datasets for Multi-hop Reading Comprehension Across Documents, Riedel et al, 2017

• Most Reading Comprehension methods limit themselves to queries which can be answered using a single sentence, paragraph, or document.

• Enabling models to combine disjoint pieces of textual evidence would extend the scope of machine comprehension

• Text understanding across multiple documents and to investigate the limits of existing methods.

• Toward ensemblist operations (union, intersection, selection … )

41

Open QuestionsAdversarial Examples

• Add a sentence or word string specifically designed to distract the model

• Drops accuracy of state-of-the-art models from 81% to 46% of Exact Match accuracy

• Current issue of deep models, alreadyobserved on image tasks

[29] Adversarial Examples for Evaluating Reading Comprehension Systems, Liang et al, 2017 42

Conclusions

43

• Machine reading paradigm, a next step toward natural languagecomprehension

• Promissing results are already available

• Deep learning is (currently) a major enabler of this recent development

• Machine reading is a playground for (deep) machine learning research

• Very active community (Datasets, papers and codes)

• A lot of challenges with numerous possible impacts

Organizing DSCT-7 workshop @AAAI’18

Naver Labs is Hiring !

44europe.naverlabs.com

Thank you