bridged the gap how deep learning has - bcs irsgbidaf (seo et al. 2016) abcnn (yin et al. 2016)...
TRANSCRIPT
![Page 1: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/1.jpg)
NLP & IRHow Deep Learning hasBridged the Gap
Ryan McDonaldMaterial from a number of Google research projects
![Page 2: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/2.jpg)
TF-IDF
QueryExpansion
Vector Space Models
Clustering
Translation
Rec.Collab.
Filtering
Formal Grammars
IE
NLI
ML /Text Cls.
Situated Environ.
Language Models
Summ.
Network Analysis
Graphical Models
Learn2Rank
Statistical Parsing
QA
InvertedIndex
IR
NLP
Time
KBC
No NLP
No IR Methods
![Page 3: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/3.jpg)
TF-IDF
QueryExpansion
Vector Space Models
Clustering
Translation
Rec.Collab.
Filtering
Formal Grammars
IE
NLI
ML /Text Cls.
Situated Environ.
Language Models
Summ.
Network Analysis
Graphical Models
Learn2Rank
Statistical Parsing
QA
InvertedIndex
IR
NLP
Time
KBC
No NLP
No IR Methods
Deep Learning
![Page 4: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/4.jpg)
f(text, text)f(text, entity)f(text, image)...f(text, object)
?
![Page 5: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/5.jpg)
f(text, text)
EncoderTransformer, LSTM, CNN, FF-NN, etc.
Entail
EncoderTransformer, LSTM, CNN, FF-NN, etc.
Contradict
[CLS] A man inspects the uniform of a figure in some East Asian country. [SEP] The man is sleeping
[CLS] A boy and his mother and father are at the beach [SEP] A family is doing something outside
A man inspects the uniform of a figure in some East Asian country. The man is sleeping
A boy and his mother and father are at the beach A family is doing something outside
EncoderTransformer, LSTM, CNN, FF-NN, etc.
Entail
EncoderTransformer, LSTM,
CNN, FF-NN, etc.
EncoderTransformer, LSTM, CNN, FF-NN, etc.
Contradict
EncoderTransformer, LSTM,
CNN, FF-NN, etc.
![Page 6: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/6.jpg)
Cross-attention (AKA one-tower; AKA interaction-based)
Query
Document
Relevance Scoring
EntailmentQA
Reading Comprehension...
DeepMatch (Lu and Li 2013)ARC-II (Hu et al. 2014)MatchPyramid (Pang et al. 2016)DRMM (Guo et al. 2016)PACRR (Hui et al. 2017)DeepRank (Pang et al. 2017)PDRMM (McDonald et al. 2018)...
Attentive Reader (Hermann et al. 2013)AOA (Cui et al. 2014)DINN (Pang et al. 2016)BiDAF (Seo et al. 2016)ABCNN (Yin et al. 2016)Ruminating Reader (Gong et al. 2017)DIIN (Gong et al. 2018)...
Neural Layers
Neural Layers
![Page 7: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/7.jpg)
Proprietary + Confidential
BERT: Transformers + Pre-training + Fine-Tuning
Pre-Training
Devlin et al. 2019
![Page 8: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/8.jpg)
Proprietary + Confidential
BERT: Transformers + Pre-training + Fine-Tuning
Pre-Training Fine-Tuning
Devlin et al. 2019
![Page 9: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/9.jpg)
Proprietary + Confidential
BERT: Transformers + Pre-training + Fine-Tuning
Pre-Training Fine-Tuning
Devlin et al. 2019
![Page 10: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/10.jpg)
Proprietary + Confidential
Transformers + Pre-training -- new dawn of NLP
Devlin et al. 2019Yang et al. 2019Lan et al. 2019
![Page 11: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/11.jpg)
Proprietary + Confidential
Transformers + Pre-training -- new dawn of NLP
Devlin et al. 2019Yang et al. 2019Lan et al. 2019
![Page 12: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/12.jpg)
Proprietary + Confidential
Transformers + Pre-training -- new dawn of NLP
Devlin et al. 2019Yang et al. 2019Lan et al. 2019
![Page 13: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/13.jpg)
Proprietary + Confidential
BERT: Fine-tuning Paradigms
This can be a relevance scoring model
![Page 14: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/14.jpg)
BERT 4 Document Relevance Scoring
CEDR: Macaveney et al. 2019
![Page 15: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/15.jpg)
Serving
Training Data
IndexingTraining
Dual Encoder Retrieval (AKA two-tower; AKA relevance-based)
Query Document
DLencoder
DLencoder
Score
Query
DLencoder
Corpus
![Page 16: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/16.jpg)
Serving
Training Data
IndexingTraining
Dual Encoder Retrieval (AKA two-tower; AKA relevance-based)
Query Document
DLencoder
DLencoder
Score
Query
DLencoder
Corpus
EncoderTransformer, LSTM, CNN, FF-NN, etc.
Relevant
Query [SEP] Document
![Page 17: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/17.jpg)
Classification with massive output spaces
When did Leonardo pain the Mona Lisa?
![Page 18: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/18.jpg)
Embedding Objects vs. Descriptions
DLencoder
Score
f(text, entity)
When did Leonardo pain the Mona Lisa?
![Page 19: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/19.jpg)
Gillick et al. 2019
Embedding Objects vs. Descriptions
f(text, text)
![Page 20: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/20.jpg)
Zero-Shot Entity Linking
Fethis alor is a dunmer*merchant living in raven rock.
* special mention vector addedto indicate this is the focus mention
dunmer ( skyrim ) the dunmer, more commonly referred to as dark elves
Lee et al. 2019
![Page 21: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/21.jpg)
Zero-Shot Entity Linking
Fethis alor is a dunmer*merchant living in raven rock.
* special mention vector addedto indicate this is the focus mention
dunmer ( skyrim ) the dunmer, more commonly referred to as dark elves
Lee et al. 2019
![Page 22: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/22.jpg)
query passages
Knowledge Retrieval → Comprehension
Reading/QA
Knowledge
![Page 23: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/23.jpg)
Confidential + Proprietary
What does the zip in zip code
stand for?
Zone Improvement
Plan
...The term ZIP is an acronym for Zone
Improvement Plan...
Retrieval Reading Comprehension
Pipelined Approach
Subject: /m/0dllrPredicate: /common/topic/aliasObject: Zone Improvement Plan
End-to-End Latent Retrieval
What does the zip in zip code
stand for?
Zone Improvement
PlanKnowledge
Input Output
![Page 24: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/24.jpg)
Confidential + Proprietary
ORQA Overview
Lee et al. 2019Approx w/ top-K
![Page 25: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/25.jpg)
Results
End-to-end learning is crucial for information-seeking queries!
Lee et al. 2019
![Page 26: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/26.jpg)
Sequential Question Answering
Building City Floors
First Canadian Place Toronto 72
Commerce Court West Toronto 57
Tour de la Bourse Montreal 47
Place Ville-Marie Montreal 44
What are the buildings in Toronto?First Canadian Place, Commerce Court West
(Iyyer et al, 2017)
![Page 27: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/27.jpg)
Sequential Question Answering
Building City Floors
First Canadian Place Toronto 72
Commerce Court West Toronto 57
Tour de la Bourse Montreal 47
Place Ville-Marie Montreal 44
What are the buildings in Toronto?First Canadian Place, Commerce Court WestOf those, which buildings have more than 60 floors?First Canadian Place
(Iyyer et al, 2017)
![Page 28: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/28.jpg)
Sequential Question Answering
Building City Floors
First Canadian Place Toronto 72
Commerce Court West Toronto 57
Tour de la Bourse Montreal 47
Place Ville-Marie Montreal 44
What are the buildings in Toronto?First Canadian Place, Commerce Court WestOf those, which buildings have more than 60 floors?First Canadian PlaceHow many floors does it have?72
(Iyyer et al, 2017)
![Page 29: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/29.jpg)
Graph Transformer Müller et al. 2019
![Page 31: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/31.jpg)
Retrieval + NLP
Retrieve + NLU
Salient Information
Knowledge
![Page 32: Bridged the Gap How Deep Learning has - BCS IRSGBiDAF (Seo et al. 2016) ABCNN (Yin et al. 2016) Ruminating Reader (Gong et al. 2017) DIIN (Gong et al. 2018)... Neural Layers Neural](https://reader033.vdocuments.net/reader033/viewer/2022050602/5fa9472dd7799444ea58f24c/html5/thumbnails/32.jpg)
Thanks!