question answering gideon mann johns hopkins university [email protected]
TRANSCRIPT
![Page 2: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/2.jpg)
Information Retrieval Tasks
Retired General Wesley Clark
How old is General Clark?
How long did Clark serve in the military?
Will Clark run for President?
![Page 3: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/3.jpg)
Ad-Hoc Queries Prior work has been concerned mainly
with answering ad-hoc queries :
General Clark Typically a few words long, not an
entire question What is desired is general information
about the subject in question
![Page 4: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/4.jpg)
Answering Ad-Hoc Queries Main focus of Information Retrieval past 2-3
decades Solution(s) :
– Vector-based methods– SVD, query expansion, language modeling– Return a page as an answer
Resulting systems Extremely Useful– Google, Altavista
![Page 5: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/5.jpg)
Traditional IR
Query
Document Collection
Document Ranking
Document retrieval
![Page 6: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/6.jpg)
But not all queries are Ad-Hoc!
How old is General Clark?
Does not fit well into an Ad-hoc paradigm– “How” and “is” are not relevant for appropriate
retrieval– Potentially useful cues in the question are
ignored in traditional ad-hoc retrieval system
![Page 7: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/7.jpg)
Documents are not Facts Traditional IR systems return Pages
– Useful when only a vague information need has been identified
Insufficient when a fact is desired:– How old is General Clark? 58– How long did Clark serve in the
mililary? 36 years– Will Clark run for president? Maybe
![Page 8: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/8.jpg)
Question Answering as Retrieval
Given a document collection and a question:
A question answering system should retrieve a short snippet of text which exactly answer the question asked.
![Page 9: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/9.jpg)
Question Answering
Query
Document Collection
Document Ranking
Document retrieval
Ranked Answers
Answer Extraction(Sentence ranking)
![Page 10: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/10.jpg)
QA as a Comprehension Task
For perfect recall, the answer only has to appear once in the collection.
In essence, this forces the QA system to function as a text understanding system
Thus QA may be interesting, not only for retrieval, but also to test understanding
![Page 11: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/11.jpg)
QA as a stepping stone Current QA focused on Fact extraction
– Answers appear verbatim in text
How old is General Clark? How can we answer questions which don’t
appear exactly in the text?How long has Clark been in the military?
Will Clark run for President? Maybe build on low-level QA extracted facts
![Page 12: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/12.jpg)
QA Methods
Two Main Categories of Methods for Question Answering
– Answer Preference Matching– Answer Context Matching
![Page 13: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/13.jpg)
Lecture Outline1. Answer Preferences
Question Analysis Type identification Learning Answering Typing
2. Answer ContextLearning Context Similarity Alignment Surface Text Patterns
![Page 14: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/14.jpg)
Answer Type Identification
From the question itself infer the likely type of the answer• How old is General Clark?
How Old
• When did Clark retire? When
• Who is the NBC war correspondent? Who
![Page 16: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/16.jpg)
Answer Type Identification
From the question itself infer the likely type of the answer• How old is General Clark?
How Old Age
• When did Clark retire? When Date
• Who is the NBC war correspondent? Correspondent Person
![Page 17: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/17.jpg)
Wh-Words
Who Person, Organization, Location
When Date, Year
Where Location
In What Location
What ??
![Page 18: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/18.jpg)
Difficult to Enumerate All Possibilities Though
What is the service ceiling for a PAC750?
![Page 19: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/19.jpg)
WordNet
wingspan
length
diameter radius altitude
ceiling
![Page 20: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/20.jpg)
WordNet For Answer Typing
wingspan
length
diameter radius altitude
ceiling
NUMBER
What is the service ceiling for a PAC750?
![Page 21: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/21.jpg)
Lecture Outline1. Answer Preferences
Question Analysis Type identification Learning Answering Typing
2. Answer ContextLearning Context Similarity Alignment Surface Text Patterns
![Page 22: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/22.jpg)
Answer Typing gives the Preference…
From Answer Typing, we have the preferences imposed by the question
But in order to use those preferences, we must have a way to detect potential candidate answers
![Page 23: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/23.jpg)
Some are Simple…
Number [0-9]+ Date ($month) ($day) ($year) Age 0 – 100
![Page 24: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/24.jpg)
… Others Complicated Who shot Martin Luther King?
– Person preferenceRequires a Named Entity Identifier
Who saved Chrysler from bankruptcy?– Not just confined to people…– Need a Tagger to identify appropriate candidates
![Page 25: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/25.jpg)
Use WordNet for Type Identification
“What 20th century poet wrote Howl?”
writer
poet
Ginsburg FrostRilke
Candidate Set
communicator
![Page 26: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/26.jpg)
Simple Answer Extraction
How old is General Clark?
Age
General Clark, from Little Rock, Arkansas, turns 58 after serving
36 years in the service, this December 23, 2002.
General Clark, from Little Rock, Arkansas, turns 58 after serving
36 years in the service, this December 23, 2002.
Age Tagger
![Page 27: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/27.jpg)
Lecture Outline1. Answer Preferences
Question Analysis Type identification Learning Answering Typing
2. Answer ContextLearning Context Similarity Alignment Surface Text Patterns
![Page 28: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/28.jpg)
Learning Answer Typing What is desired is a model which
predicts P(type|question) Usually a variety of possible types
– Who Person (“Who shot Kennedy?” Oswald) Organization (“Who rescued Chrysler from
bankruptcy?” The Government) Location (“Who won the Superbowl?” New
England)
![Page 29: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/29.jpg)
What training data? Annotated Questions
– “Who shot Kennedy” [PERSON] Problems :
– Expensive to annotate– Must be redone, every time the tag set is
devised
![Page 30: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/30.jpg)
Trivia Questions! Alternatively, use unannotated Trivia
Questions– Q: “Who shot Kennedy”– A: Lee Harvey Oswald
Run your Type-Tagger over the answers, to get tags– A: Lee Harvey Oswald [ PERSON]
![Page 31: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/31.jpg)
MI Model From tags, you can build a MI model
– Predict from the question head-word MI(Question Head Word, Type Tag)
= P(Type Tag | QuestionHeadWord)
---------------------------------------------
P(Type Tag)
– From this you can judge the fit of a question/word pair
– (Mann 2001)
![Page 32: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/32.jpg)
MaxEnt Model Rather than just use head word alone train
on the entire set of words, and build a Maximum Entropy model to combine features suggested by the entire phrase
“What was the year in which Castro was born?”
(Ittycheriah et al. 2001)
![Page 33: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/33.jpg)
Maybe you don’t even need training data!
Looking at occurrences of words in text, look at what types occur next to them
Use these co-occurrence statistics to determine appropriate type of answer
(Prager et al. 2002)
![Page 34: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/34.jpg)
Lecture Outline1. Answer Preferences
Question Analysis Type identification Learning Answering Typing
2. Answer ContextLearning Context Similarity Alignment Surface Text Patterns
![Page 35: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/35.jpg)
Is Answer Typing Enough?
Even when you’ve found the correct sentence, and know the type of the answer a lot of ambiguity in the answer still remains
Some experiments show that in every sentence, around 2/3 choices of appropriate type for a sentence which answers a question
For high precision systems, this is unacceptable
![Page 36: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/36.jpg)
Answer Context
Who shot Martin Luther King?
Answer Preference Answer Context
![Page 37: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/37.jpg)
Using Context Many systems simply look for an
answer of the correct type in a context which seems appropriate– Many matching keywords– Perhaps using query expansion
![Page 38: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/38.jpg)
Another alternative If the question is “Who shot Kennedy”
Search for all exact phrases matches “X shot Kennedy”
And simple alternations “Kennedy was shot by X”
(Brill et al. 2001)
![Page 39: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/39.jpg)
Beyond… The first step beyond simple keyword
matching, is to use relative position information
One way of doing this is to use alignment information
![Page 40: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/40.jpg)
Lecture Outline1. Answer Preferences
Question Analysis Type identification Learning Answering Typing
2. Answer ContextLearning Context Similarity Alignment Surface Text Patterns
![Page 41: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/41.jpg)
Local Alignment
Who shot Kennedy?
Jack assassinated Oswald, the man who shot Kennedy, and was Mrs. Ruby’s Husband.
Three Potential Candidates by type
![Page 42: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/42.jpg)
Local Alignment
Who shot Kennedy?
Jack assassinated Oswald, the man who shot Kennedy, and was Mrs. Ruby’s Husband.
Matching Context
Question Head word
![Page 43: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/43.jpg)
Local Alignment
Who shot Kennedy?
Jack assassinated Oswald, the man who shot Kennedy, and was Mrs. Ruby’s Husband.
Anchor word
![Page 44: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/44.jpg)
Local Alignment
Who shot Kennedy?
Jack assassinated Oswald, the man who shot Kennedy, and was Mrs. Ruby’s Husband.
Potential alignments
![Page 45: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/45.jpg)
Local Alignment
Who shot Kennedy?
Jack assassinated Oswald, the man who shot Kennedy, and was Mrs. Ruby’s Husband.
One Alignment
Three Alignment Features :
![Page 46: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/46.jpg)
Local Alignment
Who shot Kennedy?
Jack assassinated Oswald, the man who shot Kennedy, and was Mrs. Ruby’s Husband.
One Alignment
Three Alignment Features :
1. Dws : Distance between Question Head word and Anchorin the sentence
2
![Page 47: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/47.jpg)
Local Alignment
Who shot Kennedy?
Jack assassinated Oswald, the man who shot Kennedy, and was Mrs. Ruby’s Husband.
Three Alignment Features :
2. Dwq Distance between Question Head word and AnchorIn the question
1
![Page 48: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/48.jpg)
Local Alignment
Who shot Kennedy?
Jack assassinated Oswald, the man who shot Kennedy, and was Mrs. Ruby’s Husband.
Three Alignment Features :
3. R Has the Head Word changed position?
Headword position flipped
![Page 49: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/49.jpg)
Build a Statistical Model
Pr (answer | question, sentence) = Pr ( Dws | answer, question, sentence)
*Pr(Dwq | answer, question, sentence)
*Pr(R | answer, question, sentence)
• and if unsure about type preference, can add in a term there
![Page 50: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/50.jpg)
In essence, this local alignment model gives a robust method for using the context of the question to pick out the correct answer from a given sentence containing an answer
![Page 51: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/51.jpg)
Surface text Patterns Categorize question into what kind of
data it is looking for Use templates to build specialized
models Use resulting “surface text patterns”
for searching
![Page 52: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/52.jpg)
Birthday Templates
W. A. Mozart 1756
I. Newton 1642
M. Gandhi 1869
V. S. Naipaul 1932
Bill Gates 1951
![Page 53: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/53.jpg)
Web Search to generate patterns
Web pages w/“Mozart” “1756”
Sentences with “Mozart” “1756”
Substrings with “Mozart” “1756”
![Page 54: Question Answering Gideon Mann Johns Hopkins University gsm@cs.jhu.edu](https://reader034.vdocuments.net/reader034/viewer/2022051400/551a81c9550346e0158b4a1f/html5/thumbnails/54.jpg)
How can we pick good patterns?
Frequent ones may be too general
Infrequent ones not that useful
Want precise, specific ones
Use held out templates to evaluate patterns