focused entailment graphs for open ie propositions
DESCRIPTION
Focused Entailment Graphs for Open IE Propositions. Omer Levy Ido DaganJacob Goldberger Bar- Ilan University, Israel. Open IE. Extracts propositions from text “…which makes aspirin relieve headaches.” No supervision No pre-defined schema. What’s missing in Open IE?. Structure - PowerPoint PPT PresentationTRANSCRIPT
Focused Entailment Graphs for Open IE
PropositionsOmer Levy Ido Dagan Jacob Goldberger
Bar-Ilan University, Israel
Open IE• Extracts propositions from text
“…which makes aspirin relieve headaches.”
• No supervision• No pre-defined schema
What’s missing in Open IE?• Structure
• Open IE does not consolidate natural language expressions
relieve headache treat headache
Adding Structure to Open IEWhich structure?• Build a graph of Open IE propositions and their semantic relations
Adding Structure to Open IEWhich structure?• Build a graph of Open IE propositions and their entailment relations
Why entailment?• Merges paraphrases into mutual entailment cliques
aspirin relieves headache aspirin treats headache
• Organizes information hierarchically from specific to generalaspirin relieves headache painkiller relieves headache
aspirin, eliminate, headacheaspirin, cure, headache
headache, control with, aspirindrug, relieve, headache
drug, treat, headache
analgesic, banish, headache
headache, respond to, painkillerheadache, treat with, caffeine
coffee, help, headache
tea, soothe, headache
Original Open IE Output
aspirin, eliminate, headache
aspirin, cure, headache
headache, control with, aspirin
drug, relieve, headache
drug, treat, headache
analgesic, banish, headache
headache, respond to, painkillerheadache, treat with, caffeine
coffee, help, headache
tea, soothe, headache
Consolidated Open IE Output
Semantic Applications• Example: Structured Queries
• “What relieves headaches?”
Semantic Applications• Example: Structured Queries
• “What relieves headaches?”
aspirin, eliminate, headache
aspirin, cure, headache
headache, control with, aspirin
drug, relieve, headache
drug, treat, headache
analgesic, banish, headache
headache, respond to, painkillerheadache, treat with, caffeine
coffee, help, headache
tea, soothe, headache
Structured Query:
aspirin, eliminate, headache
aspirin, cure, headache
headache, control with, aspirin
drug, relieve, headache
drug, treat, headache
analgesic, banish, headache
headache, respond to, painkillerheadache, treat with, caffeine
coffee, help, headache
tea, soothe, headache
Structured Query:
aspirin
drug
analgesic
painkillercaffeine
coffee
tea
Structured Query:
Our Contributions• Structuring Open IE with Proposition Entailment Graphs
• Dataset: 30 gold-standard graphs, 1.5 million entailment annotations
• Algorithm for constructing Focused Proposition Entailment Graphs
• Analysis: Predicate entailment is not quite what we thought
Proposition Entailment Graphs
Related Work: Predicate Entailment Graphs• Berant et al. (2010,2011,2012)
• We extend Berant et al.’s work from predicates to propositions
Focused Proposition Entailment Graphs• Nodes: Open IE propositions
• Edges: Textual Entailment
Focused Proposition Entailment Graphs• Assumptions: Binary Propositions and Common Topic
• Binary Propositions
• Focused on a common topic
Focused Proposition Entailment Graphs• Assumptions: Binary Propositions and Common Topic
• Binary Propositions
• Focused on a common topic
aspirin, eliminate, headache
aspirin, cure, headache
headache, control with, aspirin
drug, relieve, headache
drug, treat, headache
analgesic, banish, headache
headache, respond to, painkillerheadache, treat with, caffeine
coffee, help, headache
tea, soothe, headache
aspirin, eliminate, headache
aspirin, cure, headache
headache, control with, aspirin
drug, relieve, headache
drug, treat, headache
analgesic, banish, headache
headache, respond to, painkillerheadache, treat with, caffeine
coffee, help, headache
tea, soothe, headache
Focused Proposition Entailment Graphs• Edges: Textual Entailment
Proposition Entailment• Simpler than sentence-level entailment• More complicated than lexical entailment• Enables investigation of inference phenomena in an isolated manner
Constructing Proposition Entailment Graphs
Task Definition:
Given a set of propositions ,find all their entailment edges.
Dataset
Dataset: High-Quality Open IE Propositions• Google’s Syntactic N-grams• Based on millions of books
• Filter for subject-verb-object• Including prepositional objects and passive
• Result: 68 million high-quality propositions
Dataset: Annotating Entailment Graphs• Select 30 healthcare topics• antibiotic, caffeine, insomnia, scurvy, …
• Collect a set of propositions focused on each topic
• Manually clean noisy extractions• Retaining 200 propositions per graph (average)
• Efficiently annotate entailment• 1.5 million entailment judgments
Algorithm
How do we recognize proposition entailment?
.
?
How do we recognize proposition entailment?
.
Observation: propositions entail their lexical components entail
How do we recognize proposition entailment?
.
Observation: propositions entail their lexical components entail
How do we recognize proposition entailment?
.
Proposition entailment is reduced to lexical entailment in context
𝑒=𝜎 (𝑤⋅ 𝑓 )Lexical Entailment(Logistic)
Lexical EntailmentLexical Entailment Features
𝑓 1
𝑒
𝑓 2 𝑓 3
Lexical Entailment(Logistic)
𝑒=𝜎 (𝑤⋅ 𝑓 )
Lexical EntailmentFeatures• WordNet Relations• UMLS• Distributional Similarity• String Edit Distance
Lexical Entailment Features
𝑓 1
𝑒
𝑓 2 𝑓 3
Supervision
From Lexical to Proposition Entailment
Lexical Entailment(Logistic)
𝑒=𝜎 (𝑤⋅ 𝑓 )
Lexical Entailment Features
𝑓 1
𝑒
𝑓 2 𝑓 3
Supervision
𝑎=𝜎 (𝑤𝑎⋅ 𝑓 𝑎 )Argument Entailment(Logistic)
𝑝=𝜎 (𝑤𝑝 ⋅ 𝑓 𝑝)Predicate Entailment(Logistic)
From Lexical to Proposition Entailment
Argument Entailment Features
𝑓 𝑎1
𝑎
𝑓 𝑎2 𝑓 𝑎3𝑓 𝑝1
𝑝
𝑓 𝑝2 𝑓 𝑝3Predicate Entailment Features
SupervisionSupervision
𝑎=𝜎 (𝑤𝑎⋅ 𝑓 𝑎 )Argument Entailment(Logistic)
𝑝=𝜎 (𝑤𝑝 ⋅ 𝑓 𝑝)Predicate Entailment(Logistic)
From Lexical to Proposition Entailment
Argument Entailment Features
𝑓 𝑎1
𝑎
𝑓 𝑎2 𝑓 𝑎3𝑓 𝑝1
𝑝
𝑓 𝑝2 𝑓 𝑝3Predicate Entailment Features
SupervisionSupervision
𝑠Proposition Entailment(Conjunction) 𝑠=𝑝⋅𝑎
Following Snow (2005), Berant (2012)
𝑎=𝜎 (𝑤𝑎⋅ 𝑓 𝑎 )Argument Entailment(Logistic)
𝑝=𝜎 (𝑤𝑝 ⋅ 𝑓 𝑝)Predicate Entailment(Logistic)
Distant Supervision (WordNet)?Argument Entailment Features
𝑓 𝑎1
𝑎
𝑓 𝑎2 𝑓 𝑎3𝑓 𝑝1
𝑝
𝑓 𝑝2 𝑓 𝑝3Predicate Entailment Features
WordNetWordNet
𝑠Proposition Entailment(Conjunction) 𝑠=𝑝⋅𝑎
Argument Entailment(Logistic)
Proposition Entailment(Conjunction) 𝑠=𝑝⋅𝑎
𝑎=𝜎 (𝑤𝑎⋅ 𝑓 𝑎 )𝑝=𝜎 (𝑤𝑝 ⋅ 𝑓 𝑝)Predicate Entailment(Logistic)
Direct Supervision (30 Annotated Graphs)
Argument Entailment Features
𝑓 𝑎1
𝑎
𝑓 𝑎2 𝑓 𝑎3𝑓 𝑝1
𝑝
𝑓 𝑝2 𝑓 𝑝3Predicate Entailment Features
Annotated Graphs
𝑠
Proposition Entailment(Conjunction) 𝑠=𝑝⋅𝑎
Direct Supervision (30 Annotated Graphs)
Argument Entailment Features
𝑓 𝑎1
𝑎
𝑓 𝑎2 𝑓 𝑎3𝑓 𝑝1
𝑝
𝑓 𝑝2 𝑓 𝑝3Predicate Entailment Features
𝑠
Hidden Layer
Annotated Graphs
Flat ModelArgument Entailment Features
𝑓 𝑎1 𝑓 𝑎2 𝑓 𝑎3
Proposition Entailment(Logistic)
𝑠=𝜎 (𝑤𝑝 ⋅ 𝑓 𝑝+𝑤𝑎⋅ 𝑓 𝑎)
𝑓 𝑝1 𝑓 𝑝2 𝑓 𝑝3Predicate Entailment Features
𝑠Annotated Graphs
Compared Methods• Component-Level Distant Supervision (WordNet)• Predicates & Arguments• Predicates Only• Arguments Only
• Proposition-Level Direct Supervision (30 Annotated Graphs)• Hierarchical (our method)• Flat
• All methods used Berant et al.’s Global Optimization method
Results
Direct Supervision: Flat vs Hierarchical• Hierarchal model performs
better than flat model
• Better to model predicate and argument entailment separately
50%
55%
60%
65%
70%
Hierarchical(Our Method)
63.7%Flat
61.6%
Perfo
rman
ce (F
1)
Distant vs Direct Supervision• Direct supervision is better
• Although WordNet provides more training examples
50%
55%
60%
65%
70%
Hierarchical(Our Method)
63.7%Flat
61.6%
BestDistant
(ArgumentsOnly)
59.7%
Perfo
rman
ce (F
1)
Predicate Entailment with Distant Supervision• Ignoring predicates improves
distant supervision baselines
0%
10%
20%
30%
40%
50%
60%
70%
ArgumentsOnly
59.7%
PredicatesOnly
8.0%
Predicates &Arguments
7.2%
Perfo
rman
ce (F
1)
Are WordNet relations capturing real-world predicate entailments?
Predicate Entailment vs WordNet RelationsOver a predicate inference subset, how many predicate entailments are covered by WordNet?
• Positive indicators• synonyms, hypernyms, entailment
Positive12%
Negative15%
None
74%
Why isn’t WordNet capturing predicate entailment?
Predicate Entailment vs WordNet RelationsOver a predicate inference subset, how many predicate entailments are covered by WordNet?
• Positive indicators• synonyms, hypernyms, entailment
• Negative Indicators• antonyms, hyponyms, cohyponyms
Positive12%
Negative15%
None
74%
Predicate Entailment is Context-Sensitive
The words do not necessarily entail,but the situations do.
Predicate Entailment is Context-Sensitive
The words do not necessarily entail,but the situations do.
Investigating Context-Sensitive Entailment• Recent work on context-sensitive lexical inference• e.g. (Melamud et al., 2013)
• Previous datasets• Lexical substitution (McCarthy and Navigli, 2007)• Predicate inference (Zeichner et al., 2012)
• We offer a new dataset of real-world lexical entailments in context!• Sample: synthetic vs naturally occurring• Size: several thousands vs 1.5 million
Conclusion
Conclusion• Structuring Open IE with Proposition Entailment Graphs
• Algorithm for constructing Focused Proposition Entailment Graphs
• Analysis: Predicate entailment is extremely context-sensitive
• Dataset: 1.5 million proposition entailment decisions
Thank you for listening!