focused entailment graphs for open ie propositions

Focused Entailment Graphs for Open IE

PropositionsOmer Levy Ido Dagan Jacob Goldberger

Bar-Ilan University, Israel

Open IE• Extracts propositions from text

“…which makes aspirin relieve headaches.”

• No supervision• No pre-defined schema

What’s missing in Open IE?• Structure

• Open IE does not consolidate natural language expressions

relieve headache treat headache

Adding Structure to Open IEWhich structure?• Build a graph of Open IE propositions and their semantic relations

Adding Structure to Open IEWhich structure?• Build a graph of Open IE propositions and their entailment relations

Why entailment?• Merges paraphrases into mutual entailment cliques

aspirin relieves headache aspirin treats headache

• Organizes information hierarchically from specific to generalaspirin relieves headache painkiller relieves headache

aspirin, eliminate, headacheaspirin, cure, headache

headache, control with, aspirindrug, relieve, headache

drug, treat, headache

analgesic, banish, headache

headache, respond to, painkillerheadache, treat with, caffeine

coffee, help, headache

tea, soothe, headache

Original Open IE Output

aspirin, eliminate, headache

aspirin, cure, headache

headache, control with, aspirin

drug, relieve, headache

drug, treat, headache

analgesic, banish, headache

headache, respond to, painkillerheadache, treat with, caffeine

coffee, help, headache

tea, soothe, headache

Consolidated Open IE Output

Semantic Applications• Example: Structured Queries

• “What relieves headaches?”

Semantic Applications• Example: Structured Queries

• “What relieves headaches?”

aspirin, eliminate, headache

aspirin, cure, headache

headache, control with, aspirin

drug, relieve, headache

drug, treat, headache

analgesic, banish, headache

headache, respond to, painkillerheadache, treat with, caffeine

coffee, help, headache

tea, soothe, headache

Structured Query:

aspirin, eliminate, headache

aspirin, cure, headache

headache, control with, aspirin

drug, relieve, headache

drug, treat, headache

analgesic, banish, headache

headache, respond to, painkillerheadache, treat with, caffeine

coffee, help, headache

tea, soothe, headache

Structured Query:

aspirin

drug

analgesic

painkillercaffeine

coffee

tea

Structured Query:

Our Contributions• Structuring Open IE with Proposition Entailment Graphs

• Dataset: 30 gold-standard graphs, 1.5 million entailment annotations

• Algorithm for constructing Focused Proposition Entailment Graphs

• Analysis: Predicate entailment is not quite what we thought

Proposition Entailment Graphs

Related Work: Predicate Entailment Graphs• Berant et al. (2010,2011,2012)

• We extend Berant et al.’s work from predicates to propositions

Focused Proposition Entailment Graphs• Nodes: Open IE propositions

• Edges: Textual Entailment

Focused Proposition Entailment Graphs• Assumptions: Binary Propositions and Common Topic

• Binary Propositions

• Focused on a common topic

Focused Proposition Entailment Graphs• Assumptions: Binary Propositions and Common Topic

• Binary Propositions

• Focused on a common topic

aspirin, eliminate, headache

aspirin, cure, headache

headache, control with, aspirin

drug, relieve, headache

drug, treat, headache

analgesic, banish, headache

headache, respond to, painkillerheadache, treat with, caffeine

coffee, help, headache

tea, soothe, headache

aspirin, eliminate, headache

aspirin, cure, headache

headache, control with, aspirin

drug, relieve, headache

drug, treat, headache

analgesic, banish, headache

headache, respond to, painkillerheadache, treat with, caffeine

coffee, help, headache

tea, soothe, headache

Focused Proposition Entailment Graphs• Edges: Textual Entailment

Proposition Entailment• Simpler than sentence-level entailment• More complicated than lexical entailment• Enables investigation of inference phenomena in an isolated manner

Constructing Proposition Entailment Graphs

Task Definition:

Given a set of propositions ,find all their entailment edges.

Dataset

Dataset: High-Quality Open IE Propositions• Google’s Syntactic N-grams• Based on millions of books

• Filter for subject-verb-object• Including prepositional objects and passive

• Result: 68 million high-quality propositions

Dataset: Annotating Entailment Graphs• Select 30 healthcare topics• antibiotic, caffeine, insomnia, scurvy, …

• Collect a set of propositions focused on each topic

• Manually clean noisy extractions• Retaining 200 propositions per graph (average)

• Efficiently annotate entailment• 1.5 million entailment judgments

Algorithm

How do we recognize proposition entailment?

.

?

How do we recognize proposition entailment?

.

Observation: propositions entail their lexical components entail

How do we recognize proposition entailment?

.

Observation: propositions entail their lexical components entail

How do we recognize proposition entailment?

.

Proposition entailment is reduced to lexical entailment in context

𝑒=𝜎 (𝑤⋅ 𝑓 )Lexical Entailment(Logistic)

Lexical EntailmentLexical Entailment Features

𝑓 1

𝑒

𝑓 2 𝑓 3

Lexical Entailment(Logistic)

𝑒=𝜎 (𝑤⋅ 𝑓 )

Lexical EntailmentFeatures• WordNet Relations• UMLS• Distributional Similarity• String Edit Distance

Lexical Entailment Features

𝑓 1

𝑒

𝑓 2 𝑓 3

Supervision

From Lexical to Proposition Entailment

Lexical Entailment(Logistic)

𝑒=𝜎 (𝑤⋅ 𝑓 )

Lexical Entailment Features

𝑓 1

𝑒

𝑓 2 𝑓 3

Supervision

𝑎=𝜎 (𝑤𝑎⋅ 𝑓 𝑎 )Argument Entailment(Logistic)

𝑝=𝜎 (𝑤𝑝 ⋅ 𝑓 𝑝)Predicate Entailment(Logistic)

From Lexical to Proposition Entailment

Argument Entailment Features

𝑓 𝑎1

𝑎

𝑓 𝑎2 𝑓 𝑎3𝑓 𝑝1

𝑝

𝑓 𝑝2 𝑓 𝑝3Predicate Entailment Features

SupervisionSupervision

𝑎=𝜎 (𝑤𝑎⋅ 𝑓 𝑎 )Argument Entailment(Logistic)

𝑝=𝜎 (𝑤𝑝 ⋅ 𝑓 𝑝)Predicate Entailment(Logistic)

From Lexical to Proposition Entailment

Argument Entailment Features

𝑓 𝑎1

𝑎

𝑓 𝑎2 𝑓 𝑎3𝑓 𝑝1

𝑝

𝑓 𝑝2 𝑓 𝑝3Predicate Entailment Features

SupervisionSupervision

𝑠Proposition Entailment(Conjunction) 𝑠=𝑝⋅𝑎

Following Snow (2005), Berant (2012)

𝑎=𝜎 (𝑤𝑎⋅ 𝑓 𝑎 )Argument Entailment(Logistic)

𝑝=𝜎 (𝑤𝑝 ⋅ 𝑓 𝑝)Predicate Entailment(Logistic)

Distant Supervision (WordNet)?Argument Entailment Features

𝑓 𝑎1

𝑎

𝑓 𝑎2 𝑓 𝑎3𝑓 𝑝1

𝑝

𝑓 𝑝2 𝑓 𝑝3Predicate Entailment Features

WordNetWordNet

𝑠Proposition Entailment(Conjunction) 𝑠=𝑝⋅𝑎

Argument Entailment(Logistic)

Proposition Entailment(Conjunction) 𝑠=𝑝⋅𝑎

𝑎=𝜎 (𝑤𝑎⋅ 𝑓 𝑎 )𝑝=𝜎 (𝑤𝑝 ⋅ 𝑓 𝑝)Predicate Entailment(Logistic)

Direct Supervision (30 Annotated Graphs)

Argument Entailment Features

𝑓 𝑎1

𝑎

𝑓 𝑎2 𝑓 𝑎3𝑓 𝑝1

𝑝

𝑓 𝑝2 𝑓 𝑝3Predicate Entailment Features

Annotated Graphs

𝑠

Proposition Entailment(Conjunction) 𝑠=𝑝⋅𝑎

Direct Supervision (30 Annotated Graphs)

Argument Entailment Features

𝑓 𝑎1

𝑎

𝑓 𝑎2 𝑓 𝑎3𝑓 𝑝1

𝑝

𝑓 𝑝2 𝑓 𝑝3Predicate Entailment Features

𝑠

Hidden Layer

Annotated Graphs

Flat ModelArgument Entailment Features

𝑓 𝑎1 𝑓 𝑎2 𝑓 𝑎3

Proposition Entailment(Logistic)

𝑠=𝜎 (𝑤𝑝 ⋅ 𝑓 𝑝+𝑤𝑎⋅ 𝑓 𝑎)

𝑓 𝑝1 𝑓 𝑝2 𝑓 𝑝3Predicate Entailment Features

𝑠Annotated Graphs

Compared Methods• Component-Level Distant Supervision (WordNet)• Predicates & Arguments• Predicates Only• Arguments Only

• Proposition-Level Direct Supervision (30 Annotated Graphs)• Hierarchical (our method)• Flat

• All methods used Berant et al.’s Global Optimization method

Results

Direct Supervision: Flat vs Hierarchical• Hierarchal model performs

better than flat model

• Better to model predicate and argument entailment separately

50%

55%

60%

65%

70%

Hierarchical(Our Method)

63.7%Flat

61.6%

Perfo

rman

ce (F

1)

Distant vs Direct Supervision• Direct supervision is better

• Although WordNet provides more training examples

50%

55%

60%

65%

70%

Hierarchical(Our Method)

63.7%Flat

61.6%

BestDistant

(ArgumentsOnly)

59.7%

Perfo

rman

ce (F

1)

Predicate Entailment with Distant Supervision• Ignoring predicates improves

distant supervision baselines

0%

10%

20%

30%

40%

50%

60%

70%

ArgumentsOnly

59.7%

PredicatesOnly

8.0%

Predicates &Arguments

7.2%

Perfo

rman

ce (F

1)

Are WordNet relations capturing real-world predicate entailments?

Predicate Entailment vs WordNet RelationsOver a predicate inference subset, how many predicate entailments are covered by WordNet?

• Positive indicators• synonyms, hypernyms, entailment

Positive12%

Negative15%

None

74%

Why isn’t WordNet capturing predicate entailment?

Predicate Entailment vs WordNet RelationsOver a predicate inference subset, how many predicate entailments are covered by WordNet?

• Positive indicators• synonyms, hypernyms, entailment

• Negative Indicators• antonyms, hyponyms, cohyponyms

Positive12%

Negative15%

None

74%

Predicate Entailment is Context-Sensitive

The words do not necessarily entail,but the situations do.

Predicate Entailment is Context-Sensitive

The words do not necessarily entail,but the situations do.

Investigating Context-Sensitive Entailment• Recent work on context-sensitive lexical inference• e.g. (Melamud et al., 2013)

• Previous datasets• Lexical substitution (McCarthy and Navigli, 2007)• Predicate inference (Zeichner et al., 2012)

• We offer a new dataset of real-world lexical entailments in context!• Sample: synthetic vs naturally occurring• Size: several thousands vs 1.5 million

Conclusion

Conclusion• Structuring Open IE with Proposition Entailment Graphs

• Algorithm for constructing Focused Proposition Entailment Graphs

• Analysis: Predicate entailment is extremely context-sensitive

• Dataset: 1.5 million proposition entailment decisions

Thank you for listening!