robust shallow semantic parsing of text › ~dipanjan › ttic_presentation.pdf · semantic parsing...

Dipanjan DasCarnegie Mellon University

Toyota Technological Institute at Chicago

February 14, 2012

Robust Shallow Semantic Parsing of Text

Natural Language Understanding

I want to go to Chicago on Sunday

I want to go to on Sunday

P V ADP V ADP N ADP N

Shallow Syntax:Part-of-Speech Tagging

Chicago

Deeper Syntax:Dependency parsing

Chicago

Shallow Semantics: Frames and Roles

Chicago

Encodes an eventor

scenario

Chicago

participant orrole

for the frame

Chicago

Experiencer

Chicago

Focus of this talk!(Das, Schneider, Chen and Smith, NAACL 2010;

Das and Smith, ACL 2011)

Experiencer

Chicago

1. Why semantic analysis?

3. Semi-supervised learning for robustness

2. Statistical models for structure prediction

ApplicationsMotivation Choice offormalism

Argument identificationFrame identification

(use of latent variables) (dual decomposition)

Novel graph-based learning algorithms16

Motivation ApplicationsChoice offormalism

Bengal ’s massive stock of food was reduced to nothing

Bengal ’s massive stock of food was reduced to nothingN X A N ADP N V V ADP N

Large body of research on syntax, includingDas and Petrov, ACL 2011

Cohen, Das and Smith, EMNLP 2011Martins, Das, Smith and Xing, EMNLP 2008

storeor

financial entity?

Store of what?Of what size?Whose store?

What was reduced?

To what?

What was reduced?

To what?

AUSE_CHANGE_OF_POSITION_ON_A_SCALE

Origins: (Computational) Linguistics

Case Grammar(“The Case for Case”, Fillmore, 1968)

I gave some money to him

(cases are words/phrases required by a predicate)

Agent Object Beneficiary

(cases are words/phrases required by a predicate)

Agent Object Beneficiary

Semantic valency of a predicate

Correlation with syntax(e.g. subject and object)

Obligatory cases / optional cases

Slide idea taken from Brendan O’Connor

Case Grammar(“The Case for Case”,

Fillmore, 1968)

Frames(“A Framework

for Representing Knowledge”, Minsky, 1975)

Fillmore, 1968)

Frame Semantics(“Frame Semantics”,

Fillmore, 1982)

Fillmore, 1968)

Fillmore, 1982)

Relates the meaning of a word with world knowledge

(e.g. gave evokes a IVING frame;

it has several participating roles;the frame is evoked by other words,

such as bequeath, contribute, donate)

Fillmore, 1968)

Fillmore, 1982)

Fillmore, 1968)

Relates the meaning of a word with world knowledge

(e.g. gave evokes a IVING frame;

it has several participating roles;the frame is evoked by other words,

such as bequeath, contribute, donate)

Fillmore, 1968)

Fillmore, 1982)

FrameNetPropBankVerbNet

NomBankOntoNotes

Datasets

Fillmore, 1968)

Fillmore, 1982)

NomBankOntoNotes

Datasets

Data-Driven ShallowSemantic Parsing

Fillmore, 1968)

Fillmore, 1982)

NomBankOntoNotes

Scripts(“Scripts, Plans,

Goals and Understanding”, Schank and Abelson, 1977)

MUCACE

Datasets

Information Extraction(template filling)

Fillmore, 1968)

Fillmore, 1982)

NomBankOntoNotes

MUCACE

Datasets

Fillmore, 1968)

Fillmore, 1982)

NomBankOntoNotes

MUCACE

Datasets

Fillmore, 1968)

Fillmore, 1982)

NomBankOntoNotes

MUCACE

Datasets

structurally similar!

Why this Linguistic Formalism?

shallow

PropBank-style Semantic Role Labeling(Pradhan et al., 2008; Punyakanok et al., 2008)

shallow

symbolic set of semantic roles(six total)

Verb-specific meaning for these labels

Conflates the meaning of different roles due to oversimplification

(Yi et al., 2007)

shallow

deepSemantic Parsing into Logical Forms

(Ge and Mooney, 2005; Zettlemoyer and Collins., 2005)

shallow

What states border the state that borders the most states

Semantic Parsing into Logical Forms(Ge and Mooney, 2005; Zettlemoyer and Collins., 2005)

shallow

Trained on very restricted domains

Poor lexical coverage

shallow

Frame-Semantic Parsing(Gildea and Jurafsky, 2002, Johansson and Nugues, 2007; this work)

shallow

Does not model quantification or negation

unlike logical forms

shallow

Deeper than PropBank-style semantic role labeling

shallow

Models all types of part-of-speech categories

shallow

Larger lexical coverage than logical form parsers

shallow

Lexicon actively increasing in size

ApplicationsMotivation ApplicationsChoice offormalism

Question Answering

Whose stock of food was diminished ?

Possible Applications

Question Answering

Whose stock of food was diminished ?reserve

Question Answering

Whose stock of food was diminished ?reserve

Bilotti et al. (2007)

Information Extraction

In 1997, France's stock of unirradiated civil plutonium increased to 72 tons.

Saudi Arabia has 267 billion barrels in reserves of oil.

Does Egypt have stockpiles of biological weapons?

Bengal’s massive stock of food was reduced to nothing.

Possessor Desc Resource

Bengal Massive food

France - unirradiated civil plutonium

Saudi Arabia 267 billion barrels oil

Egypt - Biological weapons

Multilingual Applications

��

Multilingual KBs

��

Cross lingual IR

Machine translation

(Das, Schneider, Chen and Smith, NAACL 2010)

Structure of Lexicon and Data

PLACING

framecore roles

non-core roles

PLACING

framecore roles

non-core roles

excludes relationship

PLACING

framecore roles

non-core roles

archive.V, arrange.V, bag.V, bestow.V bin.V

predicates

excludes relationship

PLACING

DISPERSAL

Individuals

Distance

TRANSITIVE_ACTION

Patient

INSTALLING

Component

Fixed_location

STORING

Location

Possessor

Resource

Supply

Descriptor

PLACING

DISPERSAL

Individuals

Distance

TRANSITIVE_ACTION

Patient

INSTALLING

Component

Fixed_location

STORING

Location

Possessor

Resource

Supply

Descriptor

PLACING

DISPERSAL

Individuals

Distance

TRANSITIVE_ACTION

Patient

INSTALLING

Component

Fixed_location

STORING

Location

Possessor

Resource

Supply

Descriptor

inheritance

used by

Benchmark Dataset(SemEval 2007)

665 frames720 role labels

8.4K unique predicate types

Training set:2.2K sentences

11.2K predicate tokens

Test set:120 sentences

1. 1K predicate tokens86

For comparison with past state of the art

(very small dataset)

New Data(FrameNet 1.5, 2010)

Frame Identification

ambiguous

Find the best among all the frames

Direct modeling using logistic regression

Two problems:1. Unable to model unknown

predicates at test time

2. Number of features:

2. Number of features: ≈ 50 million

Logistic regression with a latent variable

Predicates evoking a frame in supervised data, e.g.cargo.N, inventory.N, reserve.N, stockpile.N, store.N, supply.N

evoke STORE

Does not look at the predicate’s surface form

stock.N

stockpile.N

stock.N

stockpile.N

LexSem = {synonym}

stock.N

stockpile.N

LexSem = {synonym}

If STORE

stockpile.N

synonym LexSem

stock.N

stockpile.N

LexSem = {synonym}

If STORE

stockpile.N

synonym LexSem

comes from WordNet!

Number of features:

≈ 500 K

(1% of the features we had before)

Number of features:

≈ 500 K

(1% of the features we had before)

Aside:

Probabilistic modeling of language meaning using syntax, lexical semantics and latent structure:

Paraphrase identification (Das and Smith, ACL 2009)

Training:

Maximum conditional log-likelihood(batch training with L-BFGS)

Fast inference:

If predicate is unseen,

otherwise,

Results

Benchmark

60.464.0

UTD LTH ThisWork

F-Measure

New Data

Results

Benchmark

60.464.0

UTD LTH ThisWork

F-Measure

New Data

Bejan and Hathaway, (2007)

Results

Benchmark

60.464.0

UTD LTH ThisWork

F-Measure

New Data

Johansson and Nugues, (2007)

Results

Benchmark

60.464.0

UTD LTH ThisWork

F-Measure

New Data

Results

Benchmark

60.464.0

UTD LTH ThisWork

F-Measure

auto predicates

New Data

Results

Benchmark

60.464.0

UTD LTH ThisWork

ThisWork

F-Measure

auto predicates givenpredicates

New Data

Results

Benchmark New Data

60.464.0

UTD LTH ThisWork

ThisWork

F-Measure

ThisWork

F-Measure

givenpredicates

Results

Benchmark New Data

60.464.0

UTD LTH ThisWork

ThisWork

F-Measure

ThisWork

F-Measure

givenpredicates

no hidden variable

Argument Identification

Bengal ’s

Bengal

massive stock

of food

massive

Bengal ’s massive stock of food

massive stock of food

Bengal ’s

Bengal

massive stock

of food

massive

Ideal mapping!

Bengal ’s

Bengal

massive stock

of food

massive

Bengal ’s

Bengal

massive stock

of food

massive

Violates overlap constraints

Other types of structural constraints

PLACING

Mutual exclusion constraint

PLACING

If an agent places something, there cannot be a cause role in the sentence

PLACING

AgentThe waiter placed food on the table.

In Kabul, hauling water put food on the table.Cause

SIMILARITY

Dimension

Differentiating_fact

Entity_1

Entity_2

Degree

Requires constraint

difference.N, resemble.V,

unliike.A, vary.V

A mulberry resembles a loganberry.

second entity

first entity

SIMILARITY

Dimension

Entity_1

Entity_2

Degree

Requires constraint

unliike.A, vary.V

A mulberry resembles.

SIMILARITY

Dimension

Entity_1

Entity_2

Degree

Requires constraint

unliike.A, vary.V

Bengal ’s

Bengal

massive stock

of food

massive

A constrained optimization

problem

Bengal ’s

Bengal

massive stock

of food

massive

Bengal ’s

Bengal

massive stock

of food

massive

A constrained optimization problem

a binary variable for each role, span tuple

a binary vector for all role, span tuples

Uniqueness144

Prevents overlap 145

more structural constraints

An integer linear program (ILP)

Punyakanok, Roth and Yih(2008)

Often, very slow solutions

A constrained optimization problemAn integer linear program (ILP)

Fast ILP solvers proprietary

An alternate approach

Dual Decomposition with Alternating Direction Method of Multipliers

developed with colleagues at CMU(Das, Martins and Smith, forthcoming)

basic part:

entire space: all tuples

basic part:

Break down the problem into many small components

e.g. -- find the best span for a role

-- for a sentence position, find the best tuple

-- for a pair of mutually exclusive roles,find the best tuple

basic part:

Break down the problem into many small components

e.g. -- find the best span for a role

-- for a sentence position, find the best tuple

-- for a pair of mutually exclusive roles,find the best tuple

impose agreement between components

For each component, a binary vector:

Total score assigned to :

witness vector for consensus

Primal:

Primal':

Integer constraints relaxed

Primal':

An augmented Lagrangian function

An alternate approachAn augmented Lagrangian function

Saddle point can be found using several decoupled

worker problems

Three types of iterative updates:

1. Lagrange multiplier updates ( )2. Consensus variable updates ( )

3. updates

3. updatesAt decoupled workers

e.g. for each role, we have a worker that imposes a XOR/uniqueness constraint

Projection onto a simplexa simple sort operation

Challenge:define fast, simple workers

Advantages:

1) Significant speedup2) No proprietary solver necessary

Advantages:

CPLEX (ILP) Dual Decomposition

Time to decode the test set in seconds

Advantages:

CPLEX (ILP) Dual Decomposition

Time to decode the test set in seconds

Certificate of Optimality in

>99% of examples

Bengal ’s

Bengal

massive stock

of food

massive

Bengal ’s

Learning?

Bengal ’s

Learning?

Maximum conditional log-likelihoodof local role span pairs

(batch training using L-BFGS)

Results

New Data

Benefit of joint inference175

8283.8

76.4 76.2

79.1 79.8

Local DualDecomposition

Precision

Recall

F-Measure

Results

New Data

Benefit of joint inference

8283.8

76.4 76.2

79.1 79.8

Precision

Recall

F-Measure

501 linguistic violations

Results

New Data

Benefit of joint inference

501 linguistic violations No violations

8283.8

76.4 76.2

79.1 79.8

Precision

Recall

F-Measure

Full Parsing

Final Results

Benchmark New Data

UTD LTH ThisWork

F-Measure

auto predicates

Full Parsing

Final Results

Benchmark New Data

UTD LTH ThisWork

F-Measure

auto predicates

Full Parsing

Final Results

Benchmark New Data

50.253.6

UTD LTH ThisWork

ThisWork

F-Measure

Full Parsing

Final Results

Benchmark New Data

50.253.6

UTD LTH ThisWork

ThisWork

F-Measure

This Work

givenpredicates

F-Measure

Novel graph-based learning algorithms

(Das and Smith, ACL 2011)

405060708090

All Predicates

F-Measure

Results on Unknown Predicates

Unknown Predicates

F-Measure

405060708090

All Predicates

F-Measure

Unknown Predicates

F-Measure

25303540455055606570

All Predicates

F-Measure

25303540455055606570

Unknown Predicates

F-Measure

Full Parsing

Handling Unknown Predicates

Knowledge of only 9,263 predicates in supervised data

However, English has lot more potential predicates(~65,000 in newswire English)

Lexicon expansion using graph-based semi-supervised learning

Build a graph over potential predicates as vertices• compute similarity matrix using co-occurrence statistics

Label distribution at each vertexdistribution over frames that the predicate can evoke

How can label propagation help?

Build a graph over potential predicates as vertices• compute similarity matrix using co-occurrence statistics

Label distribution at each vertexdistribution over frames that the predicate can evoke

How can label propagation help?

Idea very similar to Das and Petrov (ACL 2011): unsupervised lexicon expansion for POS tagging

Example Graph

Seed predicates

Example Graph

Seed predicatesUnseen predicates

Example Graph

Graph Propagation193

Example Graph

Graph Propagation194

Example Graph

Graph Propagation

Continues till convergence...

Brief Overview:Graph-Based Learning

with Labeled and Unlabeled Data

= symmetric weight matrix

labeled datapointsunlabeled datapoints

supervised label distributions

distributions to be found

Label Propagation

Minimize:

Das and Smith, forthcoming 203

Label Propagation

Minimize:

Brings the distribution of observed and induced distributions over labeled vertices closer

Label Propagation

Minimize:

brings the distributions of similarvertices closer

Label Propagation

Minimize:

induces sparse distributionsin each vertex

Constrained Inference

If predicate is seen,

otherwise,

else if predicate is in graph,

Constrained Inference

If predicate is seen,

otherwise,

else if predicate is in graph,

Six times faster inference on unknown predicates!

46.642.7

40455055606570

Supervised Self-Training Graph-Based

F-Measure

46.642.7

40455055606570

F-Measure

Full Parsing

30.226.6

F-Measure

Conclusions

Parsing using the theory of frame semantics– richer output than popular SRL systems

(Kingsbury and Palmer, 2002)

– domain general in comparison to deep semantic parsers

Significantly better performance on benchmark datasets than previous work– less independence assumptions

– only two statistical models

– semi-supervised extensions

Conclusions

Train parsers in other languages– Spanish, German, Portuguese, Japanese, Chinese, Swedish

Use presented techniques for deeper semantic analysis tasks–Especially semi-supervised learning

Use parser for NLP applications

–(right now being used to bootstrap more annotations)

Future Work

Fillmore, 1968)

Fillmore, 1982)

NomBankOntoNotes

MUCACE

More annotationsLarger lexicons

Last Word

Parser available at:

http://www.ark.cs.cmu.edu/SEMAFOR

(200 downloads in the past 6 months)

UDGMENT_DIRECT_ADDRESS

robust shallow semantic parsing of text › ~dipanjan › ttic_presentation.pdf · semantic parsing...

Documents

dependency parsing framenet, semantic...

incremental nonmonotonic parsing through semantic self...

the semantic deficit hypothesis: perceptual parsing...

pyramid scene parsing network in 3d: improving semantic

parsing, ast & semantic

frame - semantic parsing

learning for semantic parsing using statistical syntactic...

executable semantic parsing

deep hierarchical parsing for semantic segmentation ·...

type constraints neural semantic parsing withlexicalized...

semantic parsing with dual learning · gen(y tjy

semantic parsing 1 semantic parsing al akhwayn summer school...

statistical models for frame-semantic...

combining contexts in lexicon learning for semantic parsing

semantic parsing: past, present, and future - yoav...

3d semantic parsing of large-scale indoor...

a medical note automated parsing and semantic extraction

an arabic semantic parser and meaning analyzerbottom-up...

multi-lingual joint parsing of syntactic and semantic...

neural semantic parsing with anonymization for command...