robust shallow semantic parsing of text › ~dipanjan › ttic_presentation.pdf · semantic parsing...

Dipanjan DasCarnegie Mellon University

Toyota Technological Institute at Chicago

February 14, 2012

Robust Shallow Semantic Parsing of Text

Natural Language Understanding

2


I want to go to Chicago on Sunday

3

I want to go to on Sunday


P V ADP V ADP N ADP N

Shallow Syntax:Part-of-Speech Tagging

4

Chicago


Deeper Syntax:Dependency parsing



5

Chicago


Shallow Semantics: Frames and Roles



6

Chicago





Encodes an eventor

scenario

7

Chicago





8

Chicago





participant orrole

for the frame

9

Chicago




Experiencer

10

Chicago



Focus of this talk!(Das, Schneider, Chen and Smith, NAACL 2010;

Das and Smith, ACL 2011)

11


Experiencer

Chicago

1. Why semantic analysis?

3. Semi-supervised learning for robustness

2. Statistical models for structure prediction

12




ApplicationsMotivation Choice offormalism

13




Argument identificationFrame identification

14





(use of latent variables) (dual decomposition)

15




Novel graph-based learning algorithms16


Motivation ApplicationsChoice offormalism

17

Bengal ’s massive stock of food was reduced to nothing

18

Bengal ’s massive stock of food was reduced to nothingN X A N ADP N V V ADP N

19


Large body of research on syntax, includingDas and Petrov, ACL 2011

Cohen, Das and Smith, EMNLP 2011Martins, Das, Smith and Xing, EMNLP 2008

20


storeor

financial entity?

21


Store of what?Of what size?Whose store?

22


Store of what?Of what size?Whose store?

23


What was reduced?

24


What was reduced?

To what?

25


What was reduced?

To what?

26


27


AUSE_CHANGE_OF_POSITION_ON_A_SCALE

28

Origins: (Computational) Linguistics

29


Case Grammar(“The Case for Case”, Fillmore, 1968)

30

I gave some money to him



31

(cases are words/phrases required by a predicate)


Agent Object Beneficiary



32

(cases are words/phrases required by a predicate)


Agent Object Beneficiary

Semantic valency of a predicate

Correlation with syntax(e.g. subject and object)

Obligatory cases / optional cases



33

Slide idea taken from Brendan O’Connor

Case Grammar(“The Case for Case”,

Fillmore, 1968)

34

Frames(“A Framework

for Representing Knowledge”, Minsky, 1975)



Fillmore, 1968)

35

Frame Semantics(“Frame Semantics”,

Fillmore, 1982)





Fillmore, 1968)

36


Fillmore, 1982)

Relates the meaning of a word with world knowledge

(e.g. gave evokes a IVING frame;

it has several participating roles;the frame is evoked by other words,

such as bequeath, contribute, donate)





Fillmore, 1968)

37


Fillmore, 1982)





Fillmore, 1968)

Relates the meaning of a word with world knowledge

(e.g. gave evokes a IVING frame;

it has several participating roles;the frame is evoked by other words,

such as bequeath, contribute, donate)

38


Fillmore, 1968)


Fillmore, 1982)

FrameNetPropBankVerbNet

NomBankOntoNotes



Datasets


39


Fillmore, 1968)


Fillmore, 1982)


NomBankOntoNotes



Datasets

Data-Driven ShallowSemantic Parsing


40


Fillmore, 1968)


Fillmore, 1982)


NomBankOntoNotes



Scripts(“Scripts, Plans,

Goals and Understanding”, Schank and Abelson, 1977)

MUCACE

GENIA

Datasets


Information Extraction(template filling)


41


Fillmore, 1968)


Fillmore, 1982)


NomBankOntoNotes





MUCACE

GENIA

Datasets




42


Fillmore, 1968)


Fillmore, 1982)


NomBankOntoNotes





MUCACE

GENIA

Datasets




43


Fillmore, 1968)


Fillmore, 1982)


NomBankOntoNotes





MUCACE

GENIA

Datasets




structurally similar!

44

Why this Linguistic Formalism?

45

shallow

deep

46


shallow

deep

PropBank-style Semantic Role Labeling(Pradhan et al., 2008; Punyakanok et al., 2008)

47


shallow

deep



48


shallow

deep


A1 A4


49


shallow

deep


A1 A4

symbolic set of semantic roles(six total)

Verb-specific meaning for these labels

Conflates the meaning of different roles due to oversimplification

(Yi et al., 2007)


50


shallow

deepSemantic Parsing into Logical Forms

(Ge and Mooney, 2005; Zettlemoyer and Collins., 2005)


51


shallow

deep

What states border the state that borders the most states

Semantic Parsing into Logical Forms(Ge and Mooney, 2005; Zettlemoyer and Collins., 2005)


52


shallow

deep




53


shallow

deep



Trained on very restricted domains

Poor lexical coverage


54


shallow

deep



Frame-Semantic Parsing(Gildea and Jurafsky, 2002, Johansson and Nugues, 2007; this work)

55


shallow

deep



56


Does not model quantification or negation

unlike logical forms


shallow

deep



Deeper than PropBank-style semantic role labeling

57



shallow

deep



Models all types of part-of-speech categories

58



shallow

deep



Larger lexical coverage than logical form parsers

59



shallow

deep



Lexicon actively increasing in size

60




ApplicationsMotivation ApplicationsChoice offormalism

61

Question Answering


Whose stock of food was diminished ?

Possible Applications

62

Question Answering




63

Question Answering




64

Question Answering




65

Question Answering


Whose stock of food was diminished ?reserve


66

Question Answering


Whose stock of food was diminished ?reserve

Bilotti et al. (2007)


67

Information Extraction

In 1997, France's stock of unirradiated civil plutonium increased to 72 tons.

Saudi Arabia has 267 billion barrels in reserves of oil.

Does Egypt have stockpiles of biological weapons?

Bengal’s massive stock of food was reduced to nothing.


68







69






Possessor Desc Resource

Bengal Massive food

France - unirradiated civil plutonium

Saudi Arabia 267 billion barrels oil

Egypt - Biological weapons


70

Multilingual Applications



71



��


72



��


73

Multilingual KBs



��

Cross lingual IR

Machine translation


74


(Das, Schneider, Chen and Smith, NAACL 2010)

75

Structure of Lexicon and Data

76


PLACING

Agent

Cause

Goal

Theme

Area

Time

77


PLACING

Agent

Cause

Goal

Theme

Area

Time

frame

78


PLACING

Agent

Cause

Goal

Theme

Area

Time

frame

roles

79


PLACING

Agent

Cause

Goal

Theme

Area

Time

framecore roles

non-core roles

80


PLACING

Agent

Cause

Goal

Theme

Area

Time

framecore roles

non-core roles

excludes relationship

81


PLACING

Agent

Cause

Goal

Theme

Area

Time

framecore roles

non-core roles

archive.V, arrange.V, bag.V, bestow.V bin.V

predicates

excludes relationship

82


PLACING

Agent

Cause

Goal

Theme

Area

Time

DISPERSAL

Agent

Cause

Individuals

Distance

Time

TRANSITIVE_ACTION

Agent

Cause

Patient

Event

Place

Time

INSTALLING

Agent

Component

Fixed_location

Area

Time

STORING

Agent

Location

Theme

Area

Time

STORE

Possessor

Resource

Supply

Descriptor

83


PLACING

Agent

Cause

Goal

Theme

Area

Time

DISPERSAL

Agent

Cause

Individuals

Distance

Time

TRANSITIVE_ACTION

Agent

Cause

Patient

Event

Place

Time

INSTALLING

Agent

Component

Fixed_location

Area

Time

STORING

Agent

Location

Theme

Area

Time

STORE

Possessor

Resource

Supply

Descriptor

84


PLACING

Agent

Cause

Goal

Theme

Area

Time

DISPERSAL

Agent

Cause

Individuals

Distance

Time

TRANSITIVE_ACTION

Agent

Cause

Patient

Event

Place

Time

INSTALLING

Agent

Component

Fixed_location

Area

Time

STORING

Agent

Location

Theme

Area

Time

STORE

Possessor

Resource

Supply

Descriptor

inheritance

used by

85


Benchmark Dataset(SemEval 2007)

665 frames720 role labels

8.4K unique predicate types

Training set:2.2K sentences

11.2K predicate tokens

Test set:120 sentences

1. 1K predicate tokens86


For comparison with past state of the art

(very small dataset)









New Data(FrameNet 1.5, 2010)

















90




91

Frame Identification


92



ambiguous

93



Find the best among all the frames

94



95



96


Direct modeling using logistic regression

97

Two problems:1. Unable to model unknown

predicates at test time



98





2. Number of features:

99





2. Number of features: ≈ 50 million

100


Logistic regression with a latent variable

101



102



103

Predicates evoking a frame in supervised data, e.g.cargo.N, inventory.N, reserve.N, stockpile.N, store.N, supply.N

evoke STORE



104

Does not look at the predicate’s surface form


105


106

TORE

stock.N

stockpile.N



107

TORE

stock.N

stockpile.N


LexSem = {synonym}


108

TORE

stock.N

stockpile.N


LexSem = {synonym}

If STORE

stockpile.N

synonym LexSem


109

TORE

stock.N

stockpile.N


LexSem = {synonym}

If STORE

stockpile.N

synonym LexSem

comes from WordNet!


110

Number of features:


111

Number of features:

≈ 500 K

(1% of the features we had before)


112

Number of features:

≈ 500 K

(1% of the features we had before)

Aside:

Probabilistic modeling of language meaning using syntax, lexical semantics and latent structure:

Paraphrase identification (Das and Smith, ACL 2009)


Training:

Maximum conditional log-likelihood(batch training with L-BFGS)

113


Fast inference:

If predicate is unseen,

otherwise,

114


Results

Benchmark

60.464.0

58

63

68

73

78

83

88

93

UTD LTH ThisWork

F-Measure

New Data

115


Results

Benchmark

60.464.0

58

63

68

73

78

83

88

93

UTD LTH ThisWork

F-Measure

New Data

Bejan and Hathaway, (2007)

116


Results

Benchmark

60.464.0

58

63

68

73

78

83

88

93

UTD LTH ThisWork

F-Measure

New Data

Johansson and Nugues, (2007)

117


Results

Benchmark

60.464.0

68.3

58

63

68

73

78

83

88

93

UTD LTH ThisWork

F-Measure

New Data

118


Results

Benchmark

60.464.0

68.3

58

63

68

73

78

83

88

93

UTD LTH ThisWork

F-Measure

auto predicates

New Data

119


Results

Benchmark

60.464.0

68.3

74.2

58

63

68

73

78

83

88

93

UTD LTH ThisWork

ThisWork

F-Measure

auto predicates givenpredicates

New Data

120


Results

Benchmark New Data

60.464.0

68.3

74.2

58

63

68

73

78

83

88

93

UTD LTH ThisWork

ThisWork

F-Measure


90.5

58

63

68

73

78

83

88

93

ThisWork

F-Measure

givenpredicates

121


Results

Benchmark New Data

60.464.0

68.3

74.2

58

63

68

73

78

83

88

93

UTD LTH ThisWork

ThisWork

F-Measure


90.5

80.0

58

63

68

73

78

83

88

93

ThisWork

ThisWork

F-Measure

givenpredicates

no hidden variable

122




123

Argument Identification


TORE

124



TORE

125


stock

TORE

Bengal ’s

Bengal

massive stock

of food

food

massive

Bengal ’s massive stock of food

massive stock of food

126


stock

TORE

Bengal ’s

Bengal

massive stock

of food

food

massive



Ideal mapping!

127


stock

TORE

Bengal ’s

Bengal

massive stock

of food

food

massive



128


stock

TORE

Bengal ’s

Bengal

massive stock

of food

food

massive



Violates overlap constraints

129


Other types of structural constraints

PLACING

Agent

Cause

Goal

Theme

Area

Time

Mutual exclusion constraint


130



PLACING

Agent

Cause

Goal

Theme

Area

Time



If an agent places something, there cannot be a cause role in the sentence

131



PLACING

Agent

Cause

Goal

Theme

Area

Time



132

AgentThe waiter placed food on the table.

In Kabul, hauling water put food on the table.Cause



SIMILARITY

Dimension

Differentiating_fact

Entity_1

Entity_2

Degree

Requires constraint

difference.N, resemble.V,

unliike.A, vary.V

133



A mulberry resembles a loganberry.

second entity

first entity

SIMILARITY

Dimension


Entity_1

Entity_2

Degree

Requires constraint


unliike.A, vary.V

134



A mulberry resembles.

SIMILARITY

Dimension


Entity_1

Entity_2

Degree

Requires constraint


unliike.A, vary.V

135

!


stock

TORE

Bengal ’s

Bengal

massive stock

of food

food

massive



A constrained optimization

problem

136


stock

TORE

Bengal ’s

Bengal

massive stock

of food

food

massive



137


stock

TORE

Bengal ’s

Bengal

massive stock

of food

food

massive



138


A constrained optimization problem

139



a binary variable for each role, span tuple

140



a binary vector for all role, span tuples

141



142



143



Uniqueness144



Prevents overlap 145



more structural constraints

146



An integer linear program (ILP)


147





148

Punyakanok, Roth and Yih(2008)



Often, very slow solutions



149


A constrained optimization problemAn integer linear program (ILP)


Fast ILP solvers proprietary

150


An alternate approach

Dual Decomposition with Alternating Direction Method of Multipliers

developed with colleagues at CMU(Das, Martins and Smith, forthcoming)

151



basic part:

entire space: all tuples

152



basic part:


Break down the problem into many small components

e.g. -- find the best span for a role

-- for a sentence position, find the best tuple

-- for a pair of mutually exclusive roles,find the best tuple

153



basic part:


Break down the problem into many small components

e.g. -- find the best span for a role

-- for a sentence position, find the best tuple

-- for a pair of mutually exclusive roles,find the best tuple

impose agreement between components

154



For each component, a binary vector:

155




Total score assigned to :

156




Total score assigned to :

uses

157



158



witness vector for consensus

159



Primal:

160



Primal':

Integer constraints relaxed

161



Primal':

An augmented Lagrangian function

162


An alternate approachAn augmented Lagrangian function

Saddle point can be found using several decoupled

worker problems

163



Three types of iterative updates:

1. Lagrange multiplier updates ( )2. Consensus variable updates ( )

3. updates

164





3. updatesAt decoupled workers

165






e.g. for each role, we have a worker that imposes a XOR/uniqueness constraint

166







Projection onto a simplexa simple sort operation

167







Projection onto a simplexa simple sort operation

Challenge:define fast, simple workers

168



Advantages:

1) Significant speedup2) No proprietary solver necessary

169



Advantages:

43.12

4.780

10

20

30

40

50

CPLEX (ILP) Dual Decomposition

Time to decode the test set in seconds


170



Advantages:

43.12

4.780

10

20

30

40

50

CPLEX (ILP) Dual Decomposition

Time to decode the test set in seconds


171

Certificate of Optimality in

>99% of examples


stock

TORE

Bengal ’s

Bengal

massive stock

of food

food

massive



172


stock

TORE

Bengal ’s

Learning?

173


stock

TORE

Bengal ’s

Learning?

Maximum conditional log-likelihoodof local role span pairs

(batch training using L-BFGS)

174


Results

New Data

Benefit of joint inference175

8283.8

76.4 76.2

79.1 79.8

70

75

80

85

Local DualDecomposition

Precision

Recall

F-Measure


Results

New Data

Benefit of joint inference

8283.8

76.4 76.2

79.1 79.8

70

75

80

85


Precision

Recall

F-Measure

501 linguistic violations

176


Results

New Data

Benefit of joint inference

501 linguistic violations No violations

177

8283.8

76.4 76.2

79.1 79.8

70

75

80

85


Precision

Recall

F-Measure

Full Parsing

Final Results

Benchmark New Data

37.9

45.6

35

40

45

50

55

60

65

70

UTD LTH ThisWork

F-Measure

auto predicates

178

Full Parsing

Final Results

Benchmark New Data

37.9

45.6

50.2

35

40

45

50

55

60

65

70

UTD LTH ThisWork

F-Measure

auto predicates

179

Full Parsing

Final Results

Benchmark New Data

37.9

45.6

50.253.6

35

40

45

50

55

60

65

70

UTD LTH ThisWork

ThisWork

F-Measure


180

Full Parsing

Final Results

Benchmark New Data

37.9

45.6

50.253.6

35

40

45

50

55

60

65

70

UTD LTH ThisWork

ThisWork

F-Measure


68.5

35

40

45

50

55

60

65

70

This Work

givenpredicates

F-Measure

181




Novel graph-based learning algorithms

182



Novel graph-based learning algorithms

(Das and Smith, ACL 2011)


183

90.5

405060708090

All Predicates

F-Measure

Results on Unknown Predicates

46.6

40

50

60

70

80

90

Unknown Predicates

F-Measure


184

90.5

405060708090

All Predicates

F-Measure


46.6

40

50

60

70

80

90

Unknown Predicates

F-Measure


68.5

25303540455055606570

All Predicates

F-Measure

30.2

25303540455055606570

Unknown Predicates

F-Measure

Full Parsing

185

Handling Unknown Predicates

Knowledge of only 9,263 predicates in supervised data

186



187

However, English has lot more potential predicates(~65,000 in newswire English)



However, English has lot more potential predicates(~65,000 in newswire English)

Lexicon expansion using graph-based semi-supervised learning

188

Build a graph over potential predicates as vertices• compute similarity matrix using co-occurrence statistics

Label distribution at each vertexdistribution over frames that the predicate can evoke

How can label propagation help?

189

Build a graph over potential predicates as vertices• compute similarity matrix using co-occurrence statistics

Label distribution at each vertexdistribution over frames that the predicate can evoke

How can label propagation help?

Idea very similar to Das and Petrov (ACL 2011): unsupervised lexicon expansion for POS tagging

190

Example Graph

Seed predicates

191

Example Graph

Seed predicatesUnseen predicates

192

Example Graph

Graph Propagation193

Example Graph

Graph Propagation194

Example Graph

Graph Propagation

Continues till convergence...

195

Brief Overview:Graph-Based Learning

with Labeled and Unlabeled Data

196

0.9

0.01

0.8

0.9

0.1

= symmetric weight matrix

0.05

198

labeled datapointsunlabeled datapoints

0.9

0.01

0.8

0.9

0.1


0.05

199

0.9

0.01

0.8

0.9

0.1


0.05

200

0.9

0.01

0.8

0.9

0.1


0.05

supervised label distributions

201

0.9

0.01

0.8

0.9

0.1


0.05

supervised label distributions

distributions to be found

202

Label Propagation

Minimize:

Das and Smith, forthcoming 203

Label Propagation

Minimize:

204

Brings the distribution of observed and induced distributions over labeled vertices closer

Label Propagation

Minimize:

205

brings the distributions of similarvertices closer

Label Propagation

Minimize:

206

induces sparse distributionsin each vertex

Constrained Inference

If predicate is seen,

otherwise,

else if predicate is in graph,

207

Constrained Inference

If predicate is seen,

otherwise,

else if predicate is in graph,

Six times faster inference on unknown predicates!

208


46.642.7

65.3

40455055606570

Supervised Self-Training Graph-Based

F-Measure


209


46.642.7

65.3

40455055606570


F-Measure


Full Parsing

30.226.6

46.7

25

30

35

40

45

50


F-Measure

210

Conclusions

211

Parsing using the theory of frame semantics– richer output than popular SRL systems

(Kingsbury and Palmer, 2002)

– domain general in comparison to deep semantic parsers

Significantly better performance on benchmark datasets than previous work– less independence assumptions

– only two statistical models

– semi-supervised extensions

Conclusions

212

Train parsers in other languages– Spanish, German, Portuguese, Japanese, Chinese, Swedish

Use presented techniques for deeper semantic analysis tasks–Especially semi-supervised learning

Use parser for NLP applications

–(right now being used to bootstrap more annotations)

Future Work

213


Fillmore, 1968)


Fillmore, 1982)


NomBankOntoNotes





MUCACE

GENIA




More annotationsLarger lexicons

214

Last Word

Parser available at:

http://www.ark.cs.cmu.edu/SEMAFOR

(200 downloads in the past 6 months)

215

UDGMENT_DIRECT_ADDRESS

217

robust shallow semantic parsing of text › ~dipanjan › ttic_presentation.pdf · semantic parsing...

Documents