msr visit talk 2016 - i.konstas · joint generator temperature time min mean max 06:00-21:00 9 15...
TRANSCRIPT
NEUGEN !
Text Generation from Meaning Representations
Yannis Konstas!!
Joint work with Mark Yatskar, Luke Zettlemoyer and Yejin Choi (UW)
~ Yonatan Bisk and Daniel Marcu (ISI)
Motivation
Motivation
Machine-generated Representation
Motivation
Machine-generated Representation
Motivation
Machine-generated Representation
Motivation
Machine-generated Representation
source block: hk
target block: ms
pos RP: W scale: small
Motivation
Machine-generated Representation
source block: hk
target block: ms
pos RP: W scale: small
Place the heineken block west of the mercedes block.
Motivation
Machine-generated Representation
source block: hk
target block: ups
pos RP: W scale: big
Motivation
Machine-generated Representation
source block: hk
target block: ups
pos RP: W scale: big
Place the heineken block in the first open space to the left of ups block.
Motivation
Motivation
Machine-generated Representation
Motivation
Machine-generated Representation
Sentence Compression
Motivation
Machine-generated Representation
Sentence Compression
Sentence Fusion
Motivation
Machine-generated Representation
Sentence Compression
Sentence FusionParaphrasing
NLG Pipeline
Content Selection
Document Planning
Surface Realisation
Sentence Planning
Text
Communicative Goal
Content Planning
A A A A
mInput
Reordering/Linearization
Splitting/Aggregation
Lexicalization
Framework
NLG Pipeline
Content Selection
Document Planning
Surface Realisation
Sentence Planning
Text
Communicative Goal
Content Planning
A A A A
mInput
Reordering/Linearization
Splitting/Aggregation
Lexicalization
Framework
NLG Pipeline
Content Selection
Document Planning
Surface Realisation
Sentence Planning
Text
Communicative Goal
Content Planning
A A A A
mInput
Reordering/Linearization
Splitting/Aggregation
Lexicalization- Neural Encoder - RNN Decoder - Graph Alignment
Framework
Joint Generator Temperature
Time Min Mean Max
06:00-21:00 9 15 21
Wind Speed
Time Min Mean Max
06:00-21:00 15 20 30
Cloud Sky Cover
Time Percent (%)
06:00-09:00 25-5009:00-12:00 50-75
Wind Direction
Time Mode
06:00-21:00 S
S
...
S
...
S
...
S
...S
...
?? ?
Cloudy, with temperaturesbetween 10 and 20 degrees.South wind around 20 mph.
Training
S ! R(start)R(ri.t)!FS(rj , start)R(rj .t)R(ri.t)!FS(rj , start)FS(r, r.fi)!F(r, r.fj)FS(r, r.fj)FS(r, r.fi)!F(r, r.fj)F(r, r.f)!W(r, r.f)F(r, r.f)F(r, r.f)!W(r, r.f)W(r, r.f)!↵W(r, r.f)!g(f.v)
FS0,1(temp1,start)
FS0,2(temp1,start)
F0,1(temp1,min)
F0,1(temp1,max)
F0,2(temp1,min)
F0,2(temp1,max)
FS1,2(temp1,start)
(2) HypergraphRepresentation
FS0,5(skyCover1.t,start)
0
BB@
mostly cloudy ? the morning
mostly cloudy ? after 11am
mostly cloudy ? then becoming
· · ·
1
CCA
F0,2(skyCover1.t,%)
0
BB@
mostly cloudy
mostly clouds
cloudy ,
· · ·
1
CCAW4,5(skyCover1.t,time)
0
BB@
morning
11am
after
· · ·
1
CCAW0,1(skyCover1.t,%)
0
BB@
mostly
cloudy
sunny
· · ·
1
CCA
W1,2(skyCover1.t,%)
0
BB@
mostly
cloudy
sunny
· · ·
1
CCA
(3) k-best decodingvia integration
Testing
(1) PCFG Grammar
Konstas and Lapata, 2012, 2013
Joint Generator Temperature
Time Min Mean Max
06:00-21:00 9 15 21
Wind Speed
Time Min Mean Max
06:00-21:00 15 20 30
Cloud Sky Cover
Time Percent (%)
06:00-09:00 25-5009:00-12:00 50-75
Wind Direction
Time Mode
06:00-21:00 S
S
...
S
...
S
...
S
...S
...
?? ?
Cloudy, with temperaturesbetween 10 and 20 degrees.South wind around 20 mph.
Training
S ! R(start)R(ri.t)!FS(rj , start)R(rj .t)R(ri.t)!FS(rj , start)FS(r, r.fi)!F(r, r.fj)FS(r, r.fj)FS(r, r.fi)!F(r, r.fj)F(r, r.f)!W(r, r.f)F(r, r.f)F(r, r.f)!W(r, r.f)W(r, r.f)!↵W(r, r.f)!g(f.v)
FS0,1(temp1,start)
FS0,2(temp1,start)
F0,1(temp1,min)
F0,1(temp1,max)
F0,2(temp1,min)
F0,2(temp1,max)
FS1,2(temp1,start)
(2) HypergraphRepresentation
FS0,5(skyCover1.t,start)
0
BB@
mostly cloudy ? the morning
mostly cloudy ? after 11am
mostly cloudy ? then becoming
· · ·
1
CCA
F0,2(skyCover1.t,%)
0
BB@
mostly cloudy
mostly clouds
cloudy ,
· · ·
1
CCAW4,5(skyCover1.t,time)
0
BB@
morning
11am
after
· · ·
1
CCAW0,1(skyCover1.t,%)
0
BB@
mostly
cloudy
sunny
· · ·
1
CCA
W1,2(skyCover1.t,%)
0
BB@
mostly
cloudy
sunny
· · ·
1
CCA
(3) k-best decodingvia integration
Testing
(1) PCFG Grammar
Konstas and Lapata, 2012, 2013
Joint Generator Temperature
Time Min Mean Max
06:00-21:00 9 15 21
Wind Speed
Time Min Mean Max
06:00-21:00 15 20 30
Cloud Sky Cover
Time Percent (%)
06:00-09:00 25-5009:00-12:00 50-75
Wind Direction
Time Mode
06:00-21:00 S
S
...
S
...
S
...
S
...S
...
?? ?
Cloudy, with temperaturesbetween 10 and 20 degrees.South wind around 20 mph.
Training
S ! R(start)R(ri.t)!FS(rj , start)R(rj .t)R(ri.t)!FS(rj , start)FS(r, r.fi)!F(r, r.fj)FS(r, r.fj)FS(r, r.fi)!F(r, r.fj)F(r, r.f)!W(r, r.f)F(r, r.f)F(r, r.f)!W(r, r.f)W(r, r.f)!↵W(r, r.f)!g(f.v)
FS0,1(temp1,start)
FS0,2(temp1,start)
F0,1(temp1,min)
F0,1(temp1,max)
F0,2(temp1,min)
F0,2(temp1,max)
FS1,2(temp1,start)
(2) HypergraphRepresentation
FS0,5(skyCover1.t,start)
0
BB@
mostly cloudy ? the morning
mostly cloudy ? after 11am
mostly cloudy ? then becoming
· · ·
1
CCA
F0,2(skyCover1.t,%)
0
BB@
mostly cloudy
mostly clouds
cloudy ,
· · ·
1
CCAW4,5(skyCover1.t,time)
0
BB@
morning
11am
after
· · ·
1
CCAW0,1(skyCover1.t,%)
0
BB@
mostly
cloudy
sunny
· · ·
1
CCA
W1,2(skyCover1.t,%)
0
BB@
mostly
cloudy
sunny
· · ·
1
CCA
(3) k-best decodingvia integration
Testing
(1) PCFG Grammar
Konstas and Lapata, 2012, 2013
More Complex Structure
More Complex Structure
know-01
I planet
lazy
ARG0 ARG1
inhabit-01
man
ARG1-of
ARG0
mod
Input (Graph or Tree)
More Complex Structure
know-01
I planet
lazy
ARG0 ARG1
inhabit-01
man
ARG1-of
ARG0
mod
Input (Graph or Tree)
AMR DAG
More Complex Structure
know-01
I planet
lazy
ARG0 ARG1
inhabit-01
man
ARG1-of
ARG0
mod
Input (Graph or Tree)
AMR DAG
λ-calculus Expression Tree
More Complex Structure
know-01
I planet
lazy
ARG0 ARG1
inhabit-01
man
ARG1-of
ARG0
mod
Input (Graph or Tree)
AMR DAG
λ-calculus Expression Tree
ECIs Tree
More Complex Structure
know-01
I planet
lazy
ARG0 ARG1
inhabit-01
man
ARG1-of
ARG0
mod
Input (Graph or Tree)
I know a planet that is inhabited by a lazy man.
Output (Text)
AMR DAG
λ-calculus Expression Tree
ECIs Tree
More Complex Structure
I knew a planet that was inhabited by a lazy man. (lpp_1943.249)
I have known a planet that was inhabited by a lazy man.
know-01
I planet
lazy
ARG0 ARG1
inhabit-01
man
ARG1-of
ARG0
mod
Input (Graph or Tree)
I know a planet that is inhabited by a lazy man.
Output (Text)
AMR DAG
λ-calculus Expression Tree
ECIs Tree
More Complex Structure
I knew a planet that was inhabited by a lazy man. (lpp_1943.249)
I have known a planet that was inhabited by a lazy man.
I know about a planet. It is inhabited by a lazy man.
know-01
I planet
lazy
ARG0 ARG1
inhabit-01
man
ARG1-of
ARG0
mod
Input (Graph or Tree)
I know a planet that is inhabited by a lazy man.
Output (Text)
AMR DAG
λ-calculus Expression Tree
ECIs Tree
NEUGEN Framework
ARG1-of
NEUGEN Framework
ARG1-of
Encoder DecoderG w
NEUGEN Framework
Encoder- Bag of Words: Concepts and roles- Attention mechanism
ARG1-of
Encoder DecoderG w
NEUGEN Framework
Encoder- Bag of Words: Concepts and roles- Attention mechanism
Decoder- Left-to-right word LSTM - Greedy beam search
ARG1-of
Encoder DecoderG w
NEUGEN Framework
Encoder- Bag of Words: Concepts and roles- Attention mechanism
Decoder- Left-to-right word LSTM - Greedy beam search
ARG1-of
Encoder DecoderG wReranker
NEUGEN Framework
Encoder- Bag of Words: Concepts and roles- Attention mechanism
Decoder- Left-to-right word LSTM - Greedy beam search
Reranker
ARG1-of
Encoder DecoderG wReranker
NEUGEN Framework
Encoder- Bag of Words: Concepts and roles- Attention mechanism
Decoder- Left-to-right word LSTM - Greedy beam search
Reranker- CCG parser -> semantics
ARG1-of
Encoder DecoderG wReranker
NEUGEN Framework
Encoder- Bag of Words: Concepts and roles- Attention mechanism
Decoder- Left-to-right word LSTM - Greedy beam search
Reranker- CCG parser -> semantics- Partial propositions graph Eincr
ARG1-of
Encoder DecoderG wReranker
NEUGEN Framework
Encoder- Bag of Words: Concepts and roles- Attention mechanism
Decoder- Left-to-right word LSTM - Greedy beam search
Reranker- CCG parser -> semantics- Partial propositions graph Eincr
ARG1-of
Aligner
Encoder DecoderG w
Eincr
G
Aligner- Feed-forward NN with attention
Reranker
NEUGEN Framework
Encoder- Bag of Words: Concepts and roles- Attention mechanism
Decoder- Left-to-right word LSTM - Greedy beam search
Reranker- CCG parser -> semantics- Partial propositions graph Eincr
ARG1-of
Aligner
Encoder DecoderG w
Eincr
G salign
Aligner- Feed-forward NN with attention- salign = fit(G, Eincr)
Reranker
Alignment
Alignment
w: A man inhabited a planet
Alignment
w: A man inhabited a planet
man
inhabit
agent
planet
patient
Eincr
Alignment
know-01
I planet
lazy
ARG0 ARG1
inhabit-01
man
ARG1-of
ARG0
mod
G
w: A man inhabited a planet
man
inhabit
agent
planet
patient
Eincr
Alignment
know-01
I planet
lazy
ARG0 ARG1
inhabit-01
man
ARG1-of
ARG0
mod
G
w: A man inhabited a planet
man
inhabit
agent
planet
patient
Eincr
fit(G, Eincr)
Alignment
know-01
I planet
lazy
ARG0 ARG1
inhabit-01
man
ARG1-of
ARG0
mod
G
w: A man inhabited a planet
man
inhabit
agent
planet
patient
Eincr
fit(G, Eincr)
<inhabit, man, agent>!<inhabit, planet, patient>
<know-01, I, A0>!<know-01, planet, A1>!<planet, inhabit-01, A1-of>!<inhabit-01, man, A0>!<man, lazy, mod>
Alignment <know, I, A0>!
!<know, planet, A1>!
!<planet, inhabit, A1-of>!
!<inhabit, man, A0>!
!<man, lazy, mod>
G
[0..1]
fitness score
<inhabit, man, agent>!!!!
<inhabit, planet, patient>
Eincr
E1:
. attention weights
+
max
Alignment
<inhabit, planet, patient>!!!!
<inhabit, man, agent>
<know, I, A0>!!
<know, planet, A1>!!
<planet, inhabit, A1-of>!!
<inhabit, man, A0>!!
<man, lazy, mod>
Eincr
G
E2:
fit(G,Eincr) =1
n
nX
i=0
fit(G,Ei)
[0..1]
fitness score
attention weights
+
max
.
Decoding
encoder(G)
know
I planet
lazy
ARG0 ARG1
inhabit
man
ARG1-of
ARG0
mod
G
Decoding
encoder(G)
;know
I planet
lazy
ARG0 ARG1
inhabit
man
ARG1-of
ARG0
mod
G
Decoding
encoder(G)
;
I The A …
know
I planet
lazy
ARG0 ARG1
inhabit
man
ARG1-of
ARG0
mod
G
Decoding
encoder(G)
;
I The A …
know
I planet
lazy
ARG0 ARG1
inhabit
man
ARG1-of
ARG0
mod
…
w11: I
Thew12:
Manw13:
Treew14:G
Decoding
encoder(G)
;
I The A …
re-score:
know
I planet
lazy
ARG0 ARG1
inhabit
man
ARG1-of
ARG0
mod
…
w11: I
Thew12:
Manw13:
Treew14:G
sLSTM
Decoding
encoder(G)
;
I The A …
know knew planet …
re-score:
know
I planet
lazy
ARG0 ARG1
inhabit
man
ARG1-of
ARG0
mod
…
w11: I
Thew12:
Manw13:
Treew14:G
sLSTM
Decoding
encoder(G)
;
I The A …
know knew planet …
re-score:
know
I planet
lazy
ARG0 ARG1
inhabit
man
ARG1-of
ARG0
mod
…
w11: I
Thew12:
Manw13:
Treew14:
…
I know
agent
w21:
I knew
agent
w22:
The planetw23:
Man planetw24:G
sLSTM + salign
Decoding
encoder(G)
;
I The A …
know knew planet …
re-score:
know
I planet
lazy
ARG0 ARG1
inhabit
man
ARG1-of
ARG0
mod
…
w11: I
Thew12:
Manw13:
Treew14:
…
I know
agent
w21:
I knew
agent
w22:
The planetw23:
Man planetw24:G
sLSTM + salign
Decoding
encoder(G)
;
I The A …
know knew planet …
a planet man …
re-score:
know
I planet
lazy
ARG0 ARG1
inhabit
man
ARG1-of
ARG0
mod
…
…
w11: I
Thew12:
Manw13:
Treew14:
…
I know
agent
w21:
I knew
agent
w22:
The planetw23:
Man planetw24:G
sLSTM + salign
Decoding
encoder(G)
;
I The A …
know knew planet …
a planet man …
re-score:
know
I planet
lazy
ARG0 ARG1
inhabit
man
ARG1-of
ARG0
mod
…
…
w11: I
Thew12:
Manw13:
Treew14:
…
I know
agent
w21:
I knew
agent
w22:
The planetw23:
Man planetw24:
w41: I know
agent
a planet
patient
w42: I knew
agent
planets that
patient
w43: The planet I knewagentpatient
w44: Man know
agent
a planet
patient
that
…G
sLSTM + salign
Decoding
encoder(G)
;
I The A …
know knew planet …
a planet man …
re-score:
know
I planet
lazy
ARG0 ARG1
inhabit
man
ARG1-of
ARG0
mod
…
…
w11: I
Thew12:
Manw13:
Treew14:
…
I know
agent
w21:
I knew
agent
w22:
The planetw23:
Man planetw24:
w41: I know
agent
a planet
patient
w42: I knew
agent
planets that
patient
w43: The planet I knewagentpatient
w44: Man know
agent
a planet
patient
that
…G
sLSTM + salign
?
Decoding
encoder(G)
;
I The A …
know knew planet …
a planet man …
re-score:
know
I planet
lazy
ARG0 ARG1
inhabit
man
ARG1-of
ARG0
mod
…
…
w11: I
Thew12:
Manw13:
Treew14:
…
I know
agent
w21:
I knew
agent
w22:
The planetw23:
Man planetw24:
w41: I know
agent
a planet
patient
w42: I knew
agent
planets that
patient
w43: The planet I knewagentpatient
w44: Man know
agent
a planet
patient
that
…G
sLSTM + salign
?
Decoding
encoder(G)
;
I The A …
know knew planet …
a planet man …
re-score:
know
I planet
lazy
ARG0 ARG1
inhabit
man
ARG1-of
ARG0
mod
…
…
w11: I
Thew12:
Manw13:
Treew14:
…
I know
agent
w21:
I knew
agent
w22:
The planetw23:
Man planetw24:
w41: I know
agent
a planet
patient
w42: I knew
agent
planets that
patient
w43: The planet I knewagentpatient
w44: Man know
agent
a planet
patient
that
…
inhabit inhabited was …
G
sLSTM + salign
Preliminary Results
Preliminary Results Experimental Setup
Preliminary Results Experimental Setup- Baseline RNN w/o alignment- NeuGen
Preliminary Results Experimental Setup- Baseline RNN w/o alignment- NeuGen
Dataset
Preliminary Results Experimental Setup- Baseline RNN w/o alignment- NeuGen
Dataset- AMR LDC2015E86- Train: 15k, Dev: 1.2k, Test: 1.2k
Preliminary Results Experimental Setup- Baseline RNN w/o alignment- NeuGen
Dataset- AMR LDC2015E86- Train: 15k, Dev: 1.2k, Test: 1.2k
0
5.5
11
16.5
22
BLEU-4 METEOR
BaselineNeuGen
Example Output
Example Output person
Gallardo found-01
domain ARG0-of
Tijuana Drug Cartel
ARG1
Gflee-01
ARG0-of
Example Output Gold: Gallardo is the fugitive founder of Tijuana Drug Cartel.!!Baseline: Gallardo is the founder of Tijuana Drug Cartel.!!NeuGen: Tijunana Drug Cartel is the founder of Gallardo.
person
Gallardo found-01
domain ARG0-of
Tijuana Drug Cartel
ARG1
Gflee-01
ARG0-of
Example Output Gold: Gallardo is the fugitive founder of Tijuana Drug Cartel.!!Baseline: Gallardo is the founder of Tijuana Drug Cartel.!!NeuGen: Tijunana Drug Cartel is the founder of Gallardo.
person
Gallardo found-01
domain ARG0-of
Tijuana Drug Cartel
ARG1
Gflee-01
ARG0-of
Gallardo !Tijuana Drug Cartel !State !Country !…
Example Output Gold: Gallardo is the fugitive founder of Tijuana Drug Cartel.!!Baseline: Gallardo is the founder of Tijuana Drug Cartel.!!NeuGen: Tijunana Drug Cartel is the founder of Gallardo.
person
Gallardo found-01
domain ARG0-of
Tijuana Drug Cartel
ARG1
Gflee-01
ARG0-of
Gallardo !Tijuana Drug Cartel !State !Country !…
Gallardo is the founder of !!Gallardo ’s Tijuana Drug Cartel is the !…
patient
agent
-19.82
-17.79
Tijuana Drug Cartel is the founder of !!!Tijuana Drug Cartel was the founder of !…
patient
patient
-18.67
-20.24
Example Output Gold: Gallardo is the fugitive founder of Tijuana Drug Cartel.!!Baseline: Gallardo is the founder of Tijuana Drug Cartel.!!NeuGen: Tijunana Drug Cartel is the founder of Gallardo.
person
Gallardo found-01
domain ARG0-of
Tijuana Drug Cartel
ARG1
Gflee-01
ARG0-of
Gallardo !Tijuana Drug Cartel !State !Country !…
Gallardo is the founder of !!Gallardo ’s Tijuana Drug Cartel is the !…
patient
agent
-19.82
-17.79
Tijuana Drug Cartel is the founder of !!!Tijuana Drug Cartel was the founder of !…
patient
patient
-18.67
-20.24
Gallardo is the founder of Tijuana Drug Cartel.
agentpatient -19.59
Tijuana Drug Cartel is the founder of Gallardo.
agentpatient -17.84
Next Steps
Next Steps
- “Machine-generated” representations - λ-calculus (ATIS) - Blocks World complex descriptions (from ISI)
Next Steps
- “Machine-generated” representations - λ-calculus (ATIS) - Blocks World complex descriptions (from ISI)
Next Steps
- “Machine-generated” representations - λ-calculus (ATIS) - Blocks World complex descriptions (from ISI)
- Downstream Text-to-Text Applications
THANK You