weakly-supervised grammar-informed bayesian ccg parser ... · weighted category grammar + + np n n...
TRANSCRIPT
![Page 1: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/1.jpg)
Weakly-Supervised Grammar-Informed
Bayesian CCG Parser Learning
Dan Garrette Chris Dyer
Jason Baldridge Noah A. Smith
UT-Austin CMU UT-Austin CMU
![Page 2: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/2.jpg)
Annotating parse trees by hand is extremely difficult.
Motivation
![Page 3: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/3.jpg)
Motivation
Can we learn new parsers cheaply?
(cheaper = less supervision)
![Page 4: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/4.jpg)
Motivation
When supervision is scarce, we have to be smarter about data.
![Page 5: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/5.jpg)
Type-Level Supervision
![Page 6: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/6.jpg)
Type-Level Supervision
• Unannotated text
• Incomplete tag dictionary: word � {tags}
![Page 7: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/7.jpg)
Type-Level Supervision
Used for part-of-speech tagging for 20+ years
[Kupiec, 1992] [Merialdo, 1994]
![Page 8: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/8.jpg)
Type-Level Supervision
Good tagger performance even with low supervision
[Ravi & Knight, 2009] [Das & Petrov, 2011] [Garrette & Baldridge, 2013] [Garrette et al., 2013]
![Page 9: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/9.jpg)
Combinatory Categorial Grammar
(CCG)
![Page 10: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/10.jpg)
CCG
Every word token is associated with a category
Categories combine to form categories of larger constituents
[Steedman, 2000] [Steedman and Baldridge, 2011]
![Page 11: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/11.jpg)
the dog
n
np
n/np
CCG
![Page 12: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/12.jpg)
s
dogs
np
sleep
\s
CCG
np
![Page 13: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/13.jpg)
Type-Supervised CCG
the lazy dogs
np/n n
np
n/n
np
wander
(s\np)/np
n
n/n
np/n
s\np…
![Page 14: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/14.jpg)
nnp npn
wanderthe lazy dogs
/n n/ \s
CCG Parsing
![Page 15: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/15.jpg)
nnp npn
wanderthe lazy dogs
/n n/ \s
CCG Parsing
![Page 16: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/16.jpg)
nnp npn
wanderthe lazy dogs
/n n/ \s
CCG Parsing
![Page 17: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/17.jpg)
n
n
np npn
wanderthe lazy dogs
/n n/ \s
CCG Parsing
![Page 18: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/18.jpg)
n
nn
np npn
wanderthe lazy dogs
/n n/ \s
CCG Parsing
![Page 19: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/19.jpg)
n
nn
np npn
wanderthe lazy dogs
/n n/ \s
CCG Parsing
![Page 20: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/20.jpg)
n
nn
np npn
wanderthe lazy dogs
np
/n n/ \s
CCG Parsing
![Page 21: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/21.jpg)
n
nn
np npn
wanderthe lazy dogs
np
/n n/ \s
CCG Parsing
![Page 22: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/22.jpg)
n
nn
np npn
wanderthe lazy dogs
np
/n n/ \s
CCG Parsing
![Page 23: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/23.jpg)
n
nn
np npn
wanderthe lazy dogs
np
/n n/ \s
s
CCG Parsing
![Page 24: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/24.jpg)
Why CCG?
Machine Translation[Weese, Callison-Burch, and Lopez, 2012]
Semantic Parsing[Zettlemoyer and Collins, 2005]
![Page 25: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/25.jpg)
Type-supervised learning for CCG is highly ambiguous
Penn Treebank parts-of-speech
CCGBank Categories
48 tags 1,300+ categories
Type-Supervised CCG
![Page 26: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/26.jpg)
The grammar formalism itself can be used to guide learning
Our Strategy
![Page 27: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/27.jpg)
Our Strategy
Incorporate universal knowledge about grammar into learning
![Page 28: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/28.jpg)
Universal Knowledge
![Page 29: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/29.jpg)
the lazy dognp/n (np\(np/n))/n n
np\(np/n)
np
the lazy dognp/n n/n n
n
np
Prefer Simpler Categories
![Page 30: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/30.jpg)
the lazy dognp/n (np\(np/n))/n n
np\(np/n)
np
the lazy dognp/n n/n n
n
np
Prefer Simpler Categories
![Page 31: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/31.jpg)
buy := (((sb\np)/pp)/pp)/np
appears 342 times in CCGbankbuy := (sb\np)/np
appears once
e.g. “Opponents don't buy such arguments.”
“Tele-Communications agreed to buy half of Showtime Networks from Viacom for $ 225 million.”pp pp
Prefer Simpler Categories
![Page 32: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/32.jpg)
transitive verb: (he) hides (the money)
(sb\np)/np
Prefer Modifier Categories
((sb\np)/np)/((sb\np)/np)
adverb: (he) quickly (hides) (the money)
![Page 33: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/33.jpg)
a {s, np, n,…}
A B / B
A
B \ B
patom(a)
A
A B \ C
B / C
× pterm
pterm
pterm
pterm
pterm
× pfwd
× pfwd
× pfwd
× pfwd
× pmod
× pmod
× pmod
× pmod
Weighted Category Grammar
![Page 34: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/34.jpg)
a {s, np, n,…}
A B / B
A
B \ B
patom(a)
A
A B \ C
B / C
× pterm
pterm
pterm
pterm
pterm
× pfwd
× pfwd
× pfwd
× pfwd
× pmod
× pmod
× pmod
× pmod
Weighted Category Grammar
+
+
![Page 35: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/35.jpg)
nnpnp npn
wanderthe lazy dogs
/nn n/ \ss
nn
np
s
Prefer Likely Categories
![Page 36: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/36.jpg)
nnpnp npn
wanderthe lazy dogs
/nn n/ \ss
nn
np
s
Prefer Likely Categories
![Page 37: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/37.jpg)
Type-Supervised Learning
unlabeled corpus
tag dictionary
universal properties of the CCG formalism
same as POS tagging
![Page 38: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/38.jpg)
Posterior Inference
[Johnson, Griffiths, and Goldwater, 2007]
![Page 39: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/39.jpg)
Posterior Inference
the lazy dogsnp/n n
npn/nnp
wander
(s\np)/np
nn/n
np/ns\np…
Priors (simple is
good)
PCFG
Inside
![Page 40: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/40.jpg)
Posterior Inference
the lazy dogsnp/n n
npn/nnp
wander
(s\np)/np
nn/n
np/ns\np…
Priors (simple is
good)
PCFG
Inside
![Page 41: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/41.jpg)
Posterior Inference
the lazy dogsnp/n n
npn/nnp
wander
(s\np)/np
nn/n
np/ns\np…
Priors (simple is
good)
PCFG
Sample
![Page 42: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/42.jpg)
Posterior Inference
the lazy dogsnp/n n
npn/nnp
wander
(s\np)/np
nn/n
np/ns\np…
Priors (simple is
good)
PCFG
Sample
![Page 43: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/43.jpg)
Posterior Inference
the lazy dogsnp/n n
npn/nnp
wander
(s\np)/np
nn/n
np/ns\np…
Priors (simple is
good)
PCFG
![Page 44: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/44.jpg)
Posterior Inference
the lazy dogsnp/n n
npn/nnp
wander
(s\np)/np
nn/n
np/ns\np…
Priors (simple is
good)
PCFG
![Page 45: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/45.jpg)
Posterior Inference
the lazy dogsnp/n n
npn/nnp
wander
(s\np)/np
nn/n
np/ns\np…
Priors (simple is
good)
PCFG
![Page 46: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/46.jpg)
Results
![Page 47: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/47.jpg)
0
25
50
75
English Chinese Italian
UniformWith Prior
0
25
50
75
English Chinese Italian
UniformWith Prior
CCG Parsing Results
55.7
42.0
60.0
pars
ing
accu
racy
53.4
35.9
58.2
![Page 48: Weakly-Supervised Grammar-Informed Bayesian CCG Parser ... · Weighted Category Grammar + + np n n np the lazy dogs wander / n/ n s\ n np s Prefer Likely Categories. np n n np the](https://reader033.vdocuments.net/reader033/viewer/2022050206/5f59777c22ccc237913e6063/html5/thumbnails/48.jpg)
Conclusion
Using universal grammatical knowledge can make better use of weak supervision