Download - Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006
![Page 1: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/1.jpg)
Coarse-to-Fine Efficient Viterbi Parsing
Nathan BodenstabOGI RPE Presentation
May 8, 2006
![Page 2: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/2.jpg)
2
Outline
• What is Natural Language Parsing?
• Data Driven Parsing
• Hypergraphs and Parsing Algorithms
• High Accuracy Parsing
• Coarse-to-Fine
• Empirical Results
![Page 3: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/3.jpg)
3
What is Natural Language Parsing?
• Provides a sentence with syntactic information by hierarchically clustering and labeling its constituents.
• A constituent is a group of one or more words that function together as a unit.
![Page 4: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/4.jpg)
4
What is Natural Language Parsing?
• Provides a sentence with syntactic information by hierarchically clustering and labeling its constituents.
• A constituent is a group of one or more words that function together as a unit.
![Page 5: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/5.jpg)
5
Why Parse Sentences?
• Syntactic structure is useful in– Speech Recognition– Machine Translation– Language Understanding
• Word Sense Disambiguation (ex. “bottle”)• Question-Answering• Document Summarization
![Page 6: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/6.jpg)
6
Outline
• What is Natural Language Parsing?
• Data Driven Parsing
• Hypergraphs and Parsing Algorithms
• High Accuracy Parsing
• Coarse-to-Fine
• Empirical Results
![Page 7: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/7.jpg)
7
Data Driven Parsing
• Parsing = Grammar + Algorithm• Probabilistic Context-Free Grammar
P(children=[Determiner, Adjective, Noun] | parent=NounPhrase)
![Page 8: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/8.jpg)
8
• Find the maximum likelihood parse tree from all grammatically valid candidates.
• The probability of a parse tree is the product of all its grammar rule (constituent) probabilities.
• The number of grammatically valid parse trees increases exponentially with the length of the sentence.
Data Driven Parsing
![Page 9: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/9.jpg)
9
Outline
• What is Natural Language Parsing?
• Data Driven Parsing
• Hypergraphs and Parsing Algorithms
• High Accuracy Parsing
• Coarse-to-Fine
• Empirical Results
![Page 10: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/10.jpg)
10
Hypergraphs
• A directed hypergraph can facilitate dynamic programming (Klein and Manning, 2001).
• A hyperedge connects a set of tail nodes to a set of head nodes.
Standard Edge Hyperedge
![Page 11: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/11.jpg)
11
Hypergraphs
![Page 12: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/12.jpg)
12
The CYK Algorithm
• Separates the hypergraph into “levels”• Exhaustively traverses every hyperedge, level by level
![Page 13: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/13.jpg)
13
The A* Algorithm
• Maintains a priority queue of traversable hyperedges• Traverses best-first until a complete parse tree is found
Pri
ori
ty Q
ueu
e
![Page 14: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/14.jpg)
14
Outline
• What is Natural Language Parsing?
• Data Driven Parsing
• Hypergraphs and Parsing Algorithms
• High Accuracy Parsing
• Coarse-to-Fine
• Empirical Results
![Page 15: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/15.jpg)
15
High(er) Accuracy Parsing
• Modify the Grammar to include more context• (Grand) Parent Annotation (Johnson, 1998)
P(children=[Determiner, Adjective, Noun] | parent=NounPhrase, grandParent=Sentence)
![Page 16: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/16.jpg)
16
Increased Search Space
Original Grammar
Parent AnnotatedGrammar
![Page 17: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/17.jpg)
17
Increased Search Space
Original Grammar
Parent AnnotatedGrammar
![Page 18: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/18.jpg)
18
Increased Search Space
Original Grammar
Parent AnnotatedGrammar
![Page 19: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/19.jpg)
19
Increased Search Space
Original Grammar
Parent AnnotatedGrammar
![Page 20: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/20.jpg)
20
Increased Search Space
Original Grammar
Parent AnnotatedGrammar
![Page 21: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/21.jpg)
21
Grammar Comparison
65
70
75
80
85
90
Ac
cu
rac
y %
• Exact Inference with the CYK algorithm becomes intractable.• Most algorithms using Lexical models resort to greedy search strategies.• We want to find the globally optimal (Viterbi) parse tree for these high- accuracy models efficiently.
![Page 22: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/22.jpg)
22
Outline
• What is Natural Language Parsing?
• Data Driven Parsing
• Hypergraphs and Parsing Algorithms
• High Accuracy Parsing
• Coarse-to-Fine
• Empirical Results
![Page 23: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/23.jpg)
23
Coarse-to-Fine• Efficiently find the optimal parse tree of a large, context-enriched
model (Fine) by following hyperedges suggested by solutions of a simpler model (Coarse).
• To evaluate the feasibility of Coarse-to-Fine, we use – Coarse = WSJ– Fine = Parent
65
70
75
80
85
90
Acc
ura
cy %
![Page 24: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/24.jpg)
24
Increased Search Space
Coarse Grammar
Fine Grammar
![Page 25: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/25.jpg)
25
Coarse-to-Fine
Build Coarse hypergraph
![Page 26: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/26.jpg)
26
Coarse-to-Fine
Choose a Coarse hyperedge
![Page 27: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/27.jpg)
27
Coarse-to-FineReplace the Coarse hyperedge with Fine hyperedge (modifies probability)
![Page 28: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/28.jpg)
28
Coarse-to-Fine
Propagate probability difference
![Page 29: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/29.jpg)
29
Coarse-to-Fine
Repeat until optimal parse treehas only Fine hyperedges
![Page 30: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/30.jpg)
30
Upper-Bound Grammar
• Replacing a Coarse hyperedge with a Fine hyperedge can increase or decrease its probability.
• Once we have found a parse tree with only Fine hyperedges, how can we be sure it is optimal?
• Modify the probability of Coarse grammar rules to be an upper-bound on the probability of Fine grammar rules.
nParentAPNn
,|max
nPFineNn
Coarse APAP max
where N is the set of non-terminals and is a grammar rule.A
![Page 31: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/31.jpg)
31
Outline
• What is Natural Language Parsing?
• Data Driven Parsing
• Hypergraphs and Parsing Algorithms
• High Accuracy Parsing
• Coarse-to-Fine
• Empirical Results
![Page 32: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/32.jpg)
32
ResultsComputational Time
0.001
0.01
0.1
1
10
100
5 7 9 11 13 15 17 19 21 23 25
Sentence Length
Tim
e (s
eco
nd
s)
CTF
CYK
A*
Search Guidance
1
10
100
1000
10000
100000
1000000
10000000
5 7 9 11 13 15 17 19 21 23 25
Sentence Length
Hy
pe
red
ge
s T
rav
ers
ed
CYK
A*
CTF
![Page 33: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/33.jpg)
33
Summary & Future Research
• Coarse-to-Fine is a new exact inference algorithm to efficiently traverse a large hypergraph space by using the solutions of simpler models.
• Full probability propagation through the hypergraph hinders computational performance. – Full propagation is not necessary; lower-bound of log2(n)
operations.
• Over 95% reduction in search space compared to baseline CYK algorithm.– Should prune even more space with higher-accuracy (Lexical)
models.
![Page 34: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/34.jpg)
34
Thanks
![Page 35: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/35.jpg)
35
Choosing a Coarse HyperedgeTop-Down vs. Bottom-Up
![Page 36: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/36.jpg)
36
Top-Down vs. Bottom-UpComputational Time Comparison
0
10
20
30
40
50
60
70
80
90
100
5 7 9 11 13 15 17 19 21 23 25
Sentence Length
Tim
e (s
eco
nd
s)
CTF Top-Down
CTF Bottom-Up
Search Guidance Comparison
0
50000
100000
150000
200000
250000
300000
5 7 9 11 13 15 17 19 21 23 25
Sentence Length
Hyp
ered
ges
Tra
vers
ed
CTF Top-Down
CTF Bottom-Up
• Top-Down• Traverses more hyperedges• Hyperedges are closer to the root• Requires less propagation (1/2)
• Bottom-Up• Traverses less hyperedges• Hyperedges are near the leaves (words) and shared by many trees• True probability of trees isn’t know at the beginning of CTF
![Page 37: Coarse-to-Fine Efficient Viterbi Parsing Nathan Bodenstab OGI RPE Presentation May 8, 2006](https://reader035.vdocuments.net/reader035/viewer/2022070401/56649f1f5503460f94c37d3a/html5/thumbnails/37.jpg)
37
Coarse-to-Fine Motivation
Optimal Coarse Tree
Optimal Fine Tree