an svm based voting algorithm with application to parse reranking
DESCRIPTION
An SVM Based Voting Algorithm with Application to Parse Reranking. Paper by Libin Shen and Aravind K. Joshi Presented by Amit Wolfenfeld. Outline. Introduction of Parse Reranking SVM An SVM Based Voting Algorithm Theoretical Justification Experiments on Parse Reranking Conclusions. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.vdocuments.net/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/1.jpg)
An SVM Based Voting Algorithm with Application to Parse Reranking
Paper by Libin Shen and Aravind K. JoshiPresented by Amit Wolfenfeld
![Page 2: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.vdocuments.net/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/2.jpg)
OutlineIntroduction of Parse RerankingSVMAn SVM Based Voting AlgorithmTheoretical JustificationExperiments on Parse RerankingConclusions
![Page 3: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.vdocuments.net/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/3.jpg)
Introduction – Parse RerankingMotivation (Collins)vote rerank f-
scoreLog-
likelihood
parses rank
3 92% -120.0 P2 1
4 90% -121.5 P3 2
x 1 96% -122.0 P1 3
2 93% -122.5 P4 4
![Page 4: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.vdocuments.net/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/4.jpg)
Support Vector MachinesThe SVM is a large margin
classifier that searches for the hyperplane that maximizes the margin between the positive samples and the negative samples
![Page 5: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.vdocuments.net/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/5.jpg)
Support Vector MachinesMeasures of the capacity of a
learning machine: VC Dimension, Fat Shattering Dimension
The capacity of a learning machine is related to the margin on the training data.- As the margin goes up, VC-dimension may go down and thus the upper bound of the test error goes down. (Vapnik 79)
![Page 6: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.vdocuments.net/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/6.jpg)
Support Vector MachinesSVMs’ theoretical accuracy is
much lower than their actual performance. The margin based upper bounds of the test error are too loose.
This is why – SVM based voting algorithm.
![Page 7: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.vdocuments.net/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/7.jpg)
SVM Based VotingPrevious work (Dijkstra 02)
- Use SVM for parse reranking directly.- Positive samples: parse with highest f-score for each sentence.
First try-Tree kernel: compute dot-product on the space of all the subtrees (Collins 02)-Linear kernel: rich features (Collins 00)
![Page 8: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.vdocuments.net/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/8.jpg)
SVM based Voting AlgorithmUsing pairwise parses as samplesLet is the j-th candidate parse
for the i-th sentence in the training data.
Let is the parse with highest f-score among all the parses for the i-th sentence.
Positive samples: Negative samples:
![Page 9: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.vdocuments.net/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/9.jpg)
Preference KernelsLet are two pairs of parses K – kernel : linear or tree kernelThe preference kernel is defined:
- +
A sample represents the difference between a good parse and a bad one, the preference computes the similarity between the two differences.
![Page 10: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.vdocuments.net/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/10.jpg)
SVM based Voting Decision function f of SVM:
for each of the pair parses:
is the i-th support vectoris the total number of support vectorsis the class of can be is the Lagrange multiplier solved by the SVM
![Page 11: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.vdocuments.net/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/11.jpg)
Theoretical IssuesJustifying the Preference Kernel
Justifying Pairwise Samples
Margin Based Bound for the SVM Based Voting Algorithm
![Page 12: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.vdocuments.net/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/12.jpg)
Justifying the Preference KernelThe kernelThe preference kernel
- - + - )(- )
![Page 13: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.vdocuments.net/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/13.jpg)
Justifying the Pairwise SamplesThe SVM using simple parses as
samples searches for a decision function score constrained by the condition:- - too strong.
Pairwise:-
![Page 14: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.vdocuments.net/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/14.jpg)
Margin Based Bound for SVM Based voting
Loss function of voting :
Loss function of classification:Expected voting loss is equal expected
classification loss(Herbrich 2000)
![Page 15: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.vdocuments.net/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/15.jpg)
Experiments – WSJ TreebankN-best parsing results (Collins 02)SVM-light (Joachims 98)Two Kernels (K) used in the preference
kernel:- Linear Kernel- Tree Kernel
Tree Kernel- very slow
![Page 16: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.vdocuments.net/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/16.jpg)
Experiments – Linear KernelTraining data are cut into slices.
Slice i contains two pairwise samples of each sentence.
22 SVMs on 22 slices of training data.
2 days to train an SVM in a Pentium III 1.13Ghz.
![Page 17: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.vdocuments.net/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/17.jpg)
Results
![Page 18: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.vdocuments.net/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/18.jpg)
Conclusions
Using an SVM approach :
- achieving state-of-the-art
results
- SVM with linear kernel is
superior to tree kernel in speed
and accuracy.
![Page 19: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.vdocuments.net/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/19.jpg)
T
n
o u
k
h a
Y
!