computing paraphrasability of syntactic variants using web...
TRANSCRIPT
![Page 1: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/1.jpg)
Computing Paraphrasability of Syntactic VariantsUsing Web Snippets
Atsushi FUJITA and Satoshi SATO
Nagoya Univ., Japan
< IJCNLP 2008, Jan. 10th, 2008 >
![Page 2: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/2.jpg)
2
Automatic Paraphrasing
Fundamental in NLP Recognition: IR, IE, QA, Multi-Doc.Summarization
Generation: MT, TTS, Authoring aids
Resources required Handcrafted knowledge
Thesauri [Many work]
Transformation rules [Mel’cuk+, 87] [Dras, 99] [Jacquemin, 99]
Automatic knowledge acquisition Distributional similarity [Lin+, 01] [Szpektor+, 04]
Aligning comparable/bilingual corpora [Many work]
![Page 3: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/3.jpg)
3
Paraphrase Knowledge
Template-like representation Lexical paraphrases
Morpho-syntactic paraphrases (syntactic variants)
Lack of applicability conditions ⇒ incorrect results
X wrote Y X is the author of Y
X show a A Y X v(Y) adv(A)
X solves Y X deals with Y
X V Y Y be v(Z)-PP by X Passivization
Removing light-verb
![Page 4: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/4.jpg)
4
Task Description
Computing paraphrasability between phrases Input: automatically generated paraphrase candidates
Pair of original and generated phrases (s and t)
Output: paraphrasability score [0,1] Is t grammatical ?
Does t hold if s holds ? (semantic equivalence or inclusion)
Is t syntactically substitutable for s in some context ?
We hurry checking it We check it in a hurrys t
![Page 5: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/5.jpg)
5
Issues and Solutions
How to measure similarity between phrases ? Contextual similarity: distributional similarity
Bag of words / Bag of dependency relations
Constituent similarity: handling syntactic variants Syntactic transformation + Lexical derivation
How to deal with data sparseness problem ? Collect example sentences of phrases from Web snippets
Assessing grammaticality
![Page 6: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/6.jpg)
Outline
1. Task Description
2. Paraphrases Handled3. Proposed Method
4. Experiments
5. Discussion
6. Conclusion
![Page 7: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/7.jpg)
7
Paraphrases of Predicate Phrases
Symmetric vs. Asymmetric
Equivalent / Inclusion / Entailment vs. Inference
Lexical vs. Morpho-syntactic
X change Y X modify Y
X show a A Y X v(Y) adv(A)
X married Y X dated Y
X snore Y X sleep Y
X sprint X run
X change Y X modify Y
X change Y X modify Y
X sprint X run
![Page 8: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/8.jpg)
8
Paraphrases Handled
Morpho-syntactic paraphrases (syntactic variants) Syntactic transformation + Lexical derivation
Constituent similarity is guaranteed a little
e.g. Head-switching, Light-verb construction, Category-shift
Predicate phrases of Japanese
We hurry checking it We check it in a hurry
Climate is in our favor Climate is favorable for us
kakunin-o isogu isoide kakunin-suruchecking-ACC to hurryto hurry checking it
in a hurry to checkto check it in a hurry
![Page 9: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/9.jpg)
9
Syntactic Variant Generator for Japanese
{v(kakunin) : genVoice() : genTense()}
okakuninN
:C
: isoguV
Trans. Pat.N:C:V ⇒ adv(V):vp(N)
adv(isogu) : vp(kakunin)
Gen. Func.vp(N)
kakunin-suru
Lex. Func.v(N)
Gen. Func.genVoice()
Gen. Func.genTense()
isoide
Lex. Func.adv(V)
{φ, reru/rareru, seru/saseru} {φ, ta/da}
isoide : {kakunin-suru : {φ, reru/rareru, seru/saseru} : {φ, ta/da}}
checking: ACC: to hurry[Fujita+, 07]
![Page 10: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/10.jpg)
Outline
1. Task Description
2. Paraphrases Handled
3. Proposed Method4. Experiments
5. Discussion
6. Conclusion
![Page 11: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/11.jpg)
11
Overview
s t
1.SnippetRetrieval
3.ParaphrasabilityComputation
Par (s⇒t)
Snippetsmimoto
kakunin-o isogu isoide kakunin-suruCandidateGeneration
1.SnippetRetrieval
2.FeatureExtraction
2.FeatureExtraction
SnippetsAnchor
Features Features
Paraphrasability score
to hurry checking it to check it in a hurry
identity
![Page 12: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/12.jpg)
12
Step 1. Snippet Retrieval
``Phrase search’’ Yahoo! JAPAN Web-search API
500 top snippets
![Page 13: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/13.jpg)
13
Step 2. Feature Extraction
HITS: # of pages Yahoo! API returns Larger HITS ⇒ t is more likely grammatical
BOW: content words around the phrase in snippets BOWs surrounding s and t have similar distribution⇒ s and t are semantically similar
MOD: modifiers and modifiees of the phrase in snippets s and t share a number of modifiers and modifiees⇒ s and t are syntactically substitutable
![Page 14: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/14.jpg)
14
Extracting MOD Features
Modifier / modifiee chunk (bunsetsu) Relation types (Depend / Appositive / Parallel)
Base form of the head-word (content word)
Some types of functional words (if any)
kuwashiku
jikken-kekka-no
saigen-sei-o
kenshou-suru
yotei-da
kare-no
FeaturesSentence within snippet(dependency tree) Modifiee/D: yotei
Modifier/D: kuwashii
Modifier/D: kare_no
(plan)
(in detail)
(his)
Given phrase
(I am) planning to verify the reproducibility of his experimental result in detail.
![Page 15: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/15.jpg)
15
Step 2. Feature Extraction (Anc)
Source-focused feature extraction 1. Determine anchor a which strongly associates with s
Noun which most frequently modifies s (one of MOD features)
2. Retrieve snippets for s AND a and t AND a 3. Extract BOW and MOD features from those snippets
doriburu-de kake agaru saido
s a
smile-ACC to showbe smiling
from ear to ear
dribble-with to run to go upto overlap by dribbling
emi-o ukaberu manmen
side
![Page 16: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/16.jpg)
16
Lin: Lin’s measure [Lin+, 01]
Fs, Ft : Feature sets for s and t w(x, f) : Weight of feature f in Fx (frequency in snippets)
skew: α-skew divergence [Lee, 99]
Ps = P(f | s), Pt = P(f | t) α : approximation degree of KL divergence [0,1]
Step 3. Paraphrasability Computation
![Page 17: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/17.jpg)
17
Summary
Features: Contextual features of entire phrase c.f. Marginal features [Torisawa, 06] [Pantel+, 07]
BOW, MOD
Weight of features: Frequency in snippets c.f. pair-wise MI [Lin+, 01] [Pantel+, 07]
c.f. Relative Focus Feature [Geffet+, 05]
DS measures Lin’s measure (symmetric) [Lin+, 01]
α-skew divergence (asymmetric) [Lee, 99]
![Page 18: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/18.jpg)
Outline
1. Task Description
2. Paraphrases Handled
3. Proposed Method
4. Experiments5. Discussion
6. Conclusion
![Page 19: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/19.jpg)
19
Setup: Candidate Generation
6 basic phrase types
Most frequent 1,000+ phrases for each type Mainichi newspaper corpus (1991-2005, 1.5GB)
Referring to dependency trees
Syntactic variant generator for Japanese [Fujita+, 07]Trans. Pat.
N:C:V ⇒ adv(V):vp(N)Gen. Func.
vp(N)Lex. Func.
adv(V)
![Page 20: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/20.jpg)
20
takai hyouka-o ukerutoukei-o tori-hajimeru
kentou-o sarani susumeru
nodo-ga itai
N:C:V
N:C:V1:V2
N:C:Adv:V
N:C:AdjN1:N2:C:V
Adj:N:C:V
kakunin-o isoguchecking-ACC to hurryto hurry checking it
songai-baisho-o motomeru
consideration-ACC further to go aheadto take consideration further
damage-reparation-ACC to demandto demand reparation for damage
statistics-ACC to take-to startto start collect statistics
high assessment-ACC to receiveto be rated high
throat-NOM be painfulto have a sore throat
Examples of Phrases
![Page 21: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/21.jpg)
21
Setup: Computing Paraphrasability Scores
15 measures: Proposed: {HITS,{BOW,MOD,HAR}×{Lin,skew}}×{Nor,Anc}
BL (Mainichi): HITS using 1.5GB newspaper corpus
![Page 22: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/22.jpg)
22
Evaluation 1: Ev.Gen
Question Can a correct paraphrase have the highest score among
candidates for a source phrase ?
Judgment (2 assesors) For 200 input, the best candidates of 15 models
![Page 23: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/23.jpg)
23
Mainichi << *.HITS ≒ *.BOW.* < *.MOD.* ≒ *.HAR.* Web enables us to compute paraphrasability accurately
Candidates with higher scores are more likely correct e.g. Lenient Prec. over 93% (th=0.5)
Nor.X.* ≒ Anc.X.* (discuss later)2 judges’ OK
1 or 2 judges’ OK
Results 1: Ev.Gen
![Page 24: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/24.jpg)
24
Evaluation 2: Ev.Rec
Question How is the method useful for collecting paraphrase
instances ?
Judgment (2 assesors) 200 best candidates for each of 15 models
![Page 25: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/25.jpg)
25
Results 2: Ev.Rec
Mainichi ≒ *.HITS << *.BOW.* ≒ *.MOD.* ≒ *.HAR.* DS measures outperformed HITS
Lenient Prec. almost reach a ceiling
Nor.X.* ≒ Anc.X.* again Anchor selection might be inappropriate 2 or more content words make s rarely ambiguous
![Page 26: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/26.jpg)
26
Results 2: Ev.Rec
Remaining problems Dropping N1 from N1:N2:C:V
Typically functions as generalization
N1 sometimes plays as the semantic head of N1:N2
Solutions: Semantic parsing, Phrase boundary detection, etc.
shukketsu-taryou-de shibou-suru taryou-de shibou-surux loss-plenty-ABL to die(*) die due to plenty
blood loss-plenty-ABL to diedie due to heavy blood loss
yagai-konsatooutdoors-concert
konsatoconcerto
![Page 27: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/27.jpg)
Outline
1. Task Description
2. Paraphrases Handled
3. Proposed Method
4. Experiments
5. Discussion6. Conclusion
![Page 28: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/28.jpg)
28
Discussion: Issues Addressed
Measurement of paraphrasability between phrases Reasonably nice (Ev.Gen: over 65%, Ev.Rec: over 96%)
Combining constituent and contextual similarities
Room for improvement Feature selection [Hagiwara+, 08]
Feature weighting [Lin+, 01] [Geffet+, 05]
Data sparseness problem Not perfectly solved
TSUBAKI offers larger number of snippets [Shinzato+, 08]
![Page 29: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/29.jpg)
29
Discussion: Technical Issues
Coverage For 50% input, no candidate is output
More robust generation system To generate a wider range of paraphrases
To handle other types of phrases with less human-labor
Portability 90% of candidates are filtered out due to 0 HITS
Use SLMs to prune incorrect candidates before querying
![Page 30: Computing Paraphrasability of Syntactic Variants Using Web ...paraphrasing.org/~fujita/publications/mine/fujita-IJCNLP2008-slides.… · Computing Paraphrasability of Syntactic Variants](https://reader033.vdocuments.net/reader033/viewer/2022052408/5f0be6a07e708231d432c62e/html5/thumbnails/30.jpg)
30
Conclusion
Computing paraphrasability between phrases Input: paraphrase candidates
Automatically generated
Syntactic variants
Predicate phrases in Japanese
Output: paraphrasability score [0,1] Is t grammatical ?
Does t hold if s holds ? (semantic equivalence or inclusion)
Is t syntactically substitutable for s in some context ?
Proposed method achieved reasonable results Ev.Gen: over 65% (over 93% w/ th=0.5), Ev.Rec: over 96%