presenter: jinhua du ( 杜金华 ) xi’an university of technology 西安理工大学
DESCRIPTION
NLP&CC, Chongqing, Nov. 17-19, 2013. Discriminative Latent Variable Based Classifier for Translation Error Detection. Presenter: Jinhua Du ( 杜金华 ) Xi’an University of Technology 西安理工大学. Outline. Introduction. 2. DPLVM for Translation Error Detection. 3. Experiments and Analysis. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Presenter: Jinhua Du ( 杜金华 ) Xi’an University of Technology 西安理工大学](https://reader036.vdocuments.net/reader036/viewer/2022081417/56813ac8550346895da2df8e/html5/thumbnails/1.jpg)
Presenter: Jinhua Du (杜金华 )
Xi’an University of Technology西安理工大学
NLP&CC, Chongqing, Nov. 17-19, 2013
Discriminative Latent Variable Based Classifier for Translation Error
Detection
![Page 2: Presenter: Jinhua Du ( 杜金华 ) Xi’an University of Technology 西安理工大学](https://reader036.vdocuments.net/reader036/viewer/2022081417/56813ac8550346895da2df8e/html5/thumbnails/2.jpg)
Outline
1. Introduction
2. DPLVM for Translation Error Detection
3. Experiments and Analysis
4. Conclusions and Future Work
![Page 3: Presenter: Jinhua Du ( 杜金华 ) Xi’an University of Technology 西安理工大学](https://reader036.vdocuments.net/reader036/viewer/2022081417/56813ac8550346895da2df8e/html5/thumbnails/3.jpg)
1. IntroductionProblem
1. In localization industry, human is always involved in post-editing the MT results;
2. MT errors always increase human cost to obtain a reasonable translation;
3. Translation error detection or word confidence estimation can improve working efficiency of post-editors in some extent.
1. In localization industry, human is always involved in post-editing the MT results;
2. MT errors always increase human cost to obtain a reasonable translation;
3. Translation error detection or word confidence estimation can improve working efficiency of post-editors in some extent.
•Research Question: how to improve the detection accuracy of detecting translation errors?
![Page 4: Presenter: Jinhua Du ( 杜金华 ) Xi’an University of Technology 西安理工大学](https://reader036.vdocuments.net/reader036/viewer/2022081417/56813ac8550346895da2df8e/html5/thumbnails/4.jpg)
Blatz et al. combined the neuralnetwork and a naive Bayes classifier
2004
Ueffing and Ney exhaustively explored various kinds of WPP features
2003/2007
Specia et al. worked on confidence estimation in CAT field2009/20
11
Xiong et al. used a MaxEnt-based classifier to predict translation errors
2010
1. IntroductionRelated Work
![Page 5: Presenter: Jinhua Du ( 杜金华 ) Xi’an University of Technology 西安理工大学](https://reader036.vdocuments.net/reader036/viewer/2022081417/56813ac8550346895da2df8e/html5/thumbnails/5.jpg)
For same feature set,
different classifiers show
different performance,
thus how to select/design a
proper classifier is
important
Classifiers
For a classifier, different
features reflect different
characteristics of problem,
how to select/design a
feature set is crucial
Features
1. IntroductionKey Factors
![Page 6: Presenter: Jinhua Du ( 杜金华 ) Xi’an University of Technology 西安理工大学](https://reader036.vdocuments.net/reader036/viewer/2022081417/56813ac8550346895da2df8e/html5/thumbnails/6.jpg)
Title in hereFeature set
Title in here
Comparison withSVM andMaxEnt
Title in hereDiscriminative
Latent Variableclassifier
1. IntroductionOur Work
![Page 7: Presenter: Jinhua Du ( 杜金华 ) Xi’an University of Technology 西安理工大学](https://reader036.vdocuments.net/reader036/viewer/2022081417/56813ac8550346895da2df8e/html5/thumbnails/7.jpg)
2. DPLVM Algorithm
Conditions: a sequence of observations x = {x1, x2,…, xm}
a sequence of labels y = {y1, y2,…, ym}
Assumption: a sequence of latent variables h = {h1, h2,…, hm}
Goal: to learn a mapping between x and y
Definition:( | , ) ( | , , ) ( | , )P P P
h
y x y h x h x (1)
![Page 8: Presenter: Jinhua Du ( 杜金华 ) Xi’an University of Technology 西安理工大学](https://reader036.vdocuments.net/reader036/viewer/2022081417/56813ac8550346895da2df8e/html5/thumbnails/8.jpg)
Simplified Algorithm
Assumptions: the model is restricted to have disjoint sets of latent
variables associated with each class label; Each hhjj is a member in a set HHyyjj of possible latent variables
for the class label yyj j ; so sequences which have any will by definition
have
Equation (1) can be re-written as:
where
jj yh H
( | , ) 0P y x
1...
( | , ) ( | , )y ym
P P
h H H
y x h x (2)
exp ( , )( | , )
exp ( , )h
P
f h x
h xf h x (3)
![Page 9: Presenter: Jinhua Du ( 杜金华 ) Xi’an University of Technology 西安理工大学](https://reader036.vdocuments.net/reader036/viewer/2022081417/56813ac8550346895da2df8e/html5/thumbnails/9.jpg)
Parameter Estimation
Decoding for test set:
Decoding algorithm: Sun and Tsujii (2009): a latent-dynamic inference (LDI)
method based on A* search and dynamic programming;
![Page 10: Presenter: Jinhua Du ( 杜金华 ) Xi’an University of Technology 西安理工大学](https://reader036.vdocuments.net/reader036/viewer/2022081417/56813ac8550346895da2df8e/html5/thumbnails/10.jpg)
DPLVM in Translation Error Detection Task
Prerequisites: Types of errors can be classified; Each class has a specific label; The classification task can be regarded as a labelling task;
2 Classes of word label C: correct Good words label: c I: incorrect Bad words label: i
![Page 11: Presenter: Jinhua Du ( 杜金华 ) Xi’an University of Technology 西安理工大学](https://reader036.vdocuments.net/reader036/viewer/2022081417/56813ac8550346895da2df8e/html5/thumbnails/11.jpg)
Feature Set
Word Posterior
Probabilities
• Fixed position based WPP• Flexible position based WPP• Word alignment based WPP
Lexical
Features
• Part of speech (POS)
• word entity
Syntactic
Features
• word links from LG parser
![Page 13: Presenter: Jinhua Du ( 杜金华 ) Xi’an University of Technology 西安理工大学](https://reader036.vdocuments.net/reader036/viewer/2022081417/56813ac8550346895da2df8e/html5/thumbnails/13.jpg)
3. Experiments and Analysis
Experimental Settings – SMT system
•Language pair: Chinese-English•Training set: NIST data set,3.4m•Devset: NIST MT 2006 current set•Testset: NIST MT 2005,2008 sets
SMT Performance
![Page 14: Presenter: Jinhua Du ( 杜金华 ) Xi’an University of Technology 西安理工大学](https://reader036.vdocuments.net/reader036/viewer/2022081417/56813ac8550346895da2df8e/html5/thumbnails/14.jpg)
Experimental Settings for Error Detection Task
• Devset: translations of NIST MT-08
• Testset: translations of NIST MT-05• Annotation: TER to determine the true labels for words, 37.99% ratio of correct words for MT-08, 41.59% RCW for MT-05
Data Set and Data Annotation
Evaluation MetricsEvaluation Metrics
![Page 17: Presenter: Jinhua Du ( 杜金华 ) Xi’an University of Technology 西安理工大学](https://reader036.vdocuments.net/reader036/viewer/2022081417/56813ac8550346895da2df8e/html5/thumbnails/17.jpg)
Observations
• The name entities are prone to be wrongly classified
• The prepositions, conjunctions, auxiliary verbs and articles are easier to be wrongly classified
• The proportion of the notional words that are wrongly classified is relatively small
![Page 18: Presenter: Jinhua Du ( 杜金华 ) Xi’an University of Technology 西安理工大学](https://reader036.vdocuments.net/reader036/viewer/2022081417/56813ac8550346895da2df8e/html5/thumbnails/18.jpg)
4. Conclusions and Future Work
Conclusions
Presents a new classifier - DPLVM-based classifier -for translation error detection
Introduces three different kinds of WPP features,three linguistic features
Compares the MaxEnt classifier, SVM classifier and our DPLVM classifier
The proposed classifier performs best compared to two other individual classifiers in terms of CER
![Page 19: Presenter: Jinhua Du ( 杜金华 ) Xi’an University of Technology 西安理工大学](https://reader036.vdocuments.net/reader036/viewer/2022081417/56813ac8550346895da2df8e/html5/thumbnails/19.jpg)
introducing paraphrases to annotate the hypothesesintroducing paraphrases to annotate the hypotheses
introducing new useful features to further improve the detection capability
performing experiments on more language pairs to verify our proposed method.
4. Conclusions and Future Work
Future Work