1 an introduction to statistical machine translation dept. of csie, ncku yao-sheng chang date:...
Post on 20-Dec-2015
218 views
TRANSCRIPT
1
An Introduction to Statistical Machine
Translation
Dept. of CSIE, NCKUYao-Sheng ChangDate: 2011.04.12
2
Outline
Introduction Peter Brown
The Mathematics of Machine Translation: Parameter Estimation, computational linguistics, vol. 19,1993, pp.263-311.
Model 1
3
Introduction (1)
Machine translation is available Statistical method, information theory Faster computer, large storage Machine-readable corpora
Statistical method have proven their value Automatic speech recognition Lexicography, Natural language
processing
4
Introduction (2)
Translations involve many cultural respects We only consider the translation of individual
sentence, just acceptable sentences.
Every sentence in one language is a possible translation of any sentence in the other Assign (S,T) a probability, Pr(T|S), to be the
probability that a translator will produce T in the target language when presented with S in the source language.
5
Statistical Machine Translation(SMT)
Noise channel problem
6
Fundamental of SMT
Given a string of French f, the job of our translation system is to find the string e that the speaker had in mind when he produced f. (Baye’s theorem)
Since denominator Pr(f) here is a constant, the best e is one which has the greatest probability.
7
Practical Challenges
Computation of translation model Pr(f|e) Computation of language model Pr(e) Decoding (i.e., search for e that maximize Pr(f|e) Pr(e))
8
Alignment of case 1
9
Alignment of case 2
10
Alignment of case 3
11
Formulation of Alignment(1) Let e = e1
le1e2…el and f = f1m f1f2…fm
An alignment between a pair of strings e and f use a mapping of every word ei to some word fj
In other words, an alignment a between e and f tells that the word ei, 1 i l is generated by the word faj, aj{1,…,m}
There are (l+1)m different alignments between e and f. (Including Null – no mapping )
e = e1e2…ei…el
f = f1 f2… fj… fm
aj =i
faj =ei
12
Formulation of Alignment(2)
Probability of an alignment
13
Translation Model
The alignment, a, can be represented by a series, a1
m = ala2... am, of m values, each between 0 and l such that if the word in position j of the French string is connected to the word in position i of the English string, then aj = i , and if it is not connected to any English word, then aj = 0 (null).
14
IBM Model I (1)
15
IBM Model I (2)
The alignment is determined by specifying the values of aj for j from 1 to m, each of which can take any value from 0 to l.
16
Constrained Maximization
We wish to adjust the translation probabilities so as to maximize Pr(f|e ) subject to the constraints that for each e
17
Lagrange Multipliers (1) Method of Lagrange multipliers(拉格朗乘數法) : Lagrange multipliers with one constraint
If there is a maximum or minimum subject to the constraint g(x,y) = 0, then it will occur at one of the critical numbers of the function F defined by is called the
f(x,y) is called the objective function(目標函數) . g(x,y) is called the constrained equation(條件限制方程式) .
F(x, y, ) is called the Lagrange function(拉格朗函數) . is called the Lagrange multiplier (拉格朗乘數) .
),(),(),,( yxgyxfyxF
18
Lagrange Multipliers (2) Example 1: Maximize
Subject to Let
Set
代入 (2) 與 (3) ,可得
(5) 與 (6) 代入 (4) ,可得 ,由此可得 因此,最大值為
xyzV
024346 zyx
)24346(),,,( zyxxyzzyxF
)4(024346
)3(03
)2(04
)1(06
zyxF
xyF
xzF
yzF
z
y
x
6)1(
yz
)5(2
30
64 xyyz
xz
)6(20
63 xzyz
xy
0241824)2(32
346
xxxx
3
4 x
y z 28
3,
9
64
3
8)2(
3
4
xyzV
19
Lagrange Multipliers (3)
Following standard practice for constrained maximization, we introduce Lagrange multipliers e, and seek an unconstrained extremum of the auxiliary function
20
Derivation (1)
The partial derivative of h with respect to t(f|e) is
where is the Kronecker delta function, equal to one when both of its arguments are the same and equal to zero otherwise
21
Derivation (2)
We call the expected number of times that e connects to f in the translation (f|e) the count of f given e for (f|e) and denote it by c(f|e; f, e). By definition,
22
Derivation (3)
replacing e by ePr(f|e), then Equation (11) can be written very compactly as
In practice, our training data consists of a set of translations, (f(1) le(1)), (f(2)
le(2)), ..., (f(s) le(s)), , so this equation becomes
23
Derivation (4)
For an expression that can be evaluated efficiently.
24
Derivation (5)
Thus, the number of operations necessary to calculate a count is proportional to l + m rather than to (l + 1)m as Equation (12)
25
EM Algorithm
26
EM Algorithm
27
Introduction(1)
In statistical computing, an expectation-maximization (EM) algorithm is an algorithm for finding maximum likelihood estimates of parameters in probabilistic models, where the model depends on unobserved latent variables. EM is frequently used for data clustering in machine learning and computer vision.
28
Introduction(2)
EM alternates between performing an expectation (E) step, which computes an expectation of the likelihood by including the latent variables as if they were observed, and a maximization (M) step, which computes the maximum likelihood estimates of the parameters by maximizing the expected likelihood found on the E step. The parameters found on the M step are then used to begin another E step, and the process is repeated.
(From: http://en.wikipedia.org/wiki/Expectation-maximization_algorithm )
29
EM algorithm is a soft version of K-means clustering.
The idea is that the observed data are generated by several underlying causes.
Each cause contributes independently to the generation process, bur we only see the final mixture –without information about which cause contributed what.
30
Observable data
Each
Unobservable / hidden data
Each zij can be interpreted as cluster membership probabilities.
The component zij is 1 if object i is a member of cluster j.
}{ iX x
Timii xx ),,( 1 x
}{ izZ
Ti ikii zzzz }{ ,,2,1
31
Initial Assumption
At first , suppose we have a data set , where each is the vector
that correspond to the ith data point.
Further , assume the samples are drawn from k mixture Gaussians , .
Notice that the p.d.f. of multivariate normal distribution is
}{ iX x
Timii xx ),,( 1 x
)()(
2
1exp
)2(
1),;(p 1
j jjT
j
jm
jj
xxx
jc kj 1
A normal distribution in a x variate with mean and 2variance is a statistic distribution with probability function
32
E-step
Let be a n by k matrix , where
Notice that , if we set for then by Bayes formula we have
ijhH
kcj
1)p(
)|p(
)|p(
)p()|p(
)p()|p()|p(
11li
k
l
ji
lli
k
l
jjiij
cx
cx
ccx
ccxxc
kj 1
);|(
);();();(*1);(
1
|||
li
k
l
jiiijiijiijij
cxp
c|xpxzPxzPxzEh
33
M-step
ij
n
i
iij
n
ij
h
xh
1
1'
i
ij
n
i
ijn
ix
h
h
1
1
ij
n
i
Tjijiij
n
ij
h
xxh
1
''
1'))((
G
Tjiji
ij
n
i
ijn
ixx
h
h))(( ''
1
1
n
hh
h
h
h
h ij
n
in
i
ij
n
i
ij
k
j
n
i
ij
n
i
ij
n
i
k
j
ij
n
ij
1
1
1
11
1
11
1'
1
.
34
log likelihood
The log likelihood of the data set X given theparameters is
,
where , and is the weight of cluster j .
Notice that
),;(log)P(log)|(111
jjijj
k
j
n
ii
n
ixpxXl
),;(log11
jjijj
k
j
n
ixp
Tk),,( 1 ),,( jjjj
j
11
j
k
j
35
計算示範 (1)
假設
則
4.01.0
3.02.0
),;(),;(
),;(),;(
22221121
22121111
xpxp
xpxpN
8.02.0
6.04.0
4.01.0
4.0
4.01.0
1.03.02.0
3.0
3.02.0
2.0
H
0
11x
1
02x
36
計算示範 (2)
33.0
67.0
1
0
2.04.0
2.0
0
1
2.04.0
4.0'1
57.0
43.0
1
0
8.06.0
8.0
0
1
8.06.0
6.0'2
8.02.0
6.04.0H
0
11x
1
02x
22.022.0
22.022.0
33.01
67.00
33.01
67.00
2.04.0
2.0
33.00
67.01
33.00
67.01
2.04.0
4.0'1
TT
25.025.0
25.025.0
57.01
43.00
57.01
43.00
8.06.0
8.0
57.00
43.01
57.00
43.01
8.06.0
6.0'2
TT
3.02
2.04.0'1
7.0
2
8.06.0'2
48.2)4.07.01.03.0log()3.07.02.03.0log()|( Xl