prml 13.2.2: the forward-backward algorithm
TRANSCRIPT
![Page 1: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/1.jpg)
PRML 13.2.2
The forward-backward algorithm
August 4, 2014 by Shinichi TAMURA
![Page 2: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/2.jpg)
The forward-backward algorithm
Today's topics
The forward-backward algorithm 1. E-step of EM algorithm for HMM 2. Multiple shorter sequences 3. Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
![Page 3: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/3.jpg)
The forward-backward algorithm
Today's topics
The forward-backward algorithm 1. E-step of EM algorithm for HMM 2. Multiple shorter sequences 3. Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
![Page 4: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/4.jpg)
The forward-backward algorithm
Today's topics
The forward-backward algorithm 1. E-step of EM algorithm for HMM 2. Multiple shorter sequences 3. Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
![Page 5: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/5.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Quick review HMM is trained by EM algorithm E-step: Evaluate γ(zn) and ξ(zn-1, zn) M-step: Update parameters
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!k ="(z1k)!Kj=1 "(z1 j)
,
Ajk =
!Nn=2 #(zn!1, j, znk)!K
l=1!Nn=2 #(zn!1, j, znl)
.
![Page 6: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/6.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Quick review HMM is trained by EM algorithm E-step: Evaluate γ(zn) and ξ(zn-1, zn) M-step: Update parameters
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!k ="(z1k)!Kj=1 "(z1 j)
,
Ajk =
!Nn=2 #(zn!1, j, znk)!K
l=1!Nn=2 #(zn!1, j, znl)
.
![Page 7: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/7.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
ξ(zn-1, zn) γ(zn)
![Page 8: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/8.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
ξ(zn-1, zn) γ(zn) ? ?
![Page 9: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/9.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
How to calculate γ(zn), ξ(zn-1, zn) ? Basic idea is message passing on tree (It will be studied later, and we shall do conventional style now)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
![Page 10: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/10.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of γ(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(zn|x1:N)
=p(x1:N |zn)p(zn)
p(x1:N)
=p(x1:n|zn)p(zn)p(xn+1:N |zn)
p(x1:N)
![Page 11: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/11.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of γ(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(zn|x1:N)
=p(x1:N |zn)p(zn)
p(x1:N)
=p(x1:n|zn)p(zn)p(xn+1:N |zn)
p(x1:N)
![Page 12: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/12.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of γ(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(zn|x1:N)
=p(x1:N |zn)p(zn)
p(x1:N)
=p(x1:n|zn)p(zn)p(xn+1:N |zn)
p(x1:N)
![Page 13: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/13.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
zn+1 zNzn-1 znz1
xn+1 xNxn-1 xnx1
Head TailTail
![Page 14: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/14.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of γ(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(zn|x1:N)
=p(x1:N |zn)p(zn)
p(x1:N)
=p(x1:n|zn)p(zn)p(xn+1:N |zn)
p(x1:N)
![Page 15: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/15.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of γ(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(zn|x1:N)
=p(x1:N |zn)p(zn)
p(x1:N)
=p(x1:n|zn)p(zn)p(xn+1:N |zn)
p(x1:N)
![Page 16: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/16.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of γ(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(zn|x1:N)
=p(x1:N |zn)p(zn)
p(x1:N)
=p(x1:n|zn)p(zn)p(xn+1:N |zn)
p(x1:N)
β(zn) α(zn)
![Page 17: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/17.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
ξ(zn-1, zn) γ(zn) ? ?
![Page 18: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/18.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
ξ(zn-1, zn) γ(zn)
β(zn) α(zn) ?
?
?
p(x1:N) ?
![Page 19: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/19.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Now problem is changed into
How to calculate α(zn), β(zn) ?
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
![Page 20: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/20.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of α(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(x1:n|zn)p(zn)= p(xn|zn)p(x1:n!1|zn)p(zn)= p(xn|zn)p(x1:n!1, zn)= p(xn|zn)
!
zn!1
p(x1:n!1, zn|zn!1)p(zn!1)
= p(xn|zn)!
zn!1
p(x1:n!1|zn!1)p(zn|zn!1)p(zn!1)
= p(xn|zn)!
zn!1
!(zn!1)A.
![Page 21: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/21.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of α(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(x1:n|zn)p(zn)= p(xn|zn)p(x1:n!1|zn)p(zn)= p(xn|zn)p(x1:n!1, zn)= p(xn|zn)
!
zn!1
p(x1:n!1, zn|zn!1)p(zn!1)
= p(xn|zn)!
zn!1
p(x1:n!1|zn!1)p(zn|zn!1)p(zn!1)
= p(xn|zn)!
zn!1
!(zn!1)A.
![Page 22: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/22.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
zn-1 znz1
xn-1 xnx1
HeadTail
![Page 23: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/23.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of α(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(x1:n|zn)p(zn)= p(xn|zn)p(x1:n!1|zn)p(zn)= p(xn|zn)p(x1:n!1, zn)= p(xn|zn)
!
zn!1
p(x1:n!1, zn|zn!1)p(zn!1)
= p(xn|zn)!
zn!1
p(x1:n!1|zn!1)p(zn|zn!1)p(zn!1)
= p(xn|zn)!
zn!1
!(zn!1)A.
![Page 24: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/24.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of α(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(x1:n|zn)p(zn)= p(xn|zn)p(x1:n!1|zn)p(zn)= p(xn|zn)p(x1:n!1, zn)= p(xn|zn)
!
zn!1
p(x1:n!1, zn|zn!1)p(zn!1)
= p(xn|zn)!
zn!1
p(x1:n!1|zn!1)p(zn|zn!1)p(zn!1)
= p(xn|zn)!
zn!1
!(zn!1)A.
![Page 25: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/25.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of α(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(x1:n|zn)p(zn)= p(xn|zn)p(x1:n!1|zn)p(zn)= p(xn|zn)p(x1:n!1, zn)= p(xn|zn)
!
zn!1
p(x1:n!1, zn|zn!1)p(zn!1)
= p(xn|zn)!
zn!1
p(x1:n!1|zn!1)p(zn|zn!1)p(zn!1)
= p(xn|zn)!
zn!1
!(zn!1)A.
![Page 26: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/26.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of α(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(x1:n|zn)p(zn)= p(xn|zn)p(x1:n!1|zn)p(zn)= p(xn|zn)p(x1:n!1, zn)= p(xn|zn)
!
zn!1
p(x1:n!1, zn|zn!1)p(zn!1)
= p(xn|zn)!
zn!1
p(x1:n!1|zn!1)p(zn|zn!1)p(zn!1)
= p(xn|zn)!
zn!1
!(zn!1)A.
![Page 27: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/27.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of α(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(x1:n|zn)p(zn)= p(xn|zn)p(x1:n!1|zn)p(zn)= p(xn|zn)p(x1:n!1, zn)= p(xn|zn)
!
zn!1
p(x1:n!1, zn|zn!1)p(zn!1)
= p(xn|zn)!
zn!1
p(x1:n!1|zn!1)p(zn|zn!1)p(zn!1)
= p(xn|zn)!
zn!1
!(zn!1)A.
![Page 28: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/28.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
zn-1 znz1
xn-1 xnx1
Head
TailTail
![Page 29: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/29.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of α(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(x1:n|zn)p(zn)= p(xn|zn)p(x1:n!1|zn)p(zn)= p(xn|zn)p(x1:n!1, zn)= p(xn|zn)
!
zn!1
p(x1:n!1, zn|zn!1)p(zn!1)
= p(xn|zn)!
zn!1
p(x1:n!1|zn!1)p(zn|zn!1)p(zn!1)
= p(xn|zn)!
zn!1
!(zn!1)A.
![Page 30: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/30.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of α(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(x1:n|zn)p(zn)= p(xn|zn)p(x1:n!1|zn)p(zn)= p(xn|zn)p(x1:n!1, zn)= p(xn|zn)
!
zn!1
p(x1:n!1, zn|zn!1)p(zn!1)
= p(xn|zn)!
zn!1
p(x1:n!1|zn!1)p(zn|zn!1)p(zn!1)
= p(xn|zn)!
zn!1
!(zn!1)A.
![Page 31: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/31.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of α(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(x1:n|zn)p(zn)= p(xn|zn)p(x1:n!1|zn)p(zn)= p(xn|zn)p(x1:n!1, zn)= p(xn|zn)
!
zn!1
p(x1:n!1, zn|zn!1)p(zn!1)
= p(xn|zn)!
zn!1
p(x1:n!1|zn!1)p(zn|zn!1)p(zn!1)
= p(xn|zn)!
zn!1
!(zn!1)A.
![Page 32: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/32.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of α(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(x1:n|zn)p(zn)= p(xn|zn)p(x1:n!1|zn)p(zn)= p(xn|zn)p(x1:n!1, zn)= p(xn|zn)
!
zn!1
p(x1:n!1, zn|zn!1)p(zn!1)
= p(xn|zn)!
zn!1
p(x1:n!1|zn!1)p(zn|zn!1)p(zn!1)
= p(xn|zn)!
zn!1
!(zn!1)A.
![Page 33: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/33.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of α(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(xn|zn)!
zn!1
!(zn!1)A.
k = 1
k = 2
k = K
...
n = 1 n = 2 n = N...
O(K)
![Page 34: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/34.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of α(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(xn|zn)!
zn!1
!(zn!1)A.
k = 1
k = 2
k = K
...
n = 1 n = 2 n = N...
O(K2)
![Page 35: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/35.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of α(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(xn|zn)!
zn!1
!(zn!1)A.
k = 1
k = 2
k = K
...
n = 1 n = 2 n = N...
O(K2N)
![Page 36: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/36.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of α(zn) Start of recursion?
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(z1) = p(x1|z1)p(z1) =
!""""""""""#
"1p(x1|!1)...
"N p(x1|!N)
$%%%%%%%%%%&.
![Page 37: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/37.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of α(zn) Start of recursion?
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(z1) = p(x1|z1)p(z1) =
!""""""""""#
"1p(x1|!1)...
"N p(x1|!N)
$%%%%%%%%%%&.
![Page 38: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/38.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
ξ(zn-1, zn) γ(zn)
β(zn) α(zn) ?
?
?
p(x1:N) ?
![Page 39: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/39.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
ξ(zn-1, zn) γ(zn)
β(zn) α(zn) ?
?
p(x1:N) ?
![Page 40: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/40.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of β(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(xn+1:N |zn)=!
zn+1
p(xn+1:N |zn, zn+1)p(zn+1|zn)
=!
zn+1
p(xn+1:N |zn+1)p(zn+1|zn)
=!
zn+1
p(xn+2:N |zn+1)p(xn+1|zn+1)p(zn+1|zn)
=!
zn+1
!(zn+1)p(xn+1|zn+1)AT.
![Page 41: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/41.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of β(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(xn+1:N |zn)=!
zn+1
p(xn+1:N |zn, zn+1)p(zn+1|zn)
=!
zn+1
p(xn+1:N |zn+1)p(zn+1|zn)
=!
zn+1
p(xn+2:N |zn+1)p(xn+1|zn+1)p(zn+1|zn)
=!
zn+1
!(zn+1)p(xn+1|zn+1)AT.
![Page 42: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/42.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of β(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(xn+1:N |zn)=!
zn+1
p(xn+1:N |zn, zn+1)p(zn+1|zn)
=!
zn+1
p(xn+1:N |zn+1)p(zn+1|zn)
=!
zn+1
p(xn+2:N |zn+1)p(xn+1|zn+1)p(zn+1|zn)
=!
zn+1
!(zn+1)p(xn+1|zn+1)AT.
![Page 43: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/43.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
zn+2 zNzn zn+1
xn+2 xNxn xn+1
HeadTail
![Page 44: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/44.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of β(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(xn+1:N |zn)=!
zn+1
p(xn+1:N |zn, zn+1)p(zn+1|zn)
=!
zn+1
p(xn+1:N |zn+1)p(zn+1|zn)
=!
zn+1
p(xn+2:N |zn+1)p(xn+1|zn+1)p(zn+1|zn)
=!
zn+1
!(zn+1)p(xn+1|zn+1)AT.
![Page 45: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/45.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of β(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(xn+1:N |zn)=!
zn+1
p(xn+1:N |zn, zn+1)p(zn+1|zn)
=!
zn+1
p(xn+1:N |zn+1)p(zn+1|zn)
=!
zn+1
p(xn+2:N |zn+1)p(xn+1|zn+1)p(zn+1|zn)
=!
zn+1
!(zn+1)p(xn+1|zn+1)AT.
![Page 46: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/46.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of β(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(xn+1:N |zn)=!
zn+1
p(xn+1:N |zn, zn+1)p(zn+1|zn)
=!
zn+1
p(xn+1:N |zn+1)p(zn+1|zn)
=!
zn+1
p(xn+2:N |zn+1)p(xn+1|zn+1)p(zn+1|zn)
=!
zn+1
!(zn+1)p(xn+1|zn+1)AT.
![Page 47: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/47.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
zn+2 zNzn zn+1
xn+2 xNxn xn+1
TailTail
![Page 48: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/48.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of β(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(xn+1:N |zn)=!
zn+1
p(xn+1:N |zn, zn+1)p(zn+1|zn)
=!
zn+1
p(xn+1:N |zn+1)p(zn+1|zn)
=!
zn+1
p(xn+2:N |zn+1)p(xn+1|zn+1)p(zn+1|zn)
=!
zn+1
!(zn+1)p(xn+1|zn+1)AT.
![Page 49: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/49.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of β(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(xn+1:N |zn)=!
zn+1
p(xn+1:N |zn, zn+1)p(zn+1|zn)
=!
zn+1
p(xn+1:N |zn+1)p(zn+1|zn)
=!
zn+1
p(xn+2:N |zn+1)p(xn+1|zn+1)p(zn+1|zn)
=!
zn+1
!(zn+1)p(xn+1|zn+1)AT.
![Page 50: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/50.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of β(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(xn+1:N |zn)=!
zn+1
p(xn+1:N |zn, zn+1)p(zn+1|zn)
=!
zn+1
p(xn+1:N |zn+1)p(zn+1|zn)
=!
zn+1
p(xn+2:N |zn+1)p(xn+1|zn+1)p(zn+1|zn)
=!
zn+1
!(zn+1)p(xn+1|zn+1)AT.
![Page 51: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/51.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of β(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(xn+1:N |zn)=!
zn+1
p(xn+1:N |zn, zn+1)p(zn+1|zn)
=!
zn+1
p(xn+1:N |zn+1)p(zn+1|zn)
=!
zn+1
p(xn+2:N |zn+1)p(xn+1|zn+1)p(zn+1|zn)
=!
zn+1
!(zn+1)p(xn+1|zn+1)AT.
![Page 52: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/52.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of β(zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) =!
zn+1
!(zn+1)p(xn+1|zn+1)AT.
k = 1
k = 2
k = K
...
n = 1 n = 2 n = N...
O(K2N)
![Page 53: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/53.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of β(zn) Start of recursion?
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
p(zN |x1:N) =p(x1:N |zN)p(zN)!(zN)
p(x1:N)
! !(zN) =!1 · · · 1
"T.
![Page 54: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/54.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of β(zn) Start of recursion?
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
p(zN |x1:N) =p(x1:N |zN)p(zN)!(zN)
p(x1:N)
! !(zN) =!1 · · · 1
"T.
![Page 55: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/55.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
ξ(zn-1, zn) γ(zn)
β(zn) α(zn) ?
?
p(x1:N) ?
![Page 56: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/56.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
ξ(zn-1, zn) γ(zn)
β(zn) α(zn)
?
p(x1:N) ?
![Page 57: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/57.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of p(x1:N) It appears on the denominator of γ(zn).
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) ="(zn)#(zn)p(x1:N)
![Page 58: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/58.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of p(x1:N) It appears on the denominator of γ(zn). In fact, we don't need it for update because it cancel out.
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!k ="(z1k)!Kj=1 "(z1 j)
,
Ajk =
!Nn=2 #(zn!1, j, znk)!K
l=1!Nn=2 #(zn!1, j, znl)
.
![Page 59: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/59.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of p(x1:N) It appears on the denominator of γ(zn). In fact, we don't need it for update because it cancel out. However, we need to evaluate it, because it's LIKELIHOOD, which is monitored.
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
![Page 60: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/60.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of p(x1:N)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) ="(zn)#(zn)p(X) .
!!
zn
!(zn) ="zn "(zn)#(zn)p(X) .
! p(X) =!
zn
"(zn)#(zn) (!!! = 1).
" p(X) =!
zN
"(zN) =!
k"k(zN).
![Page 61: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/61.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of p(x1:N)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) ="(zn)#(zn)p(X) .
!!
zn
!(zn) ="zn "(zn)#(zn)p(X) .
! p(X) =!
zn
"(zn)#(zn) (!!! = 1).
" p(X) =!
zN
"(zN) =!
k"k(zN).
![Page 62: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/62.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of p(x1:N)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) ="(zn)#(zn)p(X) .
!!
zn
!(zn) ="zn "(zn)#(zn)p(X) .
! p(X) =!
zn
"(zn)#(zn) (!!! = 1).
" p(X) =!
zN
"(zN) =!
k"k(zN).Any n will do
![Page 63: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/63.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of p(x1:N)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) ="(zn)#(zn)p(X) .
!!
zn
!(zn) ="zn "(zn)#(zn)p(X) .
! p(X) =!
zn
"(zn)#(zn) (!!! = 1).
" p(X) =!
zN
"(zN) =!
k"k(zN).
![Page 64: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/64.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
ξ(zn-1, zn) γ(zn)
β(zn) α(zn)
?
p(x1:N) ?
![Page 65: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/65.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
ξ(zn-1, zn) γ(zn)
β(zn) α(zn)
?
p(x1:N)
Nth
![Page 66: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/66.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of ξ(zn-1, zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn!1, zn) = p(zn!1, zn|x1:N)
=p(x1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1, zn)p(xn|zn!1, zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn!1, zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)...
![Page 67: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/67.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of ξ(zn-1, zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn!1, zn) = p(zn!1, zn|x1:N)
=p(x1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1, zn)p(xn|zn!1, zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn!1, zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)...
![Page 68: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/68.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of ξ(zn-1, zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn!1, zn) = p(zn!1, zn|x1:N)
=p(x1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1, zn)p(xn|zn!1, zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn!1, zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)...
![Page 69: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/69.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
zn+1 zNzn-1 znz1
xn+1 xNxn-1 xnx1
Head TailTailTail
![Page 70: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/70.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of ξ(zn-1, zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn!1, zn) = p(zn!1, zn|x1:N)
=p(x1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1, zn)p(xn|zn!1, zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn!1, zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)...
![Page 71: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/71.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of ξ(zn-1, zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn!1, zn) = p(zn!1, zn|x1:N)
=p(x1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1, zn)p(xn|zn!1, zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn!1, zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)...
![Page 72: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/72.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of ξ(zn-1, zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn!1, zn) = p(zn!1, zn|x1:N)
=p(x1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1, zn)p(xn|zn!1, zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn!1, zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)...
![Page 73: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/73.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
zn+1 zNzn-1 znz1
xn+1 xNxn-1 xnx1
Head
TailTail
![Page 74: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/74.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of ξ(zn-1, zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn!1, zn) = p(zn!1, zn|x1:N)
=p(x1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1, zn)p(xn|zn!1, zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn!1, zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)...
![Page 75: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/75.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of ξ(zn-1, zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn!1, zn) = p(zn!1, zn|x1:N)
=p(x1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1, zn)p(xn|zn!1, zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn!1, zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)...
![Page 76: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/76.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of ξ(zn-1, zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn!1, zn) = p(zn!1, zn|x1:N)
=p(x1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1, zn)p(xn|zn!1, zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn!1, zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)...
![Page 77: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/77.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
zn+1 zNzn-1 znz1
xn+1 xNxn-1 xnx1
HeadTail
![Page 78: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/78.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of ξ(zn-1, zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn!1, zn) = p(zn!1, zn|x1:N)
=p(x1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1, zn)p(xn|zn!1, zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn!1, zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)...
![Page 79: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/79.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of ξ(zn-1, zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn!1, zn) = p(zn!1, zn|x1:N)
=p(x1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1, zn)p(xn|zn!1, zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn!1, zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)...
![Page 80: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/80.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of ξ(zn-1, zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn!1, zn) = p(zn!1, zn|x1:N)
=p(x1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1, zn)p(xn|zn!1, zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn!1, zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)...
![Page 81: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/81.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of ξ(zn-1, zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
...
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn)p(zn|zn!1)p(zn!1)
p(x1:N)
=!(zn!1)p(xn|zn)p(zn|zn!1)"(zn)
p(x1:N).
![Page 82: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/82.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of ξ(zn-1, zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
...
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn)p(zn|zn!1)p(zn!1)
p(x1:N)
=!(zn!1)p(xn|zn)p(zn|zn!1)"(zn)
p(x1:N).
![Page 83: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/83.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
zn+1 zNzn-1 znz1
xn+1 xNxn-1 xnx1
HeadTail
![Page 84: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/84.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of ξ(zn-1, zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
...
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn)p(zn|zn!1)p(zn!1)
p(x1:N)
=!(zn!1)p(xn|zn)p(zn|zn!1)"(zn)
p(x1:N).
![Page 85: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/85.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of ξ(zn-1, zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
...
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn)p(zn|zn!1)p(zn!1)
p(x1:N)
=!(zn!1)p(xn|zn)p(zn|zn!1)"(zn)
p(x1:N).
![Page 86: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/86.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of ξ(zn-1, zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
...
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn)p(zn|zn!1)p(zn!1)
p(x1:N)
=!(zn!1)p(xn|zn)p(zn|zn!1)"(zn)
p(x1:N).
β(zn) α(zn)
![Page 87: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/87.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Evaluation of ξ(zn-1, zn)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
...
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn!1, zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn)p(zn!1, zn)
p(x1:N)
=p(x1:n!1|zn!1)p(xn|zn)p(xn+1:N |zn)p(zn|zn!1)p(zn!1)
p(x1:N)
=!(zn!1)p(xn|zn)p(zn|zn!1)"(zn)
p(x1:N).
![Page 88: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/88.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
ξ(zn-1, zn) γ(zn)
β(zn) α(zn)
?
p(x1:N)
Nth
![Page 89: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/89.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
ξ(zn-1, zn) γ(zn)
β(zn) α(zn)
p(x1:N)
Nth
![Page 90: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/90.jpg)
The forward-backward algorithm
E-step of EM algorithm for HMM
Summary HMM is trained by EM algorithm E-step: Evaluate α(zn) and β(zn)
M-step: Update parameters
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(zn) = p(xn|zn)!
zn!1
!(zn!1)p(zn|zn!1),
"(zn) =!
zn+1
"(zn+1)p(xn+1|zn+1)p(zn+1|zn).
![Page 91: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/91.jpg)
The forward-backward algorithm
Today's topics
The forward-backward algorithm 1. E-step of EM algorithm for HMM 2. Multiple shorter sequences 3. Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
![Page 92: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/92.jpg)
The forward-backward algorithm
Today's topics
The forward-backward algorithm 1. E-step of EM algorithm for HMM 2. Multiple shorter sequences 3. Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
![Page 93: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/93.jpg)
The forward-backward algorithm
Multiple shorter sequences
In some practical situation, we can't get a long sequence.
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
![Page 94: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/94.jpg)
The forward-backward algorithm
Multiple shorter sequences
In some practical situation, we can't get a long sequence. Instead, we get multiple shorter sequences.
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
![Page 95: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/95.jpg)
The forward-backward algorithm
Multiple shorter sequences
In some practical situation, we can't get a long sequence. Instead, we get multiple shorter sequences. In this situation, we are still able to use forward-backward algorithm (w/ bit modification).
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
![Page 96: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/96.jpg)
The forward-backward algorithm
Multiple shorter sequences
E-step Just evaluate independently
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(z(r)n ) = p(z(r)n |X(r), !old),"(z(r)n!1, z
(r)n ) = p(z
(r)n!1, z
(r)n |X(r), !old).
![Page 97: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/97.jpg)
The forward-backward algorithm
Multiple shorter sequences
E-step Just evaluate independently
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!(z(r)n ) = p(z(r)n |X(r), !old),"(z(r)n!1, z
(r)n ) = p(z
(r)n!1, z
(r)n |X(r), !old).
![Page 98: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/98.jpg)
The forward-backward algorithm
Multiple shorter sequences
M-step Just add them all
etc.
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!k =
!Rr=1 "(z
(r)1k )!R
r=1!K
j=1 "(z(r)i j ),
Ajk =
!Rr=1!Nrn=2 #(z
(r)n!1, j, z
(r)nk )
!Rr=1!Kl=1!Nrn=2 #(z
(r)n!1, j, z
(r)nl )
![Page 99: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/99.jpg)
The forward-backward algorithm
Multiple shorter sequences
M-step Just add them all
etc.
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!k =
!Rr=1 "(z
(r)1k )!R
r=1!K
j=1 "(z(r)i j ),
Ajk =
!Rr=1!Nrn=2 #(z
(r)n!1, j, z
(r)nk )
!Rr=1!Kl=1!Nrn=2 #(z
(r)n!1, j, z
(r)nl )
![Page 100: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/100.jpg)
The forward-backward algorithm
Multiple shorter sequences
M-step Just add them all
etc.
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
!k =
!Rr=1 "(z
(r)1k )!R
r=1!K
j=1 "(z(r)i j ),
Ajk =
!Rr=1!Nrn=2 #(z
(r)n!1, j, z
(r)nk )
!Rr=1!Kl=1!Nrn=2 #(z
(r)n!1, j, z
(r)nl )
Note: p(X) no longer cancel out
![Page 101: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/101.jpg)
The forward-backward algorithm
Today's topics
The forward-backward algorithm 1. E-step of EM algorithm for HMM 2. Multiple shorter sequences 3. Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
![Page 102: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/102.jpg)
The forward-backward algorithm
Today's topics
The forward-backward algorithm 1. E-step of EM algorithm for HMM 2. Multiple shorter sequences 3. Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
![Page 103: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/103.jpg)
The forward-backward algorithm
Predictive distribution
Once you trained the HMM, next thing you want to do may be PREDICTION. I.e., evaluation of p(xN+1|X)
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
![Page 104: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/104.jpg)
The forward-backward algorithm
Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
p(xN+1|X) =!
zN+1
p(xN+1, zN+1|X)
=!
zN+1
p(xN+1|zN+1,X)p(zN+1|X)
=!
zN+1
p(xN+1|zN+1)p(zN+1|X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN , zN+1 |X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN ,X)p(zN |X)
...
![Page 105: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/105.jpg)
The forward-backward algorithm
Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
p(xN+1|X) =!
zN+1
p(xN+1, zN+1|X)
=!
zN+1
p(xN+1|zN+1,X)p(zN+1|X)
=!
zN+1
p(xN+1|zN+1)p(zN+1|X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN , zN+1 |X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN ,X)p(zN |X)
...
![Page 106: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/106.jpg)
The forward-backward algorithm
Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
p(xN+1|X) =!
zN+1
p(xN+1, zN+1|X)
=!
zN+1
p(xN+1|zN+1,X)p(zN+1|X)
=!
zN+1
p(xN+1|zN+1)p(zN+1|X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN , zN+1 |X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN ,X)p(zN |X)
...
![Page 107: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/107.jpg)
The forward-backward algorithm
Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
zN zN+1z1
xN xN+1x1
HeadTail
![Page 108: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/108.jpg)
The forward-backward algorithm
Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
p(xN+1|X) =!
zN+1
p(xN+1, zN+1|X)
=!
zN+1
p(xN+1|zN+1,X)p(zN+1|X)
=!
zN+1
p(xN+1|zN+1)p(zN+1|X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN , zN+1 |X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN ,X)p(zN |X)
...
![Page 109: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/109.jpg)
The forward-backward algorithm
Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
p(xN+1|X) =!
zN+1
p(xN+1, zN+1|X)
=!
zN+1
p(xN+1|zN+1,X)p(zN+1|X)
=!
zN+1
p(xN+1|zN+1)p(zN+1|X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN , zN+1 |X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN ,X)p(zN |X)
...
![Page 110: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/110.jpg)
The forward-backward algorithm
Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
p(xN+1|X) =!
zN+1
p(xN+1, zN+1|X)
=!
zN+1
p(xN+1|zN+1,X)p(zN+1|X)
=!
zN+1
p(xN+1|zN+1)p(zN+1|X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN , zN+1 |X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN ,X)p(zN |X)
...
![Page 111: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/111.jpg)
The forward-backward algorithm
Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
p(xN+1|X) =!
zN+1
p(xN+1, zN+1|X)
=!
zN+1
p(xN+1|zN+1,X)p(zN+1|X)
=!
zN+1
p(xN+1|zN+1)p(zN+1|X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN , zN+1 |X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN ,X)p(zN |X)
...
![Page 112: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/112.jpg)
The forward-backward algorithm
Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
...
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN ,X)p(zN |X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN)p(zN |X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN)p(zN ,X)p(X)
=1
p(X)!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN)!(zN).
![Page 113: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/113.jpg)
The forward-backward algorithm
Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
...
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN ,X)p(zN |X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN)p(zN |X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN)p(zN ,X)p(X)
=1
p(X)!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN)!(zN).
![Page 114: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/114.jpg)
The forward-backward algorithm
Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
zN zN+1z1
xN xN+1x1
Head
TailTail
![Page 115: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/115.jpg)
The forward-backward algorithm
Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
...
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN ,X)p(zN |X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN)p(zN |X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN)p(zN ,X)p(X)
=1
p(X)!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN)!(zN).
![Page 116: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/116.jpg)
The forward-backward algorithm
Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
...
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN ,X)p(zN |X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN)p(zN |X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN)p(zN ,X)p(X)
=1
p(X)!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN)!(zN).
![Page 117: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/117.jpg)
The forward-backward algorithm
Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
...
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN ,X)p(zN |X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN)p(zN |X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN)p(zN ,X)p(X)
=1
p(X)!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN)!(zN).
![Page 118: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/118.jpg)
The forward-backward algorithm
Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
...
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN ,X)p(zN |X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN)p(zN |X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN)p(zN ,X)p(X)
=1
p(X)!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN)!(zN).
![Page 119: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/119.jpg)
The forward-backward algorithm
Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
...
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN ,X)p(zN |X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN)p(zN |X)
=!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN)p(zN ,X)p(X)
=1
p(X)!
zN+1
p(xN+1|zN+1)!
zN
p(zN+1|zN)!(zN).
Only depend on α(zN) i.e., whole X don't need
![Page 120: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/120.jpg)
The forward-backward algorithm
Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
ξ(zn-1, zn) γ(zn)
β(zn) α(zn)
p(x1:N)
Nth
![Page 121: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/121.jpg)
The forward-backward algorithm
Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
ξ(zn-1, zn) γ(zn)
p(x1:N)
β(zn) α(zn) p(xN+1|X) Nth
Nth
![Page 122: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/122.jpg)
The forward-backward algorithm
Today's topics
The forward-backward algorithm 1. E-step of EM algorithm for HMM 2. Multiple shorter sequences 3. Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
![Page 123: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/123.jpg)
The forward-backward algorithm
Today's topics
The forward-backward algorithm 1. E-step of EM algorithm for HMM 2. Multiple shorter sequences 3. Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA
![Page 124: PRML 13.2.2: The Forward-Backward Algorithm](https://reader031.vdocuments.net/reader031/viewer/2022032620/55cae5b6bb61eb33788b47e3/html5/thumbnails/124.jpg)
The forward-backward algorithm
Today's topics
The forward-backward algorithm 1. E-step of EM algorithm for HMM 2. Multiple shorter sequences 3. Predictive distribution
Aug 4, 2014 PRML 13.2.2 Shinichi TAMURA