the variational em algorithm for on-line identification of extended
TRANSCRIPT
![Page 1: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/1.jpg)
The Variational EM Algorithm for On-lineIdentification of Extended AR Models
Václav Šmídl Anthony Quinn
UTIA, Czech Academy of Science Trinity College Dublin, Ireland
March 22, ICASSP 05
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 1 / 1
![Page 2: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/2.jpg)
Outline
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 2 / 1
![Page 3: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/3.jpg)
Univariate AR model
AR model:
dt = −a1dt−1 − a2dt−2 − . . .− aqdt−q + σet , et ∼ N (0, 1)
with
AR parameters a = [a1, . . . , aq]′, and σ ,
regressor x t = [dt−1, . . . , dt−q]′,
history Dt = [d1, d2, . . . , dt ] ,
Observation model:
f (dt |a, σ, Dt−1) = f (dt |a, σ, x t) = N(−a′x t , σ
2) .
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 3 / 1
![Page 4: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/4.jpg)
Recursive Bayesian Inference
Conjugacy:
f (a, σ|Dt) ∝ f (dt |a, σ, x t) f (a, σ|Dt−1) ,
same functional form
f (a, σ|Dt) = f (a, σ|s (Dt))
st = s (Dt) is time-invariant function generating sufficient statistics.
BTf (a, σ|Dt−1) f (a, σ|Dt )
f (dt |a, σ, Dt−1)
st−1 st
dt
update
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 4 / 1
![Page 5: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/5.jpg)
Recursive Bayesian Inference
Conjugacy:
f (a, σ|Dt) ∝ f (dt |a, σ, x t) f (a, σ|Dt−1) ,
same functional form
f (a, σ|Dt) = f (a, σ|s (Dt))
st = s (Dt) is time-invariant function generating sufficient statistics.
BTf (a, σ|Dt−1) f (a, σ|Dt )
f (dt |a, σ, Dt−1)
st−1 st
dt
update
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 4 / 1
![Page 6: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/6.jpg)
Recursive Bayesian Inference
Conjugacy:
f (a, σ|Dt) ∝ f (dt |a, σ, x t) f (a, σ|Dt−1) ,
same functional form
f (a, σ|Dt) = f (a, σ|s (Dt))
st = s (Dt) is time-invariant function generating sufficient statistics.
BTf (a, σ|Dt−1) f (a, σ|Dt )
f (dt |a, σ, Dt−1)
st−1 st
dt
update
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 4 / 1
![Page 7: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/7.jpg)
Recursive Bayesian Inference of AR parameters
Observation model for AR:
f (dt |a, σ, x t) = N(−a′x t , σ
2) ,
Conjugate distribution is Normal-inverse-Gamma, N iG:
f (a, σ|Dt) = N iG (Vt , νt)
sufficient statistics, st = {Vt , νt}:
Vt = Vt−1 +
[dt
x t
] [dt x t
]=
[vd,d ;t v ′
x ,d ;tv x ,d ;t Vx ,x ;t
],
νt = νt−1 + 1,
Moments:Ef (a|Dt ) [a] = V−1
x ,x ;tv x ,d ;t .
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 5 / 1
![Page 8: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/8.jpg)
Extending AR model
AR Extended AR (EAR)
etσ
a2
a1
aq
z−1
z−2
z−q
x2;t
x1;t
xq;t
dtdtet
σg−1
0
g1
g2
gq
a1
a2
aq
yt
x1;t
x2;t
xq;t
z−1
a g dtetσ yt
x t
Posterior:
f (a, σ|Dt) = N iG (Vt , νt) ,Vt = Vt−1 + z tz ′
tνt = νt−1 + 1
z t = [dt , dt−1, dt−2, . . . , dt−q]′ z t = g ′ (Dt) =[g0 (Dt) , g1 (Dt−1) , . . . , gq (Dt−1)]
′
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 6 / 1
![Page 9: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/9.jpg)
Mixture-based Extension of AR (MEAR)
Relaxing the assumption of known time-variant transformation.Treating g as time-variant discrete random variable:
g → g t ∈ {g1, g2, . . . , g c} ⇔ l t ∈ {l 1, l 2, . . . , l c}
The observation model is
f (dt |a, σ, l t , Dt−1) =∏c
i=1 f (dt |a, σ, x i;t , g i)li;t ,
f (li;t) = αi ,∑c
i=1αi = 1, i = 1, . . . , c,
f (dt |a, σ, Dt−1) =∑c
i=1 αi f (dt |a, σ, x i;t , g i),
EAR MEAR
a g dtetσ yt
x t
g2AR
g c
etσ
dt
α
g1
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 7 / 1
![Page 10: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/10.jpg)
Conjugacy Lost and Conjugacy Regained
Bayes’ rule:
f (a, σ, l t |Dt) ∝ f (dt |a, σ, l t , Dt−1) f (l t) f (a, σ|Dt−1) ,
Conditional independence approximation:
f (a, σ, l t |Dt) ≈ f (a, σ, l t |Dt) = f (a, σ|Dt) f (l t |Dt) ,
Then:
f (a, σ|Dt) f (l t |Dt) ∝ f (dt |a, σ, l t , Dt−1) f (l t) f (a, σ|Dt−1) .
BTf (a, σ|Dt−1)
Pf (dt |a, σ, l t , Dt−1)
f (a, σ|Dt )Pf (a, σ, l t |Dt )
approx.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 8 / 1
![Page 11: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/11.jpg)
Restoring conjugacy in mixture-type models
Traditionally, conditional independence was achieved heuristically:
f (a, σ, l t |Dt) ≈ f (a, σ|Dt) f (l t |Dt) ,
f (l t |Dt) =∫
a,σf (a, σ, l t |Dt) dadσ,
f (a, σ|Dt) = f(
a, σ|Dt , l t
).
Quasi-Bayes: l t = Ef (l t |Dt )[l t ] ,
Viterbi-like: l t = arg maxl t f (l t |Dt) .
We favour free-form functional optimization.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 9 / 1
![Page 12: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/12.jpg)
The Variational Bayes (VB) method
Two ingredients:
1 Conditional independence assumption:
f (a, σ, l t |Dt) = f1 (a, σ|Dt) f2 (l t |Dt) ,
2 Minimization of the KL divergence,
f1 (·) , f2 (·) = arg minf1 ,f2
KL(
f (a, σ|Dt) f (l t |Dt) ||f (a, σ, l t |Dt))
,
Necessary condition for minimum:
f1 (a, σ|Dt) ∝ exp(
Ef2(l t |Dt )[ln (f (a, σ, l t , Dt))]
),
f2 (l t |Dt) ∝ exp(
Ef1(a,σ|Dt )[ln (f (a, σ, l t , Dt))]
).
Solved iteratively via the Variational EM (VEM) algorithm.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 10 / 1
![Page 13: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/13.jpg)
The Variational Bayes (VB) method
Two ingredients:
1 Conditional independence assumption:
f (a, σ, l t |Dt) = f1 (a, σ|Dt) f2 (l t |Dt) ,
2 Minimization of the KL divergence,
f1 (·) , f2 (·) = arg minf1 ,f2
KL(
f (a, σ|Dt) f (l t |Dt) ||f (a, σ, l t |Dt))
,
Necessary condition for minimum:
f1 (a, σ|Dt) ∝ exp(
Ef2(l t |Dt )[ln (f (a, σ, l t , Dt))]
),
f2 (l t |Dt) ∝ exp(
Ef1(a,σ|Dt )[ln (f (a, σ, l t , Dt))]
).
Solved iteratively via the Variational EM (VEM) algorithm.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 10 / 1
![Page 14: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/14.jpg)
The Variational Bayes (VB) method
Two ingredients:
1 Conditional independence assumption:
f (a, σ, l t |Dt) = f1 (a, σ|Dt) f2 (l t |Dt) ,
2 Minimization of the KL divergence,
f1 (·) , f2 (·) = arg minf1 ,f2
KL(
f (a, σ|Dt) f (l t |Dt) ||f (a, σ, l t |Dt))
,
Necessary condition for minimum:
f1 (a, σ|Dt) ∝ exp(
Ef2(l t |Dt )[ln (f (a, σ, l t , Dt))]
),
f2 (l t |Dt) ∝ exp(
Ef1(a,σ|Dt )[ln (f (a, σ, l t , Dt))]
).
Solved iteratively via the Variational EM (VEM) algorithm.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 10 / 1
![Page 15: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/15.jpg)
VB-observation model
f1 (a, σ|Dt) ∝ exp(
Ef (l t |Dt )
[ln
(f (dt |a, σ, l t) f (l t) f (a, σ|Dt−1)
)]),
f1 (a, σ|Dt) ∝ red}f (dt |a, σ)f (a, σ|Dt−1)
VB the generates a partial VB-observation model for each variable.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 11 / 1
![Page 16: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/16.jpg)
VB-observation model
f1 (a, σ|Dt) ∝ exp(
Ef (l t |Dt )
[ln
(f (dt |a, σ, l t) f (l t) f (a, σ|Dt−1)
)]),
f1 (a, σ|Dt) ∝ red}f (dt |a, σ)f (a, σ|Dt−1)
VB the generates a partial VB-observation model for each variable.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 11 / 1
![Page 17: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/17.jpg)
VB-observation model
f1 (a, σ|Dt) ∝ exp(
Ef (l t |Dt )
[ln
(f (dt |a, σ, l t) f (l t) f (a, σ|Dt−1)
)]),
f1 (a, σ|Dt) ∝ red}f (dt |a, σ)f (a, σ|Dt−1)
VB the generates a partial VB-observation model for each variable.
BTf (a, σ|Dt−1) f (a, σ|Dt )
f1(dt |a, σ, Dt−1)
BTf (l t |Dt−1) f (l t |Dt )
f2(dt |l t , Dt−1)
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 11 / 1
![Page 18: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/18.jpg)
VB-observation model
f1 (a, σ|Dt) ∝ exp(
Ef (l t |Dt )
[ln
(f (dt |a, σ, l t) f (l t) f (a, σ|Dt−1)
)]),
f1 (a, σ|Dt) ∝ red}f (dt |a, σ)f (a, σ|Dt−1)
VB the generates a partial VB-observation model for each variable.
BTf (a, σ|Dt−1) f (a, σ|Dt )
f1(dt |a, σ, Dt−1)
BTf (l t |Dt−1) f (l t |Dt )
f2(dt |l t , Dt−1)
Observation models interact with posteriors via moments.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 11 / 1
![Page 19: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/19.jpg)
VB for MEAR
VB-posterior distributions:
f (a, σ|Dt) = N iG (Vt , νt) ,
f (lt |Dt) = Mu (w t) ,
with statistics expressed implicitly:
Vt = Vt−1 +c∑
i=1
wi;tz i;tz ′i;t
νt = νt−1 + 1,
wi;t ∝ |Ji;t |exp[− 1
2σ z ′j;t
[−1, a′
]′ [−1, a′
]z j;t (1)
− 12 z ′
j;tV−1x ,x ;tz j;t
],
Solved iteratively.Note: Statistics Vt is updated by rank-c structure.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 12 / 1
![Page 20: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/20.jpg)
VB for MEAR
VB-posterior distributions:
f (a, σ|Dt) = N iG (Vt , νt) ,
f (lt |Dt) = Mu (w t) ,
with statistics expressed implicitly:
Vt = Vt−1 +c∑
i=1
wi;tz i;tz ′i;t
νt = νt−1 + 1,
wi;t ∝ |Ji;t |exp[− 1
2σ z ′j;t
[−1, a′
]′ [−1, a′
]z j;t (1)
− 12 z ′
j;tV−1x ,x ;tz j;t
],
Solved iteratively.Note: Statistics Vt is updated by rank-c structure.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 12 / 1
![Page 21: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/21.jpg)
VB for MEAR
VB-posterior distributions:
f (a, σ|Dt) = N iG (Vt , νt) ,
f (lt |Dt) = Mu (w t) ,
with statistics expressed implicitly:
Vt = Vt−1 +c∑
i=1
wi;tz i;tz ′i;t
νt = νt−1 + 1,
wi;t ∝ |Ji;t |exp[− 1
2σ z ′j;t
[−1, a′
]′ [−1, a′
]z j;t (1)
− 12 z ′
j;tV−1x ,x ;tz j;t
],
Solved iteratively.Note: Statistics Vt is updated by rank-c structure.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 12 / 1
![Page 22: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/22.jpg)
MEAR with time-variant weights
Consider Markov model of label evolution:
f (lt |lt−1, Q) =∏c
i,j=1 q li;t lj;t−1
i,j ,
with unknown transition matrix Q.Conditional independence is imposed between the weights:
f (a, σ, lt , lt−1|Dt) ≈ f (a, σ|Dt) f (lt |Dt) f (lt−1|Dt) ,
Posterior distributions are again Multinomial on both labels, lt , lt−1,and Dirichlet on Q.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 13 / 1
![Page 23: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/23.jpg)
MEAR with time-variant parameters
Since conjugacy was restored using VB, the technique of forgettingcan be used for dealing with time-variant parameters.
f (at , σt |Dt−1) ∝[f (at−1, σt−1|Dt−1)
]φ
at−1 → at
σt−1 → σt
[f (at , σt)
]1−φ
In combination with time-variant weights we use two approximationsof time-update step of stochastic filtering: (i) VB for weights, (ii)forgetting for a and σ.
BT f (at , σt |Dt )
f1(dt |a, σ, Dt−1)
φ
f (at , σt )
f (at−1, σt−1|Dt−1)
f (at , σt |Dt−1)
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 14 / 1
![Page 24: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/24.jpg)
Outlier corrupted AR
Outlier:
yt = −ax t + σet ,
dt = yt + ξt , ξt =
{het
0
Transformations:g1 : z t = [dt , . . . , dt−q]′,g2 :z t = 1
h [dt , . . . , dt−q]′,g i : i = 3 . . . qz t = [dt , . . . , yt−i , . . . dt−q]
′,
VB result
55 60 65 700
1
time (t )
w4,
t
0
1
w3,
t
0
1
w2,
t
0
1
w1,
t
component weights (first outlier)
60 70 80 90− 2
− 1
0
1
time (t )
Signals and reconstructions
QB result
55 60 65 700
1
time (t )
w4,
t
0
1
w3,
t
0
1
w2,
t
0
1
w1,
t
60 70 80 90− 2
− 1
0
1
time (t )
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 15 / 1
![Page 25: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/25.jpg)
Burst noise corrupted AR
Burst noise:
yt = −ax t + σet ,
dt = yt + ξt , ξt ∼ N (0, h) ,
with unknown h.Transformations:g1 : z t = [dt , . . . , dt−q]′,g i : z t = 1
κi[dt , x i;t ],
where κi and x i;t are from Kalmanfilter.Experiment was performed withforgetting factor φ = 0.95.
1100 1200 1300 1400 1500
−0.4
−0.2
0
0.2
Unc
orru
pted
Speach segment 1
1100 1200 1300 1400 1500
−0.4
−0.2
0
0.2
Cor
rupt
ed
1100 1200 1300 1400 1500
−0.4
−0.2
0
0.2
VB
reco
nstru
ctio
n1100 1200 1300 1400 1500
−0.4
−0.2
0
0.2
QB
reco
nstru
ctio
n
1100 1200 1300 1400 1500
−0.4
−0.2
0
0.2
time (t)
VL
reco
nstru
ctio
n
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 16 / 1
![Page 26: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/26.jpg)
Conclusion
Mixture-based extension of the AR for dealing with unknowntransformations was presented.
The Variational Bayes technique was used to achieve conjugateupdate of parameter statistics
Applied to reconstruction of speech corrupted by burst noise.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 17 / 1
![Page 27: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/27.jpg)
Spanning
The chosen transformation (filter-bank) forms nodes of a simplex ofpossible dyads. Update by a linear combination of these dyads allowsus to span interior of the simplex.
desired behaviour undesired behaviourg2g1
g3
g2g1
g3
This should be remembered when designing a filter-bank.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 18 / 1
![Page 28: The Variational EM Algorithm for On-line Identification of Extended](https://reader035.vdocuments.net/reader035/viewer/2022071602/613d6842736caf36b75cf7af/html5/thumbnails/28.jpg)
Computational flow
w tO.P.
w t−1
11−z−1
11−z−1
z−1
g2
z2;tO.P.
z2;t z′2;t
g c
zc;tO.P.
zc;t z′c;t
......
g1
z1;tO.P.
z1;t z′1;t
dt
w2;tEq. (1)
um
x
...
wc;t
...
w1;tEq. (1)
Eq. (1)
Φt
Vt
Reminiscent of multiple model approach. In our approach, however,we propagate statistics.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 19 / 1