e cient message passing-based inference in the multiple measurement vector...
TRANSCRIPT
![Page 1: E cient Message Passing-Based Inference in the Multiple Measurement Vector Problemschniter/pdf/asil11_mmv_slides.pdf · 2011-11-10 · E cient Message Passing-Based Inference in the](https://reader033.vdocuments.net/reader033/viewer/2022042013/5e731b5b3edc151a317cbecc/html5/thumbnails/1.jpg)
E�cient Message Passing-Based Inference in the MultipleMeasurement Vector Problem
Justin Ziniel Philip Schniter
Department of Electrical and Computer EngineeringThe Ohio State University
Asilomar Conference on Signals, Systems, and Computers, 2011
Work supported in part by NSF grant CCF-1018368 and DARPA/ONR grant N66001-10-1-4090
1 of 18
![Page 2: E cient Message Passing-Based Inference in the Multiple Measurement Vector Problemschniter/pdf/asil11_mmv_slides.pdf · 2011-11-10 · E cient Message Passing-Based Inference in the](https://reader033.vdocuments.net/reader033/viewer/2022042013/5e731b5b3edc151a317cbecc/html5/thumbnails/2.jpg)
Outline
BackgroundThe Multiple Measurement Vector (MMV) ProblemExisting ApproachesSignal Model
Our Proposed MethodBelief Propagation-Based InferenceEM Parameter Learning
Empirical Study
Conclusion
2 of 18
![Page 3: E cient Message Passing-Based Inference in the Multiple Measurement Vector Problemschniter/pdf/asil11_mmv_slides.pdf · 2011-11-10 · E cient Message Passing-Based Inference in the](https://reader033.vdocuments.net/reader033/viewer/2022042013/5e731b5b3edc151a317cbecc/html5/thumbnails/3.jpg)
The Multiple Measurement Vector (MMV) Problem
Consider a time-series of sparse, temporally correlated signal vectors thatshare a common support...
x(1)
x(2)
x(3)
x(4)
x(5)
3 of 18
![Page 4: E cient Message Passing-Based Inference in the Multiple Measurement Vector Problemschniter/pdf/asil11_mmv_slides.pdf · 2011-11-10 · E cient Message Passing-Based Inference in the](https://reader033.vdocuments.net/reader033/viewer/2022042013/5e731b5b3edc151a317cbecc/html5/thumbnails/4.jpg)
The Multiple Measurement Vector (MMV) Problem
...observed through a noisy linear measurement process, Y = AX + E.
= +
Y A X E
Applications: Magnetoencephalogaphy, direction-of-arrival estimation, parallel MRI,...
4 of 18
![Page 5: E cient Message Passing-Based Inference in the Multiple Measurement Vector Problemschniter/pdf/asil11_mmv_slides.pdf · 2011-11-10 · E cient Message Passing-Based Inference in the](https://reader033.vdocuments.net/reader033/viewer/2022042013/5e731b5b3edc151a317cbecc/html5/thumbnails/5.jpg)
Existing methods
� Greedy pursuit� M-BMP, M-OMP, M-ORMP [Cotter et al., '05]� S-OMP [Tropp et al., '06]� Subspace-augmented MUSIC* [Lee et al., '10]
� Mixed-norm (`1/`2) minimization� M-FOCUSS [Cotter et al., '05]� RX-penalty, RX-error [Tropp et al., '06]� JLZA [Hyder and Mahata, '10]� tMFOCUSS* [Zhang and Rao, '11a]
� Bayesian MMV� M-SBL [Wipf and Rao, '07]� JSSR-MP [Shedthikere and Chockalingam, '11]� T-MSBL*, T-SBL* [Zhang and Rao, '11b]
� Block-sparse single measurement vector� [Eldar and Mishali, '09]� bSBL [Zhang and Rao, '11b]
* = Accounts for temporal correlation in amplitudes5 of 18
![Page 6: E cient Message Passing-Based Inference in the Multiple Measurement Vector Problemschniter/pdf/asil11_mmv_slides.pdf · 2011-11-10 · E cient Message Passing-Based Inference in the](https://reader033.vdocuments.net/reader033/viewer/2022042013/5e731b5b3edc151a317cbecc/html5/thumbnails/6.jpg)
Comparing Di�erent Approaches
Approach Speed Performance
Greedy Fast Fair
Mixed-norm Okay Good
Bayesian Slow Great
Why Bayesian?
� Modeling assumptions are made explicit
� Model parameters have meaningful interpretations
� Principled parameter learning
� Soft inference
6 of 18
![Page 7: E cient Message Passing-Based Inference in the Multiple Measurement Vector Problemschniter/pdf/asil11_mmv_slides.pdf · 2011-11-10 · E cient Message Passing-Based Inference in the](https://reader033.vdocuments.net/reader033/viewer/2022042013/5e731b5b3edc151a317cbecc/html5/thumbnails/7.jpg)
Comparing Di�erent Approaches
Approach Speed Performance
Greedy Fast Fair
Mixed-norm Okay Good
Bayesian Slow Great
Why Bayesian?
� Modeling assumptions are made explicit
� Model parameters have meaningful interpretations
� Principled parameter learning
� Soft inference
6 of 18
![Page 8: E cient Message Passing-Based Inference in the Multiple Measurement Vector Problemschniter/pdf/asil11_mmv_slides.pdf · 2011-11-10 · E cient Message Passing-Based Inference in the](https://reader033.vdocuments.net/reader033/viewer/2022042013/5e731b5b3edc151a317cbecc/html5/thumbnails/8.jpg)
A Model of Sparse Time-Evolving Signals
We write: x(t)n = s(t)n · θ(t)n for s(t)n ∈ {0,1} and θ(t)n ∼ CN (ζ,σ2).
Xs Θ
⊙ =
Amplitude Evolution
Treat {θ(t)n }Tt=1 as a Gauss-Markov
process: θ(t)n = (1− α)θ
(t−1)n + αw(t)
n ,
where w(t)n ∼ CN (0,ρ), and α conrols
the correlation in the random process.
7 of 18
![Page 9: E cient Message Passing-Based Inference in the Multiple Measurement Vector Problemschniter/pdf/asil11_mmv_slides.pdf · 2011-11-10 · E cient Message Passing-Based Inference in the](https://reader033.vdocuments.net/reader033/viewer/2022042013/5e731b5b3edc151a317cbecc/html5/thumbnails/9.jpg)
The Factor Graph Representation
AMP
t
8 of 18
![Page 10: E cient Message Passing-Based Inference in the Multiple Measurement Vector Problemschniter/pdf/asil11_mmv_slides.pdf · 2011-11-10 · E cient Message Passing-Based Inference in the](https://reader033.vdocuments.net/reader033/viewer/2022042013/5e731b5b3edc151a317cbecc/html5/thumbnails/10.jpg)
The Factor Graph Representation: Single Timestep
Signal
Mea
sure
men
ts
8 of 18
![Page 11: E cient Message Passing-Based Inference in the Multiple Measurement Vector Problemschniter/pdf/asil11_mmv_slides.pdf · 2011-11-10 · E cient Message Passing-Based Inference in the](https://reader033.vdocuments.net/reader033/viewer/2022042013/5e731b5b3edc151a317cbecc/html5/thumbnails/11.jpg)
The Factor Graph Representation: Support Variables
AMP
t
8 of 18
![Page 12: E cient Message Passing-Based Inference in the Multiple Measurement Vector Problemschniter/pdf/asil11_mmv_slides.pdf · 2011-11-10 · E cient Message Passing-Based Inference in the](https://reader033.vdocuments.net/reader033/viewer/2022042013/5e731b5b3edc151a317cbecc/html5/thumbnails/12.jpg)
The Factor Graph Representation: Amplitude Variables
AMP
t
8 of 18
![Page 13: E cient Message Passing-Based Inference in the Multiple Measurement Vector Problemschniter/pdf/asil11_mmv_slides.pdf · 2011-11-10 · E cient Message Passing-Based Inference in the](https://reader033.vdocuments.net/reader033/viewer/2022042013/5e731b5b3edc151a317cbecc/html5/thumbnails/13.jpg)
Approximate Message Passing (AMP)M
easu
rem
ents
Signal
AMP
� Standard belief propagation is intractable here
� Simpli�cation: Approximate message passing(AMP), [Donoho, Maleki, and Montanari, '09, '10]
� Marginal for x(t)n : Bernoulli-Gaussian -
(1− π(t)n )δ(x(t)n ) + π
(t)n CN (x(t)n ;ξ(t)n ,ψ(t)
n )
� As M,N→∞, AMP behavior described precisely bystate evolution → MMSE-optimal estimates [Bayatiand Montanari, '10]
# of messages exchanged: O(N)Complexity per iteration: O(MN) (matrix-vectorproduct)
9 of 18
![Page 14: E cient Message Passing-Based Inference in the Multiple Measurement Vector Problemschniter/pdf/asil11_mmv_slides.pdf · 2011-11-10 · E cient Message Passing-Based Inference in the](https://reader033.vdocuments.net/reader033/viewer/2022042013/5e731b5b3edc151a317cbecc/html5/thumbnails/14.jpg)
Parameter Learning via Expectation-Maximization
� Signal model governed by a number of parameters: Γ, {λ,ζ,σ2,α,ρ,σ2e }
� Parameters can be tuned automatically from the data using anexpectation-maximization (EM) algorithm
AMP-MMV EM Learning
{s,Θ}i
Γi+1
� Finds local maximizer of p(Y|Γ)� EM parameter estimation �ts naturally into the existing message passingprocedure� The E-step of the EM algorithm makes use of quantities available for free asa byproduct of AMP-MMV!
10 of 18
![Page 15: E cient Message Passing-Based Inference in the Multiple Measurement Vector Problemschniter/pdf/asil11_mmv_slides.pdf · 2011-11-10 · E cient Message Passing-Based Inference in the](https://reader033.vdocuments.net/reader033/viewer/2022042013/5e731b5b3edc151a317cbecc/html5/thumbnails/15.jpg)
Empirical Study: Setup
� AMP-MMV w/ EM parameter learning was compared against 3 powerfulMMV algorithms, and an oracle-aided MMSE bound (support-awareKalman smoother)� Bayesian: MSBL and T-MSBL* [Zhang and Rao, '11b]� Greedy: Subspace-augmented MUSIC (SA-MUSIC*) [Lee et al., '10]
� Signals generated according to signal model; i.i.d. Gaussian A matrices;AWGN corrupting noise
* = Accounts for temporal correlation in amplitudes
11 of 18
![Page 16: E cient Message Passing-Based Inference in the Multiple Measurement Vector Problemschniter/pdf/asil11_mmv_slides.pdf · 2011-11-10 · E cient Message Passing-Based Inference in the](https://reader033.vdocuments.net/reader033/viewer/2022042013/5e731b5b3edc151a317cbecc/html5/thumbnails/16.jpg)
Empirical Study: MSE vs. Normalized Sparsity Rate
1.5 2 2.5 3−30
−25
−20
−15
−10
−5
0
α = 0.1 | N = 5000, M = 1563, T = 4, SNR = 25 dB
Measurements−to−Active−Coefficients (M/K)
Tim
este
p−
Avera
ged N
orm
aliz
ed M
SE
(T
NM
SE
) [d
B]
T−MSBL
MSBL
SA−MUSIC
AMP−MMV
Oracle
1.5 2 2.5 310
0
101
102
103
104
105
α = 0.1 | N = 5000, M = 1563, T = 4, SNR = 25 dB
Measurements−to−Active−Coefficients (M/K)
Runtim
e [s]
T−MSBL
MSBL
SA−MUSIC
AMP−MMV
Correlation: 1− α = 0.9012 of 18
![Page 17: E cient Message Passing-Based Inference in the Multiple Measurement Vector Problemschniter/pdf/asil11_mmv_slides.pdf · 2011-11-10 · E cient Message Passing-Based Inference in the](https://reader033.vdocuments.net/reader033/viewer/2022042013/5e731b5b3edc151a317cbecc/html5/thumbnails/17.jpg)
Empirical Study: NSER vs. Normalized Sparsity Rate
1.5 2 2.5 310
−3
10−2
10−1
100
α = 0.1 | N = 5000, M = 1563, T = 4, SNR = 25 dB
Measurements−to−Active−Coefficients (M/K)
Nor
mal
ized
Sup
port
Err
or R
ate
(NS
ER
)
T−MSBL
MSBL
SA−MUSIC
AMP−MMVAMP−MMV [p(s
n| y)]
1.5 2 2.5 310
0
101
102
103
104
105
α = 0.1 | N = 5000, M = 1563, T = 4, SNR = 25 dB
Measurements−to−Active−Coefficients (M/K)
Run
time
[s]
T−MSBL
MSBL
SA−MUSIC
AMP−MMV
Correlation: 1− α = 0.9013 of 18
![Page 18: E cient Message Passing-Based Inference in the Multiple Measurement Vector Problemschniter/pdf/asil11_mmv_slides.pdf · 2011-11-10 · E cient Message Passing-Based Inference in the](https://reader033.vdocuments.net/reader033/viewer/2022042013/5e731b5b3edc151a317cbecc/html5/thumbnails/18.jpg)
Empirical Study: MSE vs. Normalized Sparsity Rate
1.5 2 2.5 3−30
−25
−20
−15
−10
−5
0
5
α = 0.01 | N = 5000, M = 1563, T = 4, SNR = 25 dB
Measurements−to−Active−Coefficients (M/K)
Tim
este
p−
Avera
ged N
orm
aliz
ed M
SE
(T
NM
SE
) [d
B]
T−MSBL
MSBL
SA−MUSIC
AMP−MMV
Oracle
1.5 2 2.5 310
0
101
102
103
104
105α = 0.01 | N = 5000, M = 1563, T = 4, SNR = 25 dB
Measurements−to−Active−Coefficients (M/K)
Runtim
e [s]
T−MSBL
MSBL
SA−MUSIC
AMP−MMV
Correlation: 1− α = 0.9914 of 18
![Page 19: E cient Message Passing-Based Inference in the Multiple Measurement Vector Problemschniter/pdf/asil11_mmv_slides.pdf · 2011-11-10 · E cient Message Passing-Based Inference in the](https://reader033.vdocuments.net/reader033/viewer/2022042013/5e731b5b3edc151a317cbecc/html5/thumbnails/19.jpg)
Empirical Study: MSE vs. Signal Dimension
102
103
104
−30
−25
−20
−15
−10
−5
α = 0.05 | T = 4, N/M = 3, λ = 0.15, SNR = 25 dB
Signal Dimension (N)
Tim
este
p−
Avera
ged N
orm
aliz
ed M
SE
(T
NM
SE
) [d
B]
T−MSBL
MSBL
SA−MUSIC
AMP−MMV
Oracle
102
103
104
10−2
10−1
100
101
102
103
104
α = 0.05 | T = 4, N/M = 3, λ = 0.15, SNR = 25 dB
Signal Dimension (N)
Runtim
e [s]
T−MSBL
MSBL
SA−MUSIC
AMP−MMV
Correlation: 1− α = 0.9515 of 18
![Page 20: E cient Message Passing-Based Inference in the Multiple Measurement Vector Problemschniter/pdf/asil11_mmv_slides.pdf · 2011-11-10 · E cient Message Passing-Based Inference in the](https://reader033.vdocuments.net/reader033/viewer/2022042013/5e731b5b3edc151a317cbecc/html5/thumbnails/20.jpg)
Empirical Study: MSE vs. Measurement Innovation
10−2
10−1
100
−30
−25
−20
−15
−10
−5
0
5
α = 0.01 | N = 5000, M = 167, T = 4, λ = 0.017, SNR = 25 dB
Innovation Rate, β
Tim
este
p−
Avera
ged N
orm
aliz
ed M
SE
(T
NM
SE
) [d
B]
AMP−MMV
Oracle
10−2
10−1
100
100
101
102
α = 0.01 | N = 5000, M = 167, T = 4, λ = 0.017, SNR = 25 dB
Innovation Rate, β
Runtim
e [s]
AMP−MMV
Time-varying measurement matrix: A(t) = (1− β)A(t−1) + βW(t)
Correlation: 1− α = 0.99 | Undersampling Rate (N/M): 30 | Normalized Sparsity (M/K): 216 of 18
![Page 21: E cient Message Passing-Based Inference in the Multiple Measurement Vector Problemschniter/pdf/asil11_mmv_slides.pdf · 2011-11-10 · E cient Message Passing-Based Inference in the](https://reader033.vdocuments.net/reader033/viewer/2022042013/5e731b5b3edc151a317cbecc/html5/thumbnails/21.jpg)
Conclusion
� AMP-MMV� Works with temporally correlated signal amplitudes� Performance rivals an oracle-aided MMSE bound (support aware Kalmansmoother) over a wide range of problems
� Computational complexity scales linearly in all problem dimensions
� EM parameter learning� Principled method of learning signal model parameters� Closed-form updates using outputs of AMP-MMV
� Empirical study� Two orders-of-magnitude improvement in runtime� Major gains possible from matrix diversity
17 of 18
![Page 22: E cient Message Passing-Based Inference in the Multiple Measurement Vector Problemschniter/pdf/asil11_mmv_slides.pdf · 2011-11-10 · E cient Message Passing-Based Inference in the](https://reader033.vdocuments.net/reader033/viewer/2022042013/5e731b5b3edc151a317cbecc/html5/thumbnails/22.jpg)
Empirical Study: MSE vs. Undersampling Rate
0 5 10 15 20 25−30
−28
−26
−24
−22
−20
−18
−16
−14
−12
−10
α = 0.25 | N = 5000, T = 4, M/K = 3 SNR = 25 dB
Unknowns−to−Measurements Ratio (N/M)
Tim
este
p−
Avera
ged N
orm
aliz
ed M
SE
(T
NM
SE
) [d
B]
T−MSBL
MSBL
SA−MUSIC
AMP−MMV
Oracle
0 5 10 15 20 2510
−1
100
101
102
103
α = 0.25 | N = 5000, T = 4, M/K = 3 SNR = 25 dB
Unknowns−to−Measurements Ratio (N/M)
Runtim
e [s]
T−MSBL
MSBL
SA−MUSIC
AMP−MMV
Correlation: 1− α = 0.7518 of 18