Sparse Reconstruction via the Reed-Muller Sieve

Sina JafarpourPrinceton UniversityGame Theory Meets Compressed Sensing

Based on joint work with:

VolkanCevherRobertCalderbankRobSchapire

Compressed SensingMain tasks:Design a sensing matrixDesign a reconstruction algorithmObjectives:Reconstruct a k-sparse signal from efficientlyThe algorithm should recover without knowing its supportSuccessful reconstruction from as few number of measurements as possible!Robustness against noise Model selection and sparse approximation

Model Selection and Sparse ApproximationMeasurement vector: : data-domain noise : measurement noiseModel Selection Goal:Given successfully recover the support of Sparse Approximation Goal: Given find a vector with

ApplicationsData streamingPacket routingSparsity in canonical basis

Biomedical imagingMRISparsity in Fourier basis

Digital communicationMultiuser DetectionSparsity in canonical basis

DestinationSource

Restricted Isometry PropertyDefinition: A matrix satisfies if for every -sparse vector :

Tractable Compressed Sensing: [CRT06/Don06] If is , then for the solution of

satisfies the guarantee

Basis Pursuit optimizationProblems with RIP Matricesiid Gaussian/Bernoulli matrices:No efficient algorithm for verifying RIP memory requirementInefficient computation (matrix-vector multiplication)Partial Fourier/Hadamard ensembles:No efficient algorithm for verifying RIPSub-optimal number of measurements:Algebraic structures:Sub-optimal number of measurements:

DeterministicCompressed SensingviaExpander GraphsExpander GraphsSensing matrix: Normalized adjacency matrix of an expander graph Existence: A random bipartite graph with is expander.

Explicit construction of expander graphs also exist [GUV08]. Expander-Based Compressed Sensing:Message Passing AlgorithmsTheorem [JXHC09] : If is a normalized expander then givena message passing algorithm after iterations recovers .Decoding in error-free case: Based on expander codes [Sipser and Spielman96][BIR09] (SSMP): A similar message passing algorithm with running time and guarantee Decoding when error exists: Pros: Decoding time is almost linear in Message-passing methods are easy to implement Cons: Poor practical performance Basis Pursuit Algorithm[BGIKS08]: Let be the normalized adjacency of a expander graph, then the solution of the basis pursuit optimization

Satisfies the guarantee

Pros: Better practical performance Cons: Decoding time is cubic in Sub-gradient methods are hard to implementGame Theory Meets Compressed SensingGoal: an efficient approximation algorithm for BPTo solve BP, it is sufficient to be able to efficiently solve

How? By doing binary search overPlan: approximately solve efficiently By reformulating it as a zero-sum gameApproximately solving the game-value

Recall BP Optimization:Game-Theoretic Reformulation of the Problem1: Consider the following 1-dimensional example2: Fix (say ) and observe that MaximumIntervalDiscrete set(2 elements) Bregman Divergence Examples: Euclidean distance: with Relative-entropy: with

GAME Algorithm:

Alice: Updates Bob: Updates Finds a withFinds an s.t.Greedy Alice:Conservative Alice:Gradient Projection Alice:Greedy Bob: Finds an s.t.Lemma [JCS11] is 1-sparse and can be computed in

Game-value Approximation GuaranteeGAME can be viewed as a repurposing of the Multiplicative Update Algorithm [Freund&Schapire99]

After iterations finds a -sparse vector with

Tradeoff between sparsity and approximation errorAs :

In expander-based CS we are only interested in the approximation error (Basis Pursuit Theorem) Empirical PerformanceMPBPEmpirical Convergence Rate

33Slope: -1Comparison with Nesterov OptimizationNesterov: A general iterative approximation algorithm for solving non-smooth optimization problemsNesterov requires iterations (similar to GAME)Each iteration requires solving 3 smooth optimization problemMuch more complicated than one adjoint operation (GAME) Nesterovs approximation guarantee:

GAMEs theoretical guarantee

Experimental ComparisonGAMERandom vs. Expander-Based Compressed SensingMatrixNumber ofMeasurementsEncodingTimeRecoveryTimeRecoveryGuaranteeExplicit ConstructionGaussianNoFourierNoExpander(MesPas)YesExpander(GAME)YesFairBestGoodOurContribution

Take-Home Message:Random vs. DeterministicGerhard Richter

The value of a random matrix:US $ 3,703,500

The value of a deterministic matrix:US $ 27,191,525

Piet Mondrian

Thank You!

M k " N

:M N

f =

f and

fand

p = O (edq + emq)

(!p/!q guarantee)

f = (+ ed) + em

ed RN

em RM

232

232

(1 )22 22 (1 + )22

2/1

(2k, 0.4) RIP

minimize 1

s.t. f2 em2

2 = Oed1

k+ em1

(k, ) RIP

k

O(MN)

M = O(k log5N)

M = O(k2)

N

N (S)

(1 !)d|S|

|S| k |N (S)| (1 )d|S|

M = Ok logN/k

2

M

d

S

|S| k

O(N logN/k)

1/1

1 = O(ed1 + eM1)

N

2k

(2k, 1/8)

f =

1/1

1 = O (ed1 + em1)

N

minimize1

s.t. f1 em1

(2k, 1/8)

s.t. f1 em1

minimize f1

s.t. 1

minimize1

= maxP{1,1}

P

x = 0

P = 1

P = 1

min|x|1

maxP[1,1]

P (2x 1) = min|x|1

maxP{1,1}

P (2x 1)

min|x|1

|2x 1| = min|x|1

max {2x 1, 1 2x}

maxP{1,1}M

P( f)

= min1

min1

f1 = min1

maxP[1,1]M

= min|x|1

maxP{1,1}

P (2x 1)

x

maxP[1,1]

P

R(P )

R(Q) + (P Q)!R(Q)

B(P,Q)

R(P ) = P2

B(P,Q) = P Q2

P

R(P ) =

i

P (i) lnP (i)

B(P,Q) =

i

Q(i)

i

P (i) +

i

P (i) lnQ(i)P (i)

B(P,Q) .= R(P )R(Q) (P Q)!R(Q)

Q

R

R(Q)

R(P0) = 0

P0

P0

0

0

Pt.= arg max

P[1,1]P(t1 f)

Pt.= arg min

P[1,1]MB(P, Pt1) P(t1 f)

R(Qt) = R(Pt1) + (t1 f)

Pt.= arg min

P[1,1]MB(P,Qt)

Pt

t

t.= arg min

1Pt (

f)

0.= arg min

1P0 (

f)

t

P

O(N logN/k)

t

Output :

.=1T

Ti=1

i

min1

maxP[1,1]M

P( f)

Objective :

T

f1 = min1 f1 +O

M

T

T

f1 min1 f1

T

104 103 102 101103

102

101

100

101

Measurement noi se std (m)

Rec

onstru

ctionerro

r( 1

/

1)

Reconstruction Error: N=5000, M=500, k=35

l1magicSSMPeGAME

proven theoretical convergence rate is O(T0.5)

empirical convergence rate is O(T1)

Measurement error: f1

Data error: 1

T = O(M)

f1 eM1 +O(M

T

)

f1 eM1 +O(

M

T

)

101 102 103102

101

100

101

Conv

erge

nce

rate

Number of iterations (T)

Convergence of BPP Algorithm: N=5000, M=500, k=35

BPP: MSMT ErrorBPP:Data ErrorNest: Data ErrorNest: MSMT Error

k log5N

N logN

N logN log R

!1/!1

BP

k log (N/k)

BP

k log(N/k)

k log(N/k)

N log(N/k)

N log(N/k)

N log(N/k)

kN log2(N/k)R2

R = SNR .=2e2

kN log (N/k)

kN log (N/k) log R

!2/!1