submodular dictionary selection for sparse representation (pdf, 2

Submodular Dictionary Selection for Sparse Representation

Andreas Krause [email protected]

California Institute of Technology, Computing and Mathematical Sciences Department

Volkan Cevher volkan.cevher@epfl,idiap.ch

Ecole Polytechnique Federale de Lausanne, STI-IEL-LIONS & Idiap Research Institute

Abstract

We develop an efficient learning framework toconstruct signal dictionaries for sparse represen-tation by selecting the dictionary columns frommultiple candidate bases. By sparse, we meanthat only a few dictionary elements, compared tothe ambient signal dimension, can exactly repre-sent or well-approximate the signals of interest.We formulate both the selection of the dictionarycolumns and the sparse representation of signalsas a joint combinatorial optimization problem.The proposed combinatorial objective maximizesvariance reduction over the set of training signalsby constraining the size of the dictionary as wellas the number of dictionary columns that can beused to represent each signal. We show that ifthe available dictionary column vectors are inco-herent, our objective function satisfies approxi-mate submodularity. We exploit this property todevelop SDSOMP and SDSMA, two greedy algo-rithms with approximation guarantees. We alsodescribe how our learning framework enables dic-tionary selection for structured sparse represen-tations, e.g., where the sparse coefficients occurin restricted patterns. We evaluate our approachon synthetic signals and natural images for rep-resentation and inpainting problems.

1. IntroductionAn important problem in machine learning, signal pro-cessing and computational neuroscience is to deter-mine a dictionary of basis functions for sparse rep-resentation of signals. A signal y ∈ Rd has a sparserepresentation with y = Dα in a dictionary D ∈ Rd×n,when k d coefficients of α can exactly represent orwell-approximate y. Myriad applications in data anal-ysis and processing–from deconvolution to data miningand from compression to compressive sensing–involvesuch representations. Surprisingly, there are only twomain approaches for determining data-sparsifying dic-tionaries: dictionary design and dictionary learning.

Appearing in Proceedings of the 27 th International Confer-ence on Machine Learning, Haifa, Israel, 2010. Copyright2010 by the author(s)/owner(s).

In dictionary design, researchers assume an abstractfunctional space that can concisely capture the under-lying characteristics of the signals. A classical exampleis based on Besov spaces and the set of natural images,for which the Besov norm measures spatial smoothnessbetween edges (c.f., Choi & Baraniuk (2003) and thereferences therein). Along with the functional space,a matching dictionary is naturally introduced, e.g.,wavelets (W) for Besov spaces, to efficiently calculatethe induced norm. Then, the rate distortion of the par-tial signal reconstructions yDk is quantified by keepingthe k largest dictionary elements via an `p norm, such

as σp(y, yDk ) = ‖y − yDk ‖p ≡(∑d

i=1 ‖yi − yDk,i‖p)1/p

;

the faster σp(y, yDk ) decays with k, the better the ob-servations can be compressed. While the designed dic-tionaries have well-characterized rate distortion andapproximation performance on signals in the assumedfunctional space, they are data-independent and hencetheir empirical performance on the actual observa-tions can greatly vary: σ2(y, yWk ) = O(k−0.1) (prac-tice) vs. O(k−0.5) (theory) for wavelets on natural im-ages (Cevher, 2008).

In dictionary learning, researchers develop algorithmsto learn a dictionary for sparse representation di-rectly from data using techniques such as regulariza-tion, clustering, and nonparametric Bayesian infer-ence. Regularization-based approaches define an ob-jective function that minimize the data error, regular-ized by the `1 or the total variation (TV) norms toenforce sparsity under the dictionary representation.The proposed objective function is then jointly opti-mized in the dictionary entries and the sparse coeffi-cients (Olshausen & Field, 1996; Zhang & Chan, 2009;Mairal et al., 2008). Clustering approaches learn dic-tionaries by sequentially determining clusters wheresparse coefficients overlap on the dictionary and thenupdating the corresponding dictionary elements basedon singular value decomposition (Aharon et al., 2006).Bayesian approaches use hierarchical probability mod-els to nonparametrically infer the dictionary size and


its composition (Zhou et al., 2009). Although dictio-nary learning approaches have great empirical perfor-mance on many data sets in denoising and inpaintingof natural images, they lack theoretical rate distortioncharacterizations of the dictionary design approaches.

In this paper, we investigate a hybrid approach be-tween dictionary design and learning. We propose alearning framework based on dictionary selection: Webuild a sparsifying dictionary for a set of observationsby selecting the dictionary columns from multiple can-didate bases, typically designed for the observations ofinterest. We constrain the size of the dictionary as wellas the number of dictionary columns that can be usedto represent each signal with user-defined parametersn and k, respectively. We formulate both the selectionof basis functions and the sparse reconstruction as ajoint combinatorial optimization problem. Our objec-tive function maximizes a variance reduction metricover the set of observations.

We then propose SDSOMP and SDSMA, two com-putationally efficient, greedy algorithms for dictionaryselection. We show that under certain incoherence as-sumptions on the candidate vectors, the dictionary se-lection problem is approximately submodular, and weuse this insight to derive theoretical performance guar-antees for our algorithms. We also demonstrate thatour framework naturally extends to dictionary selec-tion with restrictions on the allowed sparsity patternsin signal representation. As a stylized example, westudy a dictionary selection problem where the sparsesignal coefficients exhibit block sparsity, e.g., sparsecoefficients appear in pre-specified blocks.

Lastly, we first evaluate the performance of our algo-rithms in both on synthetic and real data. Our maincontributions can be summarized as follows:

1. We introduce the problem of dictionary selectionand cast the dictionary learning/design problemsin a new, discrete optimization framework.

2. We propose new algorithms and provide their the-oretical performance characterizations by exploit-ing a geometric connection between submodular-ity and sparsity.

3. We extend our dictionary selection framework toallow structured sparse representations.

4. We evaluate our approach on several real-worldsparse representation and image inpainting prob-lems and show that it provides practical insightsto existing image coding standards.

2. The dictionary selection problemIn the dictionary selection problem (DiSP), we seek adictionary D to sparsely represent a given collection

of signals Y = y1, . . . , ym ∈ Rd×m. We composeD using the variance reduction metric, defined below,by selecting a subset of a candidate vector set V =φ1, . . . , φN ∈ Rd×N . Without loss of generality, weassume ‖yi‖2 ≤ 1 and ‖φi‖2 = 1, ∀i. In the sequel,we define ΦA = [φi1 , . . . , φiQ ] as a matrix containingthe vectors in V as indexed by A = i1, . . . , iQ whereA ⊆ V and Q = |A| is the cardinality of the set A. Wedo not assume any particular ordering of V.

DiSP objectives: For a fixed signal ys and a set ofvectors A, we define the reconstruction accuracy as

Ls(A) = σ22(ys, yA) = min

w||ys − ΦAw||22. (1)

The problem of optimal k-sparse representation withrespect to a fixed dictionary D then requires solvingthe following discrete optimization problem:

As = argminA⊆D,|A|≤k

Ls(A), (2)

where k is the user-defined sparsity constraint on thenumber of columns in the reconstruction.

In DiSP, we are interested in determining a dictionaryD ⊆ V that obtains the best possible reconstructionaccuracy for not only a single signal but all signals Y.Each signal ys can potentially use different columnsAs ⊆ D for representation; we thus define

Fs(D) = Ls(∅)− minA⊆D,|A|≤k

Ls(A), (3)

where Fs(D) measures the improvement in reconstruc-tion accuracy, also known as variance reduction, for thesignal ys and the dictionary D. Moreover, we definethe average improvement for all signals as

F (D) =1m

∑s

Fs(D).

The optimal solution to the DiSP is then given by

D∗ = argmax|D|≤n

F (D), (4)

where n is a user-defined constraint on the number ofdictionary columns. For instance, if we are interestedin selecting a basis, we have n = d.

DiSP challenges: The optimization problem in (4)presents two combinatorial challenges. (C1) Evalu-ating Fs(D) requires finding the set As of k basisfunctions–out of exponentially many options–for thebest reconstruction accuracy of ys. (C2) Even if wecould evaluate Fs, we would have to search over anexponential number of possible dictionaries to deter-mine D∗ for all signals. Even the special case of k = nis NP-hard (Davis et al., 1997). To circumvent these


combinatorial challenges, the existing dictionary learn-ing work relies on continuous relaxations, such as re-placing the combinatorial sparsity constraint with the`1-norm of the dictionary representation of the signal.However, these approaches result in non-convex objec-tives, and the performance of such relaxations is typi-cally not well-characterized for dictionary learning.

3. Submodularity in sparserepresentation

In this section, we first describe a key structure in theDiSP objective function: approximate submodularity.We then relate this structure to a geometric propertyof the candidate vector set, called incoherence. We usethese two concepts to develop efficient algorithms withprovable guarantees in the next section.

Approximate submodularity in DiSP: To definethis concept, we first note that F (∅) = 0 and when-ever D ⊆ D′ then F (D) ≤ F (D′), i.e., F increasesmonotonically with D. In the sequel, we will showthat F is approximately submodular: A set functionF is called approximately submodular with constant ε,if for D ⊆ D′ ⊆ V and v ∈ V \ D′ it holds that

F (D ∪ v)− F (D) ≥ F (D′ ∪ v)− F (D′)− ε. (5)

In the context of DiSP, the above definition impliesthat adding a new column v to a larger dictionary D′helps at most εmore than adding v to a subsetD ⊆ D′.When ε = 0, the set function is called submodular.

A fundamental result by Nemhauser et al. (1978)proves that for monotonic submodular functions Gwith G(∅) = 0, a simple greedy algorithm that startswith the empty set D0 = ∅, and at every iteration iadds a new element via

vi = argmaxv∈V\D

G(Di−1 ∪ v), (6)

where Di = v1, . . . , vi, obtains a near-optimal so-lution. That is, for the solution Dn returned by thegreedy algorithm, we have the following guarantee:

G(Dn) ≥ (1− 1/e) max|D|≤n

G(D). (7)

The solution Dn hence obtains at least a constant frac-tion of (1− 1/e) ≈ 63% of the optimal value.

Using similar arguments, Krause et al. (2008) showthat the same greedy algorithm, when applied to ap-proximately submodular functions, instead inheritsthe following–slightly weaker–guarantee

F (Dn) ≥ (1− 1/e) max|D|≤n

F (D)− nε. (8)

In Section 4, we explain how this greedy algorithm canbe adapted to DiSP. But first, we elaborate on how εdepends on the candidate vector set ΦV .

cos( )

A

v

ys

θψ

cos( )

ψ−θ

θ

sin( )ψ−θ

sin( )θ

A

v

ys

θψ

ys||A

ys||v

ws,v

ws,A

ws,vws,v1/2

1/2

Figure 1. Example geometry in DiSP. (Left) Minimum er-ror decomposition. (Right) Modular decomposition.

Geometry in DiSP (incoherence): The approx-imate submodularity of F explicitly depends on themaximum incoherency µ of ΦV = [φ1, . . . , φN ]:

µ = max∀(i,j),i6=j

|〈φi, φj〉| = max∀(i,j),i6=j

|cosψi,j | ,

where ψi,j is the angle between the vectors φi and φj .

The following lemma establishes a key relationship be-tween ε and µ for DiSP.

Theorem 1 If ΦV has incoherence µ, then the vari-ance reduction objective F in DiSP is ε-approximatelysubmodular with ε ≤ 4kµ.

Proof Let ws,v = 〈φv, ys〉2. When ΦV is an orthonor-mal basis, the reconstruction accuracy in (1) can bewritten as follows

Ls(A) =∣∣∣∣∣∣ys − Q∑

q=1

φiq 〈ys, φiq 〉∣∣∣∣∣∣2

2= ||ys||22 −

∑v∈A

ws,v.

Hence the function Rs(A) ≡ Ls(∅) − Ls(A) =∑v∈A ws,v is additive (modular). It can be seen that

then Fs(D) = maxA⊆D,|A|≤k Rs(A) is submodular.

Now suppose ΦV is incoherent with constant µ. LetA ⊆ V and v ∈ V \ A. Then we claim that|Rs(A ∪ v) − Rs(A) − ws,v| ≤ µ. Consider the spe-cial case where ys is in the span of two subspaces Aand v, and w.l.o.g., ||ys||2 = 1; refer to Fig. 1 for anillustration. The reconstruction accuracy as definedin (1) has a well-known closed form solution: Ls(A) =minw ||ys−ΦAw||22 = ||ys−ΦAΦ†Ays||22, where † denotesthe pseudoinverse; the matrix product P = ΦAΦ†A issimply the projection of the signal ys onto the sub-space of A. We therefore have Rs(A) = 1 − sin2(θ),Rs(A∪v) = 1, and Rs(v) = 1−sin2(ψ−θ), whereθ and ψ are defined in Fig. 1. We thus can boundεs ≡ |Rs(A ∪ v)−Rs(A)− wv,s| by

εs ≤ maxθ

∣∣sin2(ψ − θ) + sin2(θ)− 1∣∣

= |cosψ|maxθ|cos(ψ − 2θ)| = µ.

If ys is not in the span of A ∪ v, we apply abovereasoning to the projection of ys onto their span.


Define Rs(A) =∑v∈A ws,v. Then, by induction,

we have |Rs(A) − Rs(A)| ≤ kµ. Note that thefunction Fs(D) = maxA⊆D,|A|≤k Rs(A) is submodu-lar. Let As = argmaxA⊆D,|A|≤k Rs(A) and As =argmaxA⊆D,|A|≤k Rs(A). Therefore, it holds that

Fs(D)=Rs(As)≤R(As)+kµ≤R(As)+kµ= Fs(D)+kµ.

Similarly, Fs(D) ≤ Fs(D) + kµ. Thus, |Fs(D) −Fs(D)| ≤ kµ, and hence |F (D) − F (D)| ≤ kµ holdsfor all candidate dictionaries D. Therefore, wheneverD ⊆ D′ and v /∈ D′, we can obtain the following

F (D ∪ v)− F (D)− F (D′ ∪ v) + F (D′)

≥F (D ∪ v)− F (D)− F (D′ ∪ v) + F (D′)− 4kµ≥− 4kµ, which proves the claim.

When the incoherency µ is small, the approximationguarantee in (8) is quite useful. There has been a sig-nificant body of work establishing the existence andconstruction of collections V of columns with low co-herence µ. For example, it is possible to achieve inco-herence µ ≤ d−1/2 with the union of d/2 orthonormalbases (c.f. Theorem 2 of Gribonval & Nielsen (2002)).

Unfortunately, when n = Ω(d) and ε = 4kµ, the guar-antee (8) is vacuous since the maximum value of F forDiSP is 1. In Section 4, we will show that if, instead ofgreedily optimizing F , we optimize a modular approx-imation Fs of Fs (as defined below), we can improvethe approximation error from O(nkµ) to O(kµ).

A modular approximation to DiSP: The keyidea behind the proof of Theorem 1 is that for in-coherent dictionaries the variance reduction Rs(A) =Ls(∅) − Ls(A) is approximately additive (modular).We exploit this observation by optimizing a new objec-tive F that approximates F by disregarding the non-orthogonality of ΦV in sparse representation. We dothis by replacing the weight calculation ws,A = Φ†Aysin F with ws,A = ΦTAys:

Fs(D) = maxA⊆D,|A|≤k

∑v∈A

ws,v, and F (D) =1m

∑s

Fs(D),

(9)where ws,v = 〈φv, ys〉2 for each ys ∈ Rd and φv ∈ ΦV .We call F a modular approximation of F as it relies onthe approximate modularity of the variance reductionRs. Note that in contrast to (3), Fs(D) in (9) can beexactly evaluated by a greedy algorithm that simplypicks the k largest weights ws,v. Moreover, the weightsmust be calculated only once during algorithm execu-tion, thereby significantly increasing its efficiency.

The following immediate Corollary to Theorem 1 sum-marizes the essential properties of F :

Corollary 1 Suppose ΦV is incoherent with constantµ. Then, for any D ⊆ V, we have |F (D)−F (D)| ≤ kµ.Furthermore, F is monotonic and submodular.

Corollary 1 shows that F is a close approximation ofthe DiSP set function F . We exploit this modular ap-proximation to motivate a new algorithm for DiSP andprovide better performance bounds in Section 4.4. Sparsifying dictionary selectionIn this section, we describe two sparsifying dictionaryselection (SDS) algorithms with theoretical perfor-mance guarantees: SDSOMP and SDSMA. Both algo-rithms make locally greedy choices to handle the com-binatorial challenges C1 and C2, defined in Section 2.The algorithms differ only in the way they addressC1, which we further describe below. Both algorithmstackle C2 by the same greedy scheme in (6). That is,both algorithms start with the empty set and greedilyadd dictionary columns to solve DiSP. Interestingly,while SDSMA has better theoretical guarantees andis much faster than SDSOMP , Section 6 empiricallyshows that SDSOMP often performs better.SDSOMP : SDSOMP employs the orthogonal match-ing pursuit (OMP) (Gilbert & Tropp, 2005) to approx-imately solve the sparse representation problem in (2)and has the following theoretical guarantee:

Theorem 2 SDSOMP uses the scheme in (6) to builda dictionary DOMP one column at a time such that

F (DOMP) ≥ (1− 1/e) max|D|≤n

F (D)− k(6n+ 2− 1/e)µ.

Before we prove Theorem 2, we state the followingresult whose proof directly follows from Theorem 1.

Proposition 1 At each iteration, SDSOMP approxi-mates F with a value FOMP such that |FOMP (D) −F (D)| ≤ kµ over all dictionaries D.

Proof [Theorem 2] From Theorem 1 and Proposition 1we can see that FOMP is 6knµ-approximately submod-ular. Thus, according to Krause et al. (2008):

FOMP (DOMP) ≥ (1− 1/e) max|D|≤n

FOMP (D)− 6knµ.

(10)Using Proposition 1, we substitute F (DOMP) + kµ ≥FOMP (DOMP) and max|D|≤n FOMP (D) ≥ max|D|≤nF (D)− kµ into (10) to prove the claim.

SDSMA: SDSMA greedily (according to (6)) opti-mizes the modular approximation (MA) F of the DiSPobjective F and has the following guarantee:

Theorem 3 SDSMA builds a dictionary DMA s.t.

F (DMA) ≥ (1− 1/e) max|D|≤n

F (D)− (2− 1/e)kµ. (11)

Corollary 1 and Theorem 2 directly imply Theorem 3.


In most realistic settings with high-dimensional signalsand incoherent dictionaries, the term (2−1/e)kµ in theapproximation guarantee (11) of SDSMA is negligible.

5. Sparsifying dictionary selectionfor block sparse representationStructured sparsity: While many man-made andnatural signals can be described as sparse in simpleterms, their sparse coefficients often have an under-lying, problem dependent order. For instance, mod-ern image compression algorithms, such as JPEG, notonly exploit the fact that most of the DCT coefficientsof a natural image are small. Rather, they also ex-ploit the fact that the large coefficients have a particu-lar structure characteristic of images containing edges.Coding this structure using an appropriate model en-ables transform coding algorithms to compress imagesclose to the maximum amount possible and signifi-cantly better than a naive coder that just assigns bitsto each large coefficient independently (Mallat, 1999).

We can enforce structured sparsity for sparse coef-ficients over the learned dictionaries in DiSP, cor-responding to a restricted union-of-subspaces (RUS)sparse model by imposing the constraint that the fea-sible sparsity patterns are a strict subset of all k-dimensional subspaces (Baraniuk et al., 2008). To fa-cilitate such RUS sparse models in DiSP, we must notonly determine the constituent dictionary columns,but also their arrangement within the dictionary.While analyzing the RUS model in general is challeng-ing, we here describe below a special RUS model ofbroad interest to explain the general ideas.

Block-sparsity: Block-sparsity is abundant inmany applications. In sensor networks, multiplesensors simultaneously observe a sparse signal overa noisy channel. While recovering the sparse signaljointly from the sensors, we can use the fact that thesupport of the significant coefficients of the signal arecommon across all the sensors. In DNA microarrayapplications, specific combinations of genes arealso known a priori to cluster over tree structures,called dendrograms. In computational neuroscienceproblems, decoding of natural images in the primaryvisual cortex (V1) and statistical behavior of neuronsin the retina exhibit clustered sparse responses.

To address block-sparsity in DiSP, we replace (3) by

Fi(D) =∑s∈Bi

Ls(∅)− minA⊆D,|A|≤k

∑s∈Bi

Ls(A), (12)

where Bi is the i-th block of signals (e.g., simultane-ous recordings by multiple sensors) that must sharethe same sparsity pattern. Accordingly, we redefineF (D) =

∑i Fi(D) as the sum across blocks, rather

than individual signals, as Section 6 further elaborates.

This change preserves (approximate) submodularity.

6. ExperimentsFinding a dictionary in a haystack: To under-stand how the theoretical performance reflects on theactual performance of the proposed algorithms, wefirst perform experiments on synthetic data.

We generate a collection VU with 400 columns by form-ing a union of six orthonormal bases with d = 64,including the discrete cosine transform (DCT), dif-ferent wavelet bases (Haar, Daub4, Coiflets), noise-lets, and the Gabor frame. This collection VU is notincoherent—in fact, the various bases contain perfectlycoherent columns. As alternatives, we first create aseparate collection VS from VU , where we greedily re-moved columns based on their incoherence, until theremaining collection had incoherence of µS = 0.5. Theresulting collection contains 245 columns. We also cre-ate a collection VR with 150 random columns of VU ,which results in µR = 0.23.

For VU,S,R, we repeatedly (50 trials) pick at randoma dictionary D∗ ⊆ V of size n = 64 and generate acollection of m = 100 random 5-sparse signals withrespect to the dictionary D∗. Our goal is to recoverthe true dictionary D∗ using our SDS algorithms. Foreach random trial, we run SDSOMP and SDSMA toselect a dictionary D of size 64. We then look at theoverlap |D∩D∗| to measure the performance of select-ing the “hidden” basis D∗. We also report the fractionof remaining variance after sparse reconstruction.

Figures 2(a), 2(b), and 2(c) compare SDSOMP andSDSMA in terms of their variance reduction as a func-tion of the selected number of columns. Interestingly,in all 50 trials, SDSOMP perfectly reconstructs thehidden basis D∗ when selecting 64 columns for VS,R.SDSMA performs slightly worse than SDSOMP .

Figures 2(e), 2(f), and 2(g) compare the performancein terms of the fraction of incorrectly selected basisfunctions. Note that, as can be expected, in case ofthe perfectly coherent VU , even SDSOMP does notachieve perfect recovery. However, even with high co-herence, µ = 0.5 for VS , SDSOMP exactly identifiesD∗. SDSMA performs a slightly worse but neverthe-less correctly identifies a high fraction of D∗.

In addition to exact sparse signals, we also gener-ate compressible signals, where the coefficients havepower-law with decay rate of 2. These signals canbe well-approximated as sparse; however, the residualerror in sparse representation creates discrepancies inmeasurements which can be modeled as noise in DiSP.Figures 2(d) and 2(h) repeat the above experiments forVS ; both SDSOMP and SDSMA perform quite well.


0 50 1000

0.2

0.4

0.6

0.8

1

# columns selected

Frac

tion

of re

mai

ning

var

ianc

e

SDSOMPSDSMA

(a) VU –sparse

0 50 1000

0.2

0.4

0.6

0.8

1

# columns selected

Frac

tion

of re

mai

ning

var

ianc

e

SDSOMPSDSMA

(b) VS–sparse

0 50 1000

0.2

0.4

0.6

0.8

1

# columns selected

Frac

tion

of re

mai

ning

var

ianc

e

SDSOMPSDSMA

(c) VR–sparse

0 50 1000

0.2

0.4

0.6

0.8

1

# columns selected

Frac

tion

of re

mai

ning

var

ianc

e

SDSOMPSDSMA

(d) VS–compressible

0 50 1000

0.2

0.4

0.6

0.8

1

# columns selected

Frac

tion

of in

corr

ect a

tom

s

SDSOMPSDSMA

(e) VU –sparse

0 50 1000

0.2

0.4

0.6

0.8

1

# columns selected

Frac

tion

of in

corr

ect a

tom

s

SDSOMPSDSMA

(f) VS–sparse

0 50 1000

0.2

0.4

0.6

0.8

1

# columns selected

Frac

tion

of in

corr

ect a

tom

s

SDSOMPSDSMA

(g) VR–sparse

0 50 1000

0.2

0.4

0.6

0.8

1

# columns selected

Frac

tion

of in

corr

ect a

tom

s

SDSOMPSDSMA

(h) VS–compressible

100

101

102

10310

0

105

Run

ning

tim

e (s

)

Dimension

SDSOMP

SDSMA

(i) Dimension vs. runtime

0.1 0.2 0.3 0.40

0.02

0.04

0.06

0.08

Incoherence µ

Res

idua

l var

ianc

e

SDSOMP

SDSMA

(j) Incoherence vs. res. var.

102

1030

0.005

0.01

0.015

0.02

0.025

Number of signals

Res

idua

l var

ianc

e

SDSOMP

SDSMA

(k) # Signals vs. res. var.

0 50 1000

0.2

0.4

0.6

0.8

1

# columns selected

Frac

tion

of in

corr

ect a

tom

s

SDSOMPSDSMASDSMAB

(l) Block-sparse

Figure 2. Results of 50 trials: (a-c) Variance reduction achieved by SDSOMP and SDSMA on the collections VU,S,R for5-sparse signals in 64 dimensions. (e-g) Percentage of incorrectly selected columns on the same collections. (d) Variancereduction for compressible signals in 64 dimensions for VS . (h) Corresponding column selection performance. (i) SDSMA

is orders of magnitude faster than SDSOMP over a broad range of dimensions. (j) As incoherence decreases, the algorithmeffectiveness in variance reduction improve. (k) The variance reduction performance of SDSMA improves with the numberof training samples. (l) Exploiting block-sparse structure in signals leads to improved dictionary selection performance.

Figure 2(i) compares SDSOMP and SDSMA in run-ning time. As we increase the dimensionality of theproblem, SDSMA is several orders of magnitude fasterthan SDSOMP in our MATLAB implementation. Fig-ure 2(j) illustrates the performance of the algorithmsas a function of the incoherence. As predicted by The-orems 2 and 3, lower incoherence µ leads to improvedperformance of the algorithms. Lastly, Figure 2(k)compares the residual variance as a function of thetraining set size (number of signals). Surprisingly,as the number of signals increase, the performance ofSDSMA improves, and even exceeds that of SDSOMP .

We also test the extension of SDSMA to block-sparsesignals as discussed in Section 5. We generate 200random signals each with fixed sparsity pattern, com-prising 10 blocks, consisting of 20 signals each. Wethen compare the standard SDSMA algorithm withthe block-sparse variant SDSMAB described in Sec-tion 5 in terms of their basis identification perfor-mance (see Figure 2(l)). SDSMAB drastically outper-forms SDSMA, and even outperforms the SDSOMP

algorithm which is computationally far more expen-sive. Hence, exploiting prior knowledge of the problemstructure can significantly aid dictionary selection.

A battle of bases on image patches: In this ex-periment, we try to find the optimal dictionary amongan existing set of bases to represent natural images.Since the conventional dictionary learning approachescannot be applied to this problem, we only present theresults of SDSOMP and SDSMA.

We sample image patches from natural images, andapply our SDSOMP and SDSMA algorithms to selectdictionaries from the collection VU , as defined above.Figures 3(a) (for SDSOMP ) and 3(b) (for SDSMA)show the fractions of selected columns allocated to thedifferent bases constituting VU for 4000 image patchesof size 8 × 8. We restrict the maximum number ofdictionary coefficients k for sparse representation to10% (6). We then observe the following surprisingresults. While wavelets are considered to be animprovement over the DCT basis for compressingnatural images (JPEG2000 vs. JPG), SDSOMP prefer


0 50 100 1500

0.2

0.4

0.6

0.8

1

Number of columns

Frac

tion

per b

asis

DCTHaarDaub4Coif1NoiseGabor

(a) Bases used (SDSOMP )

0 50 100 1500

0.2

0.4

0.6

0.8

1

Number of columns

Frac

tion

per b

asis


(b) Bases used (SDSMA)

0 20 40 600.2

0.4

0.6

0.8

1

Number of columns

Frac

tion

of re

sidu

al v

aria

nce

HaarDaub4

Gabor

SDSOMP

DCT

(c) Test set var. reduction

0 500 1000 1500 20000

0.2

0.4

0.6

0.8

1

Number of columns

Frac

tion

per b

asis


(d) Bases used (SDSMA)

Figure 3. Experiments on natural image patches. (a,b,c) Fractions of bases selected for SDSOMP and SDSMA with d = 64(a,b), and the corresponding variance reduction on test patches. (d) Fractions of bases selected for SDSMA with d = 1024.

DCT over wavelets for sparse representation; the crossvalidation results show that the learned combinationof DCT (global) and Gabor functions (local) are betterthan the wavelets (multiscale) in variance reduction(compression). In particular, Fig. 3(c) demonstratesthe performance of the learned dictionary against thevarious bases that comprise VU on a held-out testset of 500 additional image patches. The variancereduction of the dictionary learned by SDSOMP is8% lower than the variance reduction achieved by thebest basis, which, in this case, is DCT.

Moreover, SDSMA, which trades off representationaccuracy with efficient computation, overwhelminglyprefers Gabor functions that are used to model neu-ronal coding of natural images. The overall dictionaryconstituency varies for SDSOMP and SDSMA; how-ever, the variance reduction performances are compa-rable. Finally, Figure 3(d) presents the fraction of se-lected bases for 32 × 32 sized patches with k = 102,which matches well with the 8×8 DiSP problem above.

Dictionary selection from dimensionality re-duced data: In this experiment, we focus on a spe-cific image processing problem, inpainting, to moti-vate a dictionary selection problem from dimension-ality reduced data. Suppose that instead of observ-ing Y as assumed in Section 2, we observe Y ′ =P1y1, . . . ,Pmym ∈ Rb, where Pi ∈ Rb×d ∀i are knownlinear projection matrices. In the inpainting setting,Pi’s are binary matrices which pass or delete pixels.From a theoretical perspective, dictionary selectionfrom dimensionality reduced data is ill-posed. For thepurposes of this demonstration, we will assume thatPi’s are information preserving.

As opposed to observing a series of signal vectors,we start with a single image in Fig. 4, albeit missing50% of its pixels. We break the noisy image intonon-overlapping 8× 8 patches, and train a dictionaryfor sparse reconstruction of those patches to minimizethe average approximation error on the observedpixels. As candidate bases, we use DCT, wavelets(Haar and Daub4), Coiflets (1 and 3), and Gabor.We test our SDSOMP and SDSMA algorithms,

approaches based on total-variation (TV), linearinterpolation, nonlocal TV and the nonparametricBayesian dictionary learning (based on Indian buffetprocesses) algorithms (Zhang & Chan, 2009; Mairalet al., 2008; Zhou et al., 2009). The TV and nonlocalTV algorithms use the linear interpolation result astheir initial estimates. We set k = 6 (10%). Figure 4illustrates the inpainting results for each algorithmsorted in increasing peak signal to noise ratio (PSNR).We do not report the reconstruction results usingindividual candidate bases since they are significantlyworse than the baseline linear interpolation.

The test image exhibits significant self similarities, re-stricting the degrees-of-freedom of the sparse coeffi-cients. Hence, for our modular and OMP-based greedyalgorithms, we ask the algorithms to select 64× 32 di-mensional dictionaries. While the modular algorithmSDSMA selects the desired dimensions, the OMP-based greedy algorithm SDSOMP terminates when thedictionary dimensions reach 64×19. Given the selecteddictionaries, we determine the sparse coefficients thatbest explain the observed pixels in a given patch andreconstruct the full patch using the same coefficients.We repeat this process for all the patches in the im-age that differ by a single pixel. In our final recon-struction, we take the pixel median of all the recon-structed patches. SDSOMP performs on par with non-local TV while taking a fraction of its computationaltime. While the Bayesian approach takes significantlymore time (a few order of magnitudes slower), it bestexploits the self similarities in the observed image toresult in the best reconstruction.

7. ConclusionsOver the last decade, a great deal of research revolvedaround recovering, processing, and coding sparsesignals. To leverage this experience in new problems,many researchers are now interested in automati-cally determining data sparsifying dictionaries fortheir applications. We discussed two alternativesthat focus on this problem: dictionary design anddictionary learning. In this paper, we developed acombinatorial theory for dictionary selection that


Original Incomplete TV (27.55dB) Lin. Interp. (27.58dB)

SDSMA (30.32dB) SDSOMP (31.91dB) Nonlocal TV (32.86dB) Bayesian (35.17dB)Figure 4. Comparison of inpainting algorithms.

bridges the gap between the two approaches. Weexplored new connections between the combinatorialstructure of submodularity and the geometric conceptof incoherence. We presented two computationallyefficient algorithms, SDSOMP based on the OMPalgorithm, and SDSMA using a modular approxima-tion. By exploiting the approximate submodularityproperty of the DiSP objective, we derived theoreticalapproximation guarantees for the performance of ouralgorithms. We also demonstrated the ability of ourlearning framework to incorporate structured sparsityrepresentations in dictionary learning. Compared tothe dictionary design approaches, our approach isdata adaptive and has better empirical performanceon data sets. Compared to the continuous natureof the dictionary learning approaches, our approachis discrete and provides new theoretical insights tothe dictionary learning problem. We believe thatour results pave a promising direction for furtherresearch, exploiting combinatorial optimization forsparse representations, in particular submodularity.

Acknowledgements. This research is supported byONR grants N00014-09-1-1044 and N00014-08-1-1112,NSF grant CNS-0932392, AFOSR FA9550-07-1-0301,ARO W911NF-09-1-0383, DARPA N66001-08-1-2065,ARO MURI W911NF-09-1-0383.

ReferencesAharon, M., Elad, M., and Bruckstein, A. The k-

SVD: An algorithm for designing of overcompletedictionaries for sparse representation. IEEE Trans.on Signal Processing, 54(11):4311–4322, 2006.

Baraniuk, R. G., Cevher, V., Duarte, M. F., andHegde, C. Model-based compressive sensing. IEEE

Trans. on Inform. Theory, 56(11):1982–2001, 2010.Cevher, V. Learning with compressible priors. In

NIPS, Vancouver, B.C., Canada, 2008.Choi, H. and Baraniuk, R. G. Wavelet statistical mod-

els and Besov spaces. Lect. Notes in Statistics, 2003.Davis, G., Mallat, S., and Avellaneda, M. Greedy

adaptive approximation. J. Const. Approx., 13:57–98, 1997.

Gilbert, A. C. and Tropp, J. A. Signal recovery fromrandom measurements via orthogonal matching pur-suit. Technical report, University of Michigan, 2005.

Gribonval, R. and Nielsen, M. Sparse decompositionsin “incoherent” dictionaries. In ICIP, 2002.

Krause, A., Singh, A., and Guestrin, C. Near-optimalsensor placements in Gaussian processes: Theory,efficient algorithms and empirical studies. In JMLR,vol. 9, 2008.

Mairal, J. and Bach, F. and Ponce, J. and Sapiro,G. Online dictionary learning for sparse coding InICML, 2008.

Mallat, S. G. A wavelet tour of signal processing. Aca-demic Press, 1999.

Nemhauser, G., Wolsey, L., and Fisher, M. An analysisof the approximations for maximizing submodularset functions. Math. Prog., 14:265–294, 1978.

Olshausen, B. A. and J., Field D. Emergence of simple-cell receptive field properties by learning a sparsecode for natural images. Nature, 381:607–609, 1996.

Zhang, X. and Chan, T. F. Wavelet Inpainting by Non-local Total Variation. CAM Report (09-64), 2009.

Zhou, M., Chen, H., Paisley, J., Ren, L., Sapiro, G.,and Carin, L. Non-parametric bayesian dictionarylearning for sparse image representations. NIPS, ’09.

submodular dictionary selection for sparse representation (pdf, 2

Documents