ℓ1-norm methods for convex-cardinality problems · 1-norm heuristics for cardinality problems •...
TRANSCRIPT
![Page 1: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/1.jpg)
ℓ1-Norm Methods for
Convex-Cardinality Problems
EE364b, Stanford University
![Page 2: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/2.jpg)
Outline
• problems involving cardinality
• the ℓ1-norm heuristic
• convex relaxation and convex envelope interpretations
• examples
• recent results
• total variation
• iterated weighted ℓ1 heuristic
• matrix rank constraints
ℓ1-norm methods for convex-cardinality problems 1
![Page 3: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/3.jpg)
ℓ1-norm heuristics for cardinality problems
• cardinality problems arise often, but are hard to solve exactly
• a simple heuristic, that relies on ℓ1-norm, seems to work well
• used for many years, in many fields
– sparse design– LASSO, robust estimation in statistics– support vector machine (SVM) in machine learning– total variation reconstruction in signal processing, geophysics– compressed sensing
• recent theoretical results guarantee the method works, at least for a fewproblems
ℓ1-norm methods for convex-cardinality problems 2
![Page 4: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/4.jpg)
Cardinality
• the cardinality of x ∈ Rn, denoted card(x), is the number of nonzerocomponents of x
• card is separable; for scalar x, card(x) =
{
0 x = 01 x 6= 0
• card is quasiconcave on Rn+ (but not Rn) since
card(x+ y) ≥ min{card(x), card(y)}
holds for x, y � 0
• but otherwise has no convexity properties
• arises in many problems
ℓ1-norm methods for convex-cardinality problems 3
![Page 5: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/5.jpg)
General convex-cardinality problems
a convex-cardinality problem is one that would be convex, except forappearance of card in objective or constraints
examples (with C, f convex):
• convex minimum cardinality problem:
minimize card(x)subject to x ∈ C
• convex problem with cardinality constraint:
minimize f(x)subject to x ∈ C, card(x) ≤ k
ℓ1-norm methods for convex-cardinality problems 4
![Page 6: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/6.jpg)
Solving convex-cardinality problems
convex-cardinality problem with x ∈ Rn
• if we fix the sparsity pattern of x (i.e., which entries are zero/nonzero)we get a convex problem
• by solving 2n convex problems associated with all possible sparsitypatterns, we can solve convex-cardinality problem(possibly practical for n ≤ 10; not practical for n > 15 or so . . . )
• general convex-cardinality problem is (NP-) hard
• can solve globally by branch-and-bound
– can work for particular problem instances (with some luck)– in worst case reduces to checking all (or many of) 2n sparsity patterns
ℓ1-norm methods for convex-cardinality problems 5
![Page 7: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/7.jpg)
Boolean LP as convex-cardinality problem
• Boolean LP:minimize cTxsubject to Ax � b, xi ∈ {0, 1}
includes many famous (hard) problems, e.g., 3-SAT, traveling salesman
• can be expressed as
minimize cTxsubject to Ax � b, card(x) + card(1− x) ≤ n
since card(x) + card(1− x) ≤ n ⇐⇒ xi ∈ {0, 1}
• conclusion: general convex-cardinality problem is hard
ℓ1-norm methods for convex-cardinality problems 6
![Page 8: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/8.jpg)
Sparse design
minimize card(x)subject to x ∈ C
• find sparsest design vector x that satisfies a set of specifications
• zero values of x simplify design, or correspond to components thataren’t even needed
• examples:
– FIR filter design (zero coefficients reduce required hardware)– antenna array beamforming (zero coefficients correspond to unneeded
antenna elements)– truss design (zero coefficients correspond to bars that are not needed)– wire sizing (zero coefficients correspond to wires that are not needed)
ℓ1-norm methods for convex-cardinality problems 7
![Page 9: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/9.jpg)
Sparse modeling / regressor selection
fit vector b ∈ Rm as a linear combination of k regressors (chosen from npossible regressors)
minimize ‖Ax− b‖2subject to card(x) ≤ k
• gives k-term model
• chooses subset of k regressors that (together) best fit or explain b
• can solve (in principle) by trying all
(
nk
)
choices
• variations:
– minimize card(x) subject to ‖Ax− b‖2 ≤ ǫ– minimize ‖Ax− b‖2 + λ card(x)
ℓ1-norm methods for convex-cardinality problems 8
![Page 10: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/10.jpg)
Sparse signal reconstruction
• estimate signal x, given
– noisy measurement y = Ax+ v, v ∼ N (0, σ2I) (A is known; v is not)– prior information card(x) ≤ k
• maximum likelihood estimate xml is solution of
minimize ‖Ax− y‖2subject to card(x) ≤ k
ℓ1-norm methods for convex-cardinality problems 9
![Page 11: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/11.jpg)
Estimation with outliers
• we have measurements yi = aTi x+ vi + wi, i = 1, . . . ,m
• noises vi ∼ N (0, σ2) are independent
• only assumption on w is sparsity: card(w) ≤ k
• B = {i | wi 6= 0} is set of bad measurements or outliers
• maximum likelihood estimate of x found by solving
minimize∑
i 6∈B(yi − aTi x)2
subject to |B| ≤ k
with variables x and B ⊆ {1, . . . ,m}
• equivalent tominimize ‖y −Ax− w‖22subject to card(w) ≤ k
ℓ1-norm methods for convex-cardinality problems 10
![Page 12: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/12.jpg)
Minimum number of violations
• set of convex inequalities
f1(x) ≤ 0, . . . , fm(x) ≤ 0, x ∈ C
• choose x to minimize the number of violated inequalities:
minimize card(t)subject to fi(x) ≤ ti, i = 1, . . . ,m
x ∈ C, t ≥ 0
• determining whether zero inequalities can be violated is (easy) convexfeasibility problem
ℓ1-norm methods for convex-cardinality problems 11
![Page 13: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/13.jpg)
Linear classifier with fewest errors
• given data (x1, y1), . . . , (xm, ym) ∈ Rn × {−1, 1}
• we seek linear (affine) classifier y ≈ sign(wTx+ v)
• classification error corresponds to yi(wTx+ v) ≤ 0
• to find w, v that give fewest classification errors:
minimize card(t)subject to yi(w
Txi + v) + ti ≥ 1, i = 1, . . . ,m
with variables w, v, t (we use homogeneity in w, v here)
ℓ1-norm methods for convex-cardinality problems 12
![Page 14: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/14.jpg)
Smallest set of mutually infeasible inequalities
• given a set of mutually infeasible convex inequalitiesf1(x) ≤ 0, . . . , fm(x) ≤ 0
• find smallest (cardinality) subset of these that is infeasible
• certificate of infeasibility is g(λ) = infx(∑m
i=1 λifi(x)) ≥ 1, λ � 0
• to find smallest cardinality infeasible subset, we solve
minimize card(λ)subject to g(λ) ≥ 1, λ � 0
(assuming some constraint qualifications)
ℓ1-norm methods for convex-cardinality problems 13
![Page 15: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/15.jpg)
Portfolio investment with linear and fixed costs
• we use budget B to purchase (dollar) amount xi ≥ 0 of stock i
• trading fee is fixed cost plus linear cost: β card(x) + αTx
• budget constraint is 1Tx+ β card(x) + αTx ≤ B
• mean return on investment is µTx; variance is xTΣx
• minimize investment variance (risk) with mean return ≥ Rmin:
minimize xTΣxsubject to µTx ≥ Rmin, x � 0
1Tx+ β card(x) + αTx ≤ B
ℓ1-norm methods for convex-cardinality problems 14
![Page 16: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/16.jpg)
Piecewise constant fitting
• fit corrupted xcor by a piecewise constant signal x with k or fewer jumps
• problem is convex once location (indices) of jumps are fixed
• x is piecewise constant with ≤ k jumps ⇐⇒ card(Dx) ≤ k, where
D =
1 −11 −1
. . . . . .1 −1
∈ R(n−1)×n
• as convex-cardinality problem:
minimize ‖x− xcor‖2subject to card(Dx) ≤ k
ℓ1-norm methods for convex-cardinality problems 15
![Page 17: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/17.jpg)
Piecewise linear fitting
• fit xcor by a piecewise linear signal x with k or fewer kinks
• as convex-cardinality problem:
minimize ‖x− xcor‖2subject to card(∇2x) ≤ k
where
∇2 =
−1 2 −1−1 2 −1
. . . . . . . . .−1 2 −1
ℓ1-norm methods for convex-cardinality problems 16
![Page 18: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/18.jpg)
ℓ1-norm heuristic
• replace card(z) with γ‖z‖1, or add regularization term γ‖z‖1 toobjective
• γ > 0 is parameter used to achieve desired sparsity(when card appears in constraint, or as term in objective)
• more sophisticated versions use∑
iwi|zi| or∑
iwi(zi)+ +∑
i vi(zi)−,where w, v are positive weights
ℓ1-norm methods for convex-cardinality problems 17
![Page 19: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/19.jpg)
Example: Minimum cardinality problem
• start with (hard) minimum cardinality problem
minimize card(x)subject to x ∈ C
(C convex)
• apply heuristic to get (easy) ℓ1-norm minimization problem
minimize ‖x‖1subject to x ∈ C
ℓ1-norm methods for convex-cardinality problems 18
![Page 20: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/20.jpg)
Example: Cardinality constrained problem
• start with (hard) cardinality constrained problem (f , C convex)
minimize f(x)subject to x ∈ C, card(x) ≤ k
• apply heuristic to get (easy) ℓ1-constrained problem
minimize f(x)subject to x ∈ C, ‖x‖1 ≤ β
or ℓ1-regularized problem
minimize f(x) + γ‖x‖1subject to x ∈ C
β, γ adjusted so that card(x) ≤ k
ℓ1-norm methods for convex-cardinality problems 19
![Page 21: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/21.jpg)
Polishing
• use ℓ1 heuristic to find x with required sparsity
• fix the sparsity pattern of x
• re-solve the (convex) optimization problem with this sparsity pattern toobtain final (heuristic) solution
ℓ1-norm methods for convex-cardinality problems 20
![Page 22: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/22.jpg)
Interpretation as convex relaxation
• start with
minimize card(x)subject to x ∈ C, ‖x‖∞ ≤ R
• equivalent to mixed Boolean convex problem
minimize 1Tzsubject to |xi| ≤ Rzi, i = 1, . . . , n
x ∈ C, zi ∈ {0, 1}, i = 1, . . . , n
with variables x, z
ℓ1-norm methods for convex-cardinality problems 21
![Page 23: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/23.jpg)
• now relax zi ∈ {0, 1} to zi ∈ [0, 1] to obtain
minimize 1Tzsubject to |xi| ≤ Rzi, i = 1, . . . , n
x ∈ C0 ≤ zi ≤ 1, i = 1, . . . , n
which is equivalent to
minimize (1/R)‖x‖1subject to x ∈ C
the ℓ1 heuristic
• optimal value of this problem is lower bound on original problem
ℓ1-norm methods for convex-cardinality problems 22
![Page 24: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/24.jpg)
Interpretation via convex envelope
• convex envelope f env of a function f on set C is the largest convexfunction that is an underestimator of f on C
• epi(f env) = Co(epi(f))
• f env = (f∗)∗ (with some technical conditions)
• for x scalar, |x| is the convex envelope of card(x) on [−1, 1]
• for x ∈ Rn, (1/R)‖x‖1 is convex envelope of card(x) on{z | ‖z‖∞ ≤ R}
ℓ1-norm methods for convex-cardinality problems 23
![Page 25: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/25.jpg)
Weighted and asymmetric ℓ1 heuristics
• minimize card(x) over convex set C
• suppose we know lower and upper bounds on xi over C
x ∈ C =⇒ li ≤ xi ≤ ui
(best values for these can be found by solving 2n convex problems)
• if ui < 0 or li > 0, then card(xi) = 1 (i.e., xi 6= 0) for all x ∈ C
• assuming li < 0, ui > 0, convex relaxation and convex envelopeinterpretations suggest using
n∑
i=1
(
(xi)+ui
+(xi)−−li
)
as surrogate (and also lower bound) for card(x)
ℓ1-norm methods for convex-cardinality problems 24
![Page 26: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/26.jpg)
Regressor selection
minimize ‖Ax− b‖2subject to card(x) ≤ k
• heuristic:
– minimize ‖Ax− b‖2 + γ‖x‖1– find smallest value of γ that gives card(x) ≤ k– fix associated sparsity pattern (i.e., subset of selected regressors) and
find x that minimizes ‖Ax− b‖2
ℓ1-norm methods for convex-cardinality problems 25
![Page 27: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/27.jpg)
Example (6.4 in BV book)
• A ∈ R10×20, x ∈ R20, b ∈ R10
• dashed curve: exact optimal (via enumeration)
• solid curve: ℓ1 heuristic with polishing
0 1 2 3 40
2
4
6
8
10
‖Ax − b‖2
card(x
)
ℓ1-norm methods for convex-cardinality problems 26
![Page 28: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/28.jpg)
Sparse signal reconstruction
• convex-cardinality problem:
minimize ‖Ax− y‖2subject to card(x) ≤ k
• ℓ1 heuristic:minimize ‖Ax− y‖2subject to ‖x‖1 ≤ β
(called LASSO)
• another form: minimize ‖Ax− y‖2 + γ‖x‖1(called basis pursuit denoising)
ℓ1-norm methods for convex-cardinality problems 27
![Page 29: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/29.jpg)
Example
• signal x ∈ Rn with n = 1000, card(x) = 30
• m = 200 (random) noisy measurements: y = Ax+ v, v ∼ N (0, σ2I),Aij ∼ N (0, 1)
• left: original; right: ℓ1 reconstruction with γ = 10−3
100 200 300 400 500 600 700 800 900 1000
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
100 200 300 400 500 600 700 800 900 1000
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
ℓ1-norm methods for convex-cardinality problems 28
![Page 30: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/30.jpg)
• ℓ2 reconstruction; minimizes ‖Ax− y‖2 + γ‖x‖2, where γ = 10−3
• left: original; right: ℓ2 reconstruction
100 200 300 400 500 600 700 800 900 1000
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
100 200 300 400 500 600 700 800 900 1000
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
ℓ1-norm methods for convex-cardinality problems 29
![Page 31: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/31.jpg)
Some recent theoretical results
• suppose y = Ax, A ∈ Rm×n, card(x) ≤ k
• to reconstruct x, clearly need m ≥ k
• if m ≥ n and A is full rank, we can reconstruct x without cardinalityassumption
• when does the ℓ1 heuristic (minimizing ‖x‖1 subject to Ax = y)reconstruct x (exactly)?
ℓ1-norm methods for convex-cardinality problems 30
![Page 32: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/32.jpg)
recent results by Candes, Donoho, Romberg, Tao, . . .
• (for some choices of A) if m ≥ (C log n)k, ℓ1 heuristic reconstructs xexactly, with overwhelming probability
• C is absolute constant; valid A’s include
– Aij ∼ N (0, σ2)– Ax gives Fourier transform of x at m frequencies, chosen from
uniform distribution
ℓ1-norm methods for convex-cardinality problems 31
![Page 33: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/33.jpg)
Total variation reconstruction
• fit xcor with piecewise constant x, no more than k jumps
• convex-cardinality problem: minimize ‖x− xcor‖2 subject tocard(Dx) ≤ k (D is first order difference matrix)
• heuristic: minimize ‖x− xcor‖2 + γ‖Dx‖1; vary γ to adjust number ofjumps
• ‖Dx‖1 is total variation of signal x
• method is called total variation reconstruction
• unlike ℓ2 based reconstruction, TVR filters high frequency noise outwhile preserving sharp jumps
ℓ1-norm methods for convex-cardinality problems 32
![Page 34: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/34.jpg)
Example (§6.3.3 in BV book)
signal x ∈ R2000 and corrupted signal xcor ∈ R2000
0 500 1000 1500 2000−2
−1
0
1
2
0 500 1000 1500 2000−2
−1
0
1
2
i
xxcor
ℓ1-norm methods for convex-cardinality problems 33
![Page 35: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/35.jpg)
Total variation reconstruction
for three values of γ
0 500 1000 1500 2000−2
0
2
0 500 1000 1500 2000−2
0
2
0 500 1000 1500 2000−2
0
2
i
xx
x
ℓ1-norm methods for convex-cardinality problems 34
![Page 36: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/36.jpg)
ℓ2 reconstruction
for three values of γ
0 500 1000 1500 2000−2
0
2
0 500 1000 1500 2000−2
0
2
0 500 1000 1500 2000−2
0
2
i
xx
x
ℓ1-norm methods for convex-cardinality problems 35
![Page 37: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/37.jpg)
Example: 2D total variation reconstruction
• x ∈ Rn are values of pixels on N ×N grid (N = 31, so n = 961)
• assumption: x has relatively few big changes in value (i.e., boundaries)
• we have m = 120 linear measurements, y = Fx (Fij ∼ N (0, 1))
• as convex-cardinality problem:
minimize card(xi,j − xi+1,j) + card(xi,j − xi,j+1)subject to y = Fx
• ℓ1 heuristic (objective is a 2D version of total variation)
minimize∑
|xi,j − xi+1,j|+∑
|xi,j − xi,j+1|subject to y = Fx
ℓ1-norm methods for convex-cardinality problems 36
![Page 38: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/38.jpg)
TV reconstruction
510
1520
2530
0
5
10
15
20
25
30
35
−0.5
0
0.5
1
1.5
original
510
1520
2530
0
5
10
15
20
25
30
35
−0.5
0
0.5
1
1.5
TV reconstruction
. . . not bad for 8× more variables than measurements!
ℓ1-norm methods for convex-cardinality problems 37
![Page 39: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/39.jpg)
ℓ2 reconstruction
510
1520
2530
0
5
10
15
20
25
30
35
−0.5
0
0.5
1
1.5
original
510
1520
2530
0
5
10
15
20
25
30
35
−0.5
0
0.5
1
1.5
ℓ2 reconstruction
. . . this is what you’d expect with 8× more variables than measurements
ℓ1-norm methods for convex-cardinality problems 38
![Page 40: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/40.jpg)
Iterated weighted ℓ1 heuristic
• to minimize card(x) over x ∈ C
w := 1
repeatminimize ‖diag(w)x‖1 over x ∈ Cwi := 1/(ǫ+ |xi|)
• first iteration is basic ℓ1 heuristic
• increases relative weight on small xi
• typically converges in 5 or fewer steps
• often gives a modest improvement (i.e., reduction in card(x)) overbasic ℓ1 heuristic
ℓ1-norm methods for convex-cardinality problems 39
![Page 41: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/41.jpg)
Interpretation
• wlog we can take x � 0 (by writing x = x+ − x−, x+, x− � 0, andreplacing card(x) with card(x+) + card(x−))
• we’ll use approximation card(z) ≈ log(1 + z/ǫ), where ǫ > 0, z ∈ R+
• using this approximation, we get (nonconvex) problem
minimize∑n
i=1 log(1 + xi/ǫ)subject to x ∈ C, x � 0
• we’ll find a local solution by linearizing objective at current point,
n∑
i=1
log(1 + xi/ǫ) ≈n∑
i=1
log(1 + x(k)i /ǫ) +
n∑
i=1
xi − x(k)i
ǫ+ x(k)i
ℓ1-norm methods for convex-cardinality problems 40
![Page 42: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/42.jpg)
and solving resulting convex problem
minimize∑n
i=1wixi
subject to x ∈ C, x � 0
with wi = 1/(ǫ+ xi), to get next iterate
• repeat until convergence to get a local solution
ℓ1-norm methods for convex-cardinality problems 41
![Page 43: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/43.jpg)
Sparse solution of linear inequalities
• minimize card(x) over polyhedron {x | Ax � b}, A ∈ R100×50
• ℓ1 heuristic finds x ∈ R50 with card(x) = 44
• iterated weighted ℓ1 heuristic finds x with card(x) = 36(global solution, via branch & bound, is card(x) = 32)
1 2 3 4 5 60
10
20
30
40
50
iteration
card(x
)
iterated ℓ1ℓ1
ℓ1-norm methods for convex-cardinality problems 42
![Page 44: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/44.jpg)
Detecting changes in time series model
• AR(2) scalar time-series model
y(t+ 2) = a(t)y(t+ 1) + b(t)y(t) + v(t), v(t) IID N (0, 0.52)
• assumption: a(t) and b(t) are piecewise constant, change infrequently
• given y(t), t = 1, . . . , T , estimate a(t), b(t), t = 1, . . . , T − 2
• heuristic: minimize over variables a(t), b(t), t = 1, . . . , T − 1
∑T−2t=1 (y(t+ 2)− a(t)y(t+ 1)− b(t)y(t))2
+γ∑T−2
t=1 (|a(t+ 1)− a(t)|+ |b(t+ 1)− b(t)|)
• vary γ to trade off fit versus number of changes in a, b
ℓ1-norm methods for convex-cardinality problems 43
![Page 45: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/45.jpg)
Time series and true coefficients
0 50 100 150 200 250 300−3
−2
−1
0
1
2
3
t
y(t)
50 100 150 200 250 300
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
t
a(t)b(t)
ℓ1-norm methods for convex-cardinality problems 44
![Page 46: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/46.jpg)
TV heuristic and iterated TV heuristic
left: TV with γ = 10; right: iterated TV, 5 iterations, ǫ = 0.005
50 100 150 200 250 300
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
t50 100 150 200 250 300
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
t
ℓ1-norm methods for convex-cardinality problems 45
![Page 47: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/47.jpg)
Extension to matrices
• Rank is natural analog of card for matrices
• convex-rank problem: convex, except for Rank in objective orconstraints
• rank problem reduces to card problem when matrices are diagonal:Rank(diag(x)) = card(x)
• analog of ℓ1 heuristic: use nuclear norm, ‖X‖∗ =∑
i σi(X)(sum of singular values; dual of spectral norm)
• for X � 0, reduces to TrX (for x � 0, ‖x‖1 reduces to 1Tx)
ℓ1-norm methods for convex-cardinality problems 46
![Page 48: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/48.jpg)
Factor modeling
• given matrix Σ ∈ Sn+, find approximation of form Σ = FFT +D, where
F ∈ Rn×r, D is diagonal nonnegative
• gives underlying factor model (with r factors)
x = Fz + v, v ∼ N (0, D), z ∼ N (0, I)
• model with fewest factors:
minimize RankXsubject to X � 0, D � 0 diagonal
X +D ∈ C
with variables D, X ∈ Sn
C is convex set of acceptable approximations to Σ
ℓ1-norm methods for convex-cardinality problems 47
![Page 49: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/49.jpg)
• e.g., via KL divergence
C = {Σ | − log det(Σ−1/2ΣΣ−1/2) +Tr(Σ−1/2ΣΣ−1/2)− n ≤ ǫ}
• trace heuristic:
minimize TrXsubject to X � 0, D � 0 diagonal
X +D ∈ C
with variables d ∈ Rn, X ∈ Sn
ℓ1-norm methods for convex-cardinality problems 48
![Page 50: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/50.jpg)
Example
• x = Fz + v, z ∼ N (0, I), v ∼ N (0, D), D diagonal; F ∈ R20×3
• Σ is empirical covariance matrix from N = 3000 samples
• set of acceptable approximations
C = {Σ | ‖Σ−1/2(Σ− Σ)Σ−1/2‖ ≤ β}
• trace heuristic
minimize TrXsubject to X � 0, d � 0
‖Σ−1/2(X + diag(d)− Σ)Σ−1/2‖ ≤ β
ℓ1-norm methods for convex-cardinality problems 49
![Page 51: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/51.jpg)
Trace approximation results
10−2
10−1
100
2
4
6
8
10
12
14
16
β
Rank(X
)
10−2
10−1
100
10−4
10−2
100
102
β
λi(X)
ℓ1-norm methods for convex-cardinality problems 50
![Page 52: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/52.jpg)
• for β = 0.1357 (knee of the tradeoff curve) we find
– 6(
range(X), range(FFT ))
= 6.8◦
– ‖d− diag(D)‖/‖diag(D)‖ = 0.07
• i.e., we have recovered the factor model from the empirical covariance
ℓ1-norm methods for convex-cardinality problems 51
![Page 53: ℓ1-Norm Methods for Convex-Cardinality Problems · 1-norm heuristics for cardinality problems • cardinality problems arise often, but are hard to solve exactly • a simple heuristic,](https://reader034.vdocuments.net/reader034/viewer/2022051512/603975da78609809c4357eb5/html5/thumbnails/53.jpg)
Summary and conclusions
• convex-cardinality (and rank) problems arise in many applications
• these problems are hard (to solve exactly, in general)
• heuristics based on ℓ1 norm (or nuclear norm for rank)
– are convex, hence solvable– give very good results in practice
• is basis of many well known methods(lasso, SVM, compressed sensing, TV denoising, . . . )
ℓ1-norm methods for convex-cardinality problems 52