lecture 17 sparse convex optimization...lecture 17 sparse convex optimization texpoint fonts used in...
TRANSCRIPT
![Page 1: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/1.jpg)
Lecture 17 Sparse Convex Optimization
TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAA
![Page 2: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/2.jpg)
Compressed sensing
![Page 3: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/3.jpg)
A short introduction to Compressed Sensing
• An imaging perspective
• Image compression Scene Picture
10 Mega Pixels
Why do we compress images?
![Page 4: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/4.jpg)
Introduction to Compressed Sensing
• Images are compressible
• Image compression – Take an input image u
– Pick a good dictionary ©
– Find a sparse representation x of u such that ||©x - u||2 is small – Save x
Because • Only certain part of information is
important (e.g. objects and their edges)
• Some information is unwanted (e.g. noise)
This is traditional compression.
![Page 5: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/5.jpg)
Introduction to Compressed Sensing
• An imaging perspective
This is traditional compression.
10 Mega Pixels
100 Kilobytes
![Page 6: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/6.jpg)
Introduction to Compressed Sensing
• An imaging perspective
This is traditional compression.
n: 10 Mega Pixels
k:100 Kilobytes
![Page 7: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/7.jpg)
Introduction to Compressed Sensing
• If only 100 kilobytes are saved, why do we need a 10-megapixel camera in the first place?
• Answer: a traditional compression algorithm needs the complete image to compute © and x
• Can we do better than this?
![Page 8: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/8.jpg)
Introduction to Compressed Sensing
• Let k=||x||0, n=dim(x)=dim(u).
• In compressed sensing based on l1 minimization, the number of measurements is m=O(k log(n/k)) (Donoho, Candés-Tao)
![Page 9: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/9.jpg)
Introduction to Compressed Sensing
Input Linear encoding
Signal acquisition
Signal reconstruction
Signal representation
![Page 10: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/10.jpg)
Introduction to Compressed Sensing
Input Linear encoding
Signal acquisition
Signal reconstruction
Signal representation
![Page 11: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/11.jpg)
Introduction to Compressed Sensing
• Input: b=Bu=B©x, A=B©
• Output: x
• In compressed sensing, m=dim(b)<<dim(u)=dim(x)=n
• Therefore, Ax = b is an underdetermined system
• Approaches for recovering x (hence the image u): – Solve min ||x||0, subject to Ax = b – Solve min ||x||1, subject to Ax = b – Other approaches
![Page 12: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/12.jpg)
Difficulties
• Large scales
• Completely dense data: A However
• Solutions x are expected to be sparse
• The matrices A are often fast transforms
![Page 13: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/13.jpg)
Recovery by using the l1-norm
Sparse signal reconstruction
Sparse signal , matrix The system is underdetermined, but if card(x)<m, can recover signal. The problem is NP-hard in general. Typical relaxation,
![Page 14: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/14.jpg)
Signal recovery
• Shown by Candes & Tao and Donoho that under certain conditions on matrix A the sparse signal
is recovered exactly by solving the convex relaxation
• The matrix property is called “restricted isometry property”
![Page 15: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/15.jpg)
Restricted Isometry Property
• A vector is said to be s-sparse if it has at most s nonzero entries. • For a given s the isometric constant of a matrix A is the smallest
constant such that
• for any s-sparse .
![Page 16: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/16.jpg)
Why ||·||1 norm?
`1 ball
`2 ball
`0 ball
Ax= b
![Page 17: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/17.jpg)
Why ||·||1 norm?
Ax= b
![Page 18: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/18.jpg)
Why ||·||1 norm?
Ax= b
![Page 19: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/19.jpg)
Sparse regularized regression
![Page 20: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/20.jpg)
Least Squares Linear Regression
![Page 21: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/21.jpg)
Least squares problem
Standard form of LS problem
Includes solution of a system of linear equations Ax=b. May be used with additional linear constraints, e.g. Ridge regression ¸ is the regularization parameter – the trade-off weight.
![Page 22: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/22.jpg)
Robust least squares regression
Less straightforward than for SVM but it is possible to show that the above problem leads to
Another interpretation – feature selection
![Page 23: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/23.jpg)
Lasso and other formulations
Sparse regularized regression or Lasso: Sparse regressor selection Noisy signal recovery
![Page 24: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/24.jpg)
Connection between different formulations
If solution exists
![Page 25: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/25.jpg)
![Page 26: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/26.jpg)
![Page 27: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/27.jpg)
Types of convex problems
Linear programming problem
Variable substitution:
![Page 28: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/28.jpg)
Types of convex problems
Convex non-smooth objective with linear inequality constraints
Variable substitution:
![Page 29: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/29.jpg)
Types of convex problems
Convex QP with linear inequality constraints SOCP
![Page 30: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/30.jpg)
Optimization approaches
![Page 31: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/31.jpg)
Lasso
Regularized regression or Lasso:
Convex QP with nonnegativity constraints
![Page 32: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/32.jpg)
Standard QP formulation
Reformulate as How is it different from SVMs dual QP?
![Page 33: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/33.jpg)
Standard QP formulation
Reformulate as
![Page 34: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/34.jpg)
Standard QP formulation
Reformulate as Features of this QP
1. Q=MTM, where M is m£n, with n>>m. 2. Forming Q is O(m2n), factorizing Q+D is O(m3) 3. There are no upper bound constraints.
IPM complexity is O(m3) per iteration
![Page 35: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/35.jpg)
Dual Problem
![Page 36: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/36.jpg)
Dual Problem
Using:
![Page 37: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/37.jpg)
Lasso
Primal-Dual pair of problems
![Page 38: Lecture 17 Sparse Convex Optimization...Lecture 17 Sparse Convex Optimization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAIntroduction](https://reader036.vdocuments.net/reader036/viewer/2022071505/61264066ac113b4e7c3d2950/html5/thumbnails/38.jpg)
Optimality Conditions