l14: orthogonal matching pursuit & lasso and compressed...
TRANSCRIPT
![Page 1: L14: Orthogonal Matching Pursuit & Lasso and Compressed ...jeffp/teaching/cs5140/L14-notes.pdf · L14: Orthogonal Matching Pursuit & Lasso and Compressed Sensing Je↵M. Phillips](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fe0cf12572ec90612709f8c/html5/thumbnails/1.jpg)
L14: Orthogonal Matching Pursuit & Lasso
and Compressed Sensing
Je↵ M. Phillips
February 26, 2020
![Page 2: L14: Orthogonal Matching Pursuit & Lasso and Compressed ...jeffp/teaching/cs5140/L14-notes.pdf · L14: Orthogonal Matching Pursuit & Lasso and Compressed Sensing Je↵M. Phillips](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fe0cf12572ec90612709f8c/html5/thumbnails/2.jpg)
Midterm• No computers ,
calculators, planes
• I ' ' cheat she - t" ( both sides )
• Understand core definitions3 Qeesttxon C sub parts)
• Clustering-
Assessment - bas .ee
- HACa
'
k - grams• Similarityo Distance a Streaming
.Mitha shrug
NG
n Jace card
![Page 3: L14: Orthogonal Matching Pursuit & Lasso and Compressed ...jeffp/teaching/cs5140/L14-notes.pdf · L14: Orthogonal Matching Pursuit & Lasso and Compressed Sensing Je↵M. Phillips](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fe0cf12572ec90612709f8c/html5/thumbnails/3.jpg)
Ridge & Lasso Regression-
laps IcyX EIR
'd
get" X - ft ;D
God : RR : go=arsmi "
II. ( yi- 24 ;D )2tsll*R
a end " '
if rest?
Lasso ids = argentinian II. Cgi - Gord 't shake
-
I = Inseminated:( gi - 2x ;D } sit .
Hallie
#kill !
* = LIFE , II. Csi - LAW set . Hallett
equivalent tf chooses s → If
so Go =L : I RR )Let t= 1108115
or 29=27 C Lasso )
![Page 4: L14: Orthogonal Matching Pursuit & Lasso and Compressed ...jeffp/teaching/cs5140/L14-notes.pdf · L14: Orthogonal Matching Pursuit & Lasso and Compressed Sensing Je↵M. Phillips](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fe0cf12572ec90612709f8c/html5/thumbnails/4.jpg)
It = I n II Csi - Hird set .
lkh÷÷÷i÷÷÷÷:* .
![Page 5: L14: Orthogonal Matching Pursuit & Lasso and Compressed ...jeffp/teaching/cs5140/L14-notes.pdf · L14: Orthogonal Matching Pursuit & Lasso and Compressed Sensing Je↵M. Phillips](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fe0cf12572ec90612709f8c/html5/thumbnails/5.jpg)
Lasso Illustration
Find ↵⇤= argmin↵2Rd kX↵ � yk2 + sk↵k1
k↵k2 = t
k↵k1 = t
t↵�
t with ridge regression
↵� with OLS
↵�t with lasso
Hall lie
←
:
![Page 6: L14: Orthogonal Matching Pursuit & Lasso and Compressed ...jeffp/teaching/cs5140/L14-notes.pdf · L14: Orthogonal Matching Pursuit & Lasso and Compressed Sensing Je↵M. Phillips](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fe0cf12572ec90612709f8c/html5/thumbnails/6.jpg)
Matching Pursuit (MP)
Find ↵⇤= argmin↵2Rd kX↵ � yk2 + sk↵k1
Forward Subset Selection:
Matching Pursuit
Set r = y ; ↵ = 0.for i = 1 to t doSet Xj = argmaxXj02X |hr ,Xj 0i|.Set ↵j = argmin↵ kr � Xj↵k + s|↵|.Set r = r � Xj↵j .
Return ↵.
%. .
es
°
![Page 7: L14: Orthogonal Matching Pursuit & Lasso and Compressed ...jeffp/teaching/cs5140/L14-notes.pdf · L14: Orthogonal Matching Pursuit & Lasso and Compressed Sensing Je↵M. Phillips](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fe0cf12572ec90612709f8c/html5/thumbnails/7.jpg)
Orthogonal Matching Pursuit (OMP)
Find ↵⇤= argmin↵2Rd kX↵ � yk2 + sk↵k22
Forward Subset Selection:
Orthogonal Matching Pursuit
Set r = y ; ↵ = 0.for i = 1 to t doSet Xj = argmaxXj02X |hr ,Xj 0i|.Set ↵ = argmin↵ kr � [X1;X2; . . . ;Xj ]↵k + sk↵k22.Set r = r � Xj↵j . (Update using other ↵j 0 for j 0 < j)
Return ↵.
Ridge
full !! efficient'
a
![Page 8: L14: Orthogonal Matching Pursuit & Lasso and Compressed ...jeffp/teaching/cs5140/L14-notes.pdf · L14: Orthogonal Matching Pursuit & Lasso and Compressed Sensing Je↵M. Phillips](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fe0cf12572ec90612709f8c/html5/thumbnails/8.jpg)
Lasso Illustration
Find ↵⇤= argmin↵2Rd kX↵ � yk2 + sk↵k1
k↵k2 = t
k↵k1 = t
t↵�
t with ridge regression
↵� with OLS
↵�t with lasso
( east Angle Regression
µMP
i nst
¥D\÷*a. . . .¥i.on ,
![Page 9: L14: Orthogonal Matching Pursuit & Lasso and Compressed ...jeffp/teaching/cs5140/L14-notes.pdf · L14: Orthogonal Matching Pursuit & Lasso and Compressed Sensing Je↵M. Phillips](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fe0cf12572ec90612709f8c/html5/thumbnails/9.jpg)
Sparse Sensing
ST = [0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 1 1 0 1 0 0 0 1 0 0 1 0 0]
xTi = [-1 0 1 0 1 1 -1 1 0 -1 0 0 1 -1 -1 1 0 1 0 1 -1 -1 -1 0 1 0 0 -1 0 1 0 0]
yi = hS , xi i= 0+0+0+0+1+0+0+0+0+0+0+0+0+0+0+1+0+0+0+1-1+0-1+0+0+0+0+0+0+1+0+0
= 2
what if d > n
but k ? n
keenLos I ( unknown )
ksnpasse C
-44279galbiddies ,
ord
"" " " "
I:*.
.
. " "
![Page 10: L14: Orthogonal Matching Pursuit & Lasso and Compressed ...jeffp/teaching/cs5140/L14-notes.pdf · L14: Orthogonal Matching Pursuit & Lasso and Compressed Sensing Je↵M. Phillips](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fe0cf12572ec90612709f8c/html5/thumbnails/10.jpg)
Matching Pursuit (OMP)
Matching Pursuit
Set r = y .for i = 1 to t doSet Xj = argmaxXj02X |hr ,Xj 0i|.Set �j = argmin� kr � Xj�k.Set r = r � Xj�j .
Return S where sj = �j (or 0).
![Page 11: L14: Orthogonal Matching Pursuit & Lasso and Compressed ...jeffp/teaching/cs5140/L14-notes.pdf · L14: Orthogonal Matching Pursuit & Lasso and Compressed Sensing Je↵M. Phillips](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fe0cf12572ec90612709f8c/html5/thumbnails/11.jpg)
OMP Example
signal: S = [0, 0, 1, 0, 0, 1, 0, 0, 1, 0]
measurement: X =
2
666664
0 1 1 �1 �1 0 �1 0 �1 0�1 �1 0 1 �1 0 0 �1 0 11 �1 1 �1 0 �1 1 1 0 01 0 �1 0 0 1 �1 �1 1 1�1 0 0 0 1 0 1 0 1 �10 0 �1 �1 �1 0 �1 1 �1 0
3
777775
observation: y = XST = [0, 0, 0, 1, 1,�2]T000
⑧
r,
= £ I,
o,
0, 0,0 ,
- I ]