new low rank approximation with entrywise 1-norm error - ibm · 2017. 2. 16. · opt ba - a 1...

Low Rank Approximation withEntrywise `1-Norm Error

Zhao Song, David P. Woodruff, Peilin Zhong

UT-Austin, IBM Almaden, Columbia University

. .

. .

Song-Woodruff-Zhong Low Rank Approximation with Entrywise `1-Norm Error 1 / 31

`1-Low Rank Approximations. .

. .Song-Woodruff-Zhong Low Rank Approximation with Entrywise `1-Norm Error 2 / 31


. .

Given : A ∈ Rn×d , n > d , k ∈ N, α > 1



. .

Given : A ∈ Rn×d , n > d , k ∈ N, α > 1

Output :



. .

Given : A ∈ Rn×d , n > d , k ∈ N, α > 1

Output : rank-k A ∈ Rn×d s.t.

A A−

1



. .

Given : A ∈ Rn×d , n > d , k ∈ N, α > 1

Output : rank-k A ∈ Rn×d s.t.‖A − A‖1 6 α · min

rank−k A ′‖A ′ − A‖1

A A−

1



. .

Given : A ∈ Rn×d , n > d , k ∈ N, α > 1



.

∑i,j|A ′i,j − Ai,j |

A A−

1



. .

Given : A ∈ Rn×d , n > d , k ∈ N, α > 1



.


.OPT

A A−

1



. .

Given : A ∈ Rn×d , n > d , k ∈ N, α > 1

Output :

..

U ∈ Rn×k ,V ∈ Rk×d s.t.‖UV − A‖1 6 α · OPT

U

V

A−

1


Why is `1-low Rank Approximation Interesting?

It is more robust than the Frobenius norm in the presence ofoutliers

It is indicated in models where Gaussian assumptions on thenoise may not applyThe problem was shown to be NP-hard by Gillis-Vavasis’15 and anumber of heuristics have been proposedIt was asked in multiple places if there are any approximationalgorithms



It is more robust than the Frobenius norm in the presence ofoutliersIt is indicated in models where Gaussian assumptions on thenoise may not apply

The problem was shown to be NP-hard by Gillis-Vavasis’15 and anumber of heuristics have been proposedIt was asked in multiple places if there are any approximationalgorithms



It is more robust than the Frobenius norm in the presence ofoutliersIt is indicated in models where Gaussian assumptions on thenoise may not applyThe problem was shown to be NP-hard by Gillis-Vavasis’15 and anumber of heuristics have been proposed

It was asked in multiple places if there are any approximationalgorithms



It is more robust than the Frobenius norm in the presence ofoutliersIt is indicated in models where Gaussian assumptions on thenoise may not applyThe problem was shown to be NP-hard by Gillis-Vavasis’15 and anumber of heuristics have been proposedIt was asked in multiple places if there are any approximationalgorithms


Visualization of `p-norm. .


Visualization of `p-norm. .

. .

p = ∞ p = 2 2 > p > 1 p = 1 1 > p > 0 p = 0


Thought Experiments. .



. .

Given : A set of points in 2-dimension



. .


Goal : Find a line to fit those points



. .



.



. .



.

Output : `2 minimizer line,



. .



.

Output : `2 minimizer line, `1 minimizer line



. .

Given : k = 1 and A =

4 0 0 00 1 1 10 1 1 10 1 1 1



. .


4 0 0 00 1 1 10 1 1 10 1 1 1

Output : rank-1 ‖‖F−solution A =

4 0 0 00 0 0 00 0 0 00 0 0 0



. .


4 0 0 00 1 1 10 1 1 10 1 1 1

Output : rank-1 ‖‖F−solution A =

4 0 0 00 0 0 00 0 0 00 0 0 0

Output : rank-1 ‖‖1−solution A =

0 0 0 00 1 1 10 1 1 10 1 1 1


Real-life Applications and Datasets

Recover image with outliers3D model reconstruction from a sequence of 2D imagesBackground modeling from surveillance videoMore applications

. .

. .


Real-life Applications and DatasetsRecover image with outliers

3D model reconstruction from a sequence of 2D imagesBackground modeling from surveillance videoMore applications

. .

. .

2D Image


Real-life Applications and DatasetsRecover image with outliers3D model reconstruction from a sequence of 2D images

Background modeling from surveillance videoMore applications

. .

. .

2D Image

3DReconstruction


Real-life Applications and DatasetsRecover image with outliers3D model reconstruction from a sequence of 2D imagesBackground modeling from surveillance video

More applications

. .

. .

2D Image

3DReconstruction

Video


Real-life Applications and DatasetsRecover image with outliers3D model reconstruction from a sequence of 2D imagesBackground modeling from surveillance videoMore applications

. .

. .

2D Image

3DReconstruction

Video

Milk,glasshandwritten


Other `1 Problems

Linear regression Clarkson’05, Sohler-W’11,Clarkson-Drineas-Magon-Ismail-Mahoney-Meng-W’13,Clarkson-W’13, Li-Miller-Peng’13 W-Zhang’13,Mahoney-Meng’13, Cohen-Peng’15, Yang-Chow-Re-Mahoney’16

I Given A,b, solve minx ‖Ax − b‖1

Sparse recovery Gormode-Muthukrishnan’04,Gilbert-Strauss-Tropp-Vershynin’06, Indyk-Ruzic’08,Berinde-Indyk-Ruzic’08, Berinde-Gilbert-Indyk-Harloff-Strauss’08,Do Ba-Indyk-Price-W’10, Nelson-Nguyen-W’12,Bhattacharyya-Nair’14, Osher-Yin’14

I Given Ax , find x s.t. ‖x − x‖1 6 C mink−sparse x ′ ‖x ′ − x‖1

Regularizers for sparsity(Lasso) Tibshirani’96, Breiman’95,Tibshirani’97

I Given A,b, solve minx ‖Ax − b‖22 + λ‖x‖1


Previous Work

Non `1-setting, Low rank approximation and related

I Frobenious norm Sarlos’06, Mahoney-Meng’13, Li-Miller-Peng’13Clarkson-W’09, Clarkson-W’13, Nelson-Nguyen’13,Bourgain-Dirksen-Nelson’15, Cohen’16

I Spectral norm W’14, Cohen-Lee-Musco-Musco-Peng-Sidford’15,Cohen-Elder-Musco-Musco-Persu’15

I Weighted low rank approximation Srebro-Jaakkola’03,Gillis-Glineur’11, Razenshteyn-Song-W’16, Li-Liang-Risteski’16

I Matrix completion Candes-Recht’09, Candes-Recht’10,Keshavan’12, Hardt-Wootters’14, Jain-Netrapalli-Sanghavi’13,Hardt’15, Sun-Luo’15, Peeters’96, Gillis-Glineur’11,Hardt-Meka-Raghavendra-Weitz’14


Previous Work

Non `1-setting, Low rank approximation and relatedI Frobenious norm Sarlos’06, Mahoney-Meng’13, Li-Miller-Peng’13

Clarkson-W’09, Clarkson-W’13, Nelson-Nguyen’13,Bourgain-Dirksen-Nelson’15, Cohen’16

I Spectral norm W’14, Cohen-Lee-Musco-Musco-Peng-Sidford’15,Cohen-Elder-Musco-Musco-Persu’15

I Weighted low rank approximation Srebro-Jaakkola’03,Gillis-Glineur’11, Razenshteyn-Song-W’16, Li-Liang-Risteski’16

I Matrix completion Candes-Recht’09, Candes-Recht’10,Keshavan’12, Hardt-Wootters’14, Jain-Netrapalli-Sanghavi’13,Hardt’15, Sun-Luo’15, Peeters’96, Gillis-Glineur’11,Hardt-Meka-Raghavendra-Weitz’14


Heuristics and Nearly `1 Setting`1-setting

I Heuristic algorithms Ke-Kanade’05, Ding-Zhou-He-Zha’06,Kwak’08, Brooks-Dula-Boone’13

I Robust PCA Candes-Li-Ma-Wright’11,Wright-Ganesh-Rao-Peng-Ma’09,Netrapalli-Niranjan-Sanghavi-Anandkumar-Jain’14,Yi-Park-Chen-Caramanis’16

. .





. .

. .

.





. .

. .

.For any β ∈ (0,0.5) and γ > 0, k = 3





. .

. .

.For any β ∈ (0,0.5) and γ > 0, k = 3Construct A ∈ R(2n+2)×(2n+2) as follows





. .

. .


A =

n2+γ 0 0 0

0 n1.5+β 0 00 0 B 00 0 0 B

where B ∈ Rn×n is all 1s





. .

. .


A =

n2+γ 0 0 0

0 n1.5+β 0 00 0 B 00 0 0 B

where B ∈ Rn×n is all 1s

Then any of them is not able to achievebetter than α = nmin(γ,0.5−β) approximation ratio!


Main Question

QuestionGiven matrix A ∈ Rn×d , is there a provable aglorithm that is able tooutput a (factorization of a) rank-k matrix A such that

‖A − A‖1 6 α · minrank−k A ′

‖A ′ − A‖1?

(From the other perspective) Is there any inapproximability hardness?

This question has been asked at least 4 different places.

I Kannan-Vempala’09I Stack Exchange’13I W’14I Gillis-Vavasis’15I Some machine learning, computer vision papers

Very recent, independent work,Chierichetti-Gollapudi-Kumar-Lattanzi-Panigrahy’16


Main Question

QuestionGiven matrix A ∈ Rn×d , is there a provable aglorithm that is able tooutput a (factorization of a) rank-k matrix A such that

‖A − A‖1 6 α · minrank−k A ′

‖A ′ − A‖1?

(From the other perspective) Is there any inapproximability hardness?

This question has been asked at least 4 different places.I Kannan-Vempala’09I Stack Exchange’13I W’14I Gillis-Vavasis’15I Some machine learning, computer vision papers

Very recent, independent work,Chierichetti-Gollapudi-Kumar-Lattanzi-Panigrahy’16


Main Results - Algorithms

Upper bound (Algorithm)

I poly(k , log n)-approximation algorithm for an arbitrary n × d matrixA

F Polynomial time

I poly(k) log d-approximation algorithm for an arbitrary n× d matrix A

F Polynomial time

I O(k)-approximation algorithm for an arbitrary n × d matrix A

F Exponential in k running time.

I CUR decomposition algorithm for an arbitrary n × d matrix A

F C has O(k log k) columns from AF R has O(k log k) rows from AF rank-k UF poly(k) log d-approximation, polynomial running time

I Bicriteria algorithm, rank-2k O(1)-approximation algorithm



Upper bound (Algorithm)I poly(k , log n)-approximation algorithm for an arbitrary n × d matrix

A

F Polynomial timeI poly(k) log d-approximation algorithm for an arbitrary n× d matrix A

F Polynomial time









AF Polynomial time


F Polynomial time









AF Polynomial time


F Polynomial timeI O(k)-approximation algorithm for an arbitrary n × d matrix A








AF Polynomial time


F Polynomial time









AF Polynomial time



F Exponential in k running time.I CUR decomposition algorithm for an arbitrary n × d matrix A






AF Polynomial time










AF Polynomial time




F C has O(k log k) columns from AF R has O(k log k) rows from AF rank-k U

F poly(k) log d-approximation, polynomial running timeI Bicriteria algorithm, rank-2k O(1)-approximation algorithm




AF Polynomial time







Main Results - ApplicationsFor any p ∈ (1,2), `p-low rank approximation

Eath-mover-distance, EMD-low rank approximation`1-low rank approximation with limited independent variablesStreaming settingDistributed setting

. .


Main Results - ApplicationsFor any p ∈ (1,2), `p-low rank approximationEath-mover-distance, EMD-low rank approximation

`1-low rank approximation with limited independent variablesStreaming settingDistributed setting

. .

. .

.


Main Results - ApplicationsFor any p ∈ (1,2), `p-low rank approximationEath-mover-distance, EMD-low rank approximation`1-low rank approximation with limited independent variables

Streaming settingDistributed setting

. .

. .

..


Main Results - ApplicationsFor any p ∈ (1,2), `p-low rank approximationEath-mover-distance, EMD-low rank approximation`1-low rank approximation with limited independent variablesStreaming setting

Distributed setting

. .

. .

...

Receive data stream (i , j ,∆):

(3,4,61), (1,7,23), (5,2,44), (2,3,16), (4,1,98), (2,8,54), · · · · · ·


Main Results - ApplicationsFor any p ∈ (1,2), `p-low rank approximationEath-mover-distance, EMD-low rank approximation`1-low rank approximation with limited independent variablesStreaming settingDistributed setting

. .

. .

...


Input Sparsity Time Algorithm. .



. .

Given : A ∈ Rn×d , k ∈ N



. .


Output :



. .


Output : matrices U ∈ Rn×k V ∈ Rk×d s.t.



. .


Output : matrices U ∈ Rn×k V ∈ Rk×d s.t.‖UV − A‖1 6 poly(k) log d · min


U

V

A−

1



. .


Output : matrices U ∈ Rn×k V ∈ Rk×d s.t.‖UV − A‖1 6 poly(k) log d · min

rank−k A ′

.


U

V

A−

1



. .


Output : matrices U ∈ Rn×k V ∈ Rk×d s.t.‖UV − A‖1 6 poly(k) log d ·

..

OPT

U

V

A−

1



. .



..

OPTwith prob. 9/10

U

V

A−

1



. .



..

OPTwith prob. 9/10in nnz(A) + (n + d) poly(k) time

U

V

A−

1


O(k)-Approximation Algorithm. .



. .




. .


Output :



. .


Output : matrices U ∈ Rn×k V ∈ Rk×d s.t.



. .


Output : matrices U ∈ Rn×k V ∈ Rk×d s.t.‖UV − A‖1 6 O(k) · min


U

V

A−

1



. .


Output : matrices U ∈ Rn×k V ∈ Rk×d s.t.‖UV − A‖1 6 O(k) · min

rank−k A ′

.


U

V

A−

1



. .


Output : matrices U ∈ Rn×k V ∈ Rk×d s.t.‖UV − A‖1 6 O(k) ·

..

OPT

U

V

A−

1



. .



..

OPTwith prob. 9/10

U

V

A−

1



. .



..

OPTwith prob. 9/10in poly(n)d O(k)2O(k2) time

U

V

A−

1


CUR Decomposition Algorithm. .



. .




. .


Output : C ∈ Rn×s, rank-k U ∈ Rs×r , R ∈ Rr×d s.t.

C

U R

A−

1



. .


Output : C ∈ Rn×s, rank-k U ∈ Rs×r , R ∈ Rr×d s.t.‖CUR − A‖1 6 poly(k) log d · min


C

U R

A−

1



. .




with prob. 9/10

C

U R

A−

1



. .




with prob. 9/10in nnz(A) + (n + d) poly(k) time

C

U R

A−

1



. .




with prob. 9/10in nnz(A) + (n + d) poly(k) timeC has s = O(k) columns from AR has r = O(k) rows from A

C

U R

A−

1


Bicriteria O(1)-Approximation Algorithm. .



. .




. .


Output : matrices U ∈ Rn×2k V ∈ R2k×d s.t.

U

V

A−

1



. .


Output : matrices U ∈ Rn×2k V ∈ R2k×d s.t.‖UV − A‖1 6 O(1) · min


U

V

A−

1



. .




with prob. 9/10

U

V

A−

1



. .




with prob. 9/10in (nd)O(k2) time

U

V

A−

1


AlgorithmChoose sketching matrix S ′ (a Cauchy matrix or sparse Cauchymatrix.)

Compute S ′A, form C by C i ← arg minx ‖xS ′A − Ai‖1. FormB = C · S ′A.Choose sketching matrices T1,R,S,T2 (Cauchy matrices orsparse Cauchy matrices.)Solve minX ,Y ‖T1BRXYSBT2 − T1BT2‖FOutput BRX , YSB

. .


AlgorithmChoose sketching matrix S ′ (a Cauchy matrix or sparse Cauchymatrix.)Compute S ′A, form C by C i ← arg minx ‖xS ′A − Ai‖1. FormB = C · S ′A.

Choose sketching matrices T1,R,S,T2 (Cauchy matrices orsparse Cauchy matrices.)Solve minX ,Y ‖T1BRXYSBT2 − T1BT2‖FOutput BRX , YSB

. .

. .

.


AlgorithmChoose sketching matrix S ′ (a Cauchy matrix or sparse Cauchymatrix.)Compute S ′A, form C by C i ← arg minx ‖xS ′A − Ai‖1. FormB = C · S ′A.Choose sketching matrices T1,R,S,T2 (Cauchy matrices orsparse Cauchy matrices.)

Solve minX ,Y ‖T1BRXYSBT2 − T1BT2‖FOutput BRX , YSB

. .

. .

..


AlgorithmChoose sketching matrix S ′ (a Cauchy matrix or sparse Cauchymatrix.)Compute S ′A, form C by C i ← arg minx ‖xS ′A − Ai‖1. FormB = C · S ′A.Choose sketching matrices T1,R,S,T2 (Cauchy matrices orsparse Cauchy matrices.)Solve minX ,Y ‖T1BRXYSBT2 − T1BT2‖F

Output BRX , YSB

. .

. .

...

T1 B R X Y S B T2 T1 B T2−

F


AlgorithmChoose sketching matrix S ′ (a Cauchy matrix or sparse Cauchymatrix.)Compute S ′A, form C by C i ← arg minx ‖xS ′A − Ai‖1. FormB = C · S ′A.Choose sketching matrices T1,R,S,T2 (Cauchy matrices orsparse Cauchy matrices.)Solve minX ,Y ‖T1BRXYSBT2 − T1BT2‖FOutput BRX , YSB

. .

. .

....

T1 B R X Y S B T2 T1 B T2−

F


Main Ideas of Meta Algorithm. .



. .

1. Multiple regression sketch



. .


2. No closed-form for `1, but we can use `2 relaxation



. .



3. There exists a solution in the row span of A



. .




4. Obtain B by projecting rows of A onto SA



. .





5. Repeatedly apply multiple regression sketch



. .





5. Repeatedly apply multiple regression sketch

6. A low rank approximation to B givesa low rank approximation to A


Multiple Regression Sketch. .



. .

Given : matrix A ∈ Rn×d ,



. .

Given : matrix A ∈ Rn×d ,Choose : sketching matrix S ∈ Rm×n



. .


Define : U∗,V ∗ = arg minU,V‖UV − A‖1



. .


Define : U∗,V ∗ = arg minU,V‖UV − A‖1V ′ = arg minV ‖SU∗V − SA‖1

SU∗

V SA−

1



. .



If : with prob. 9/10for all V , ‖SU∗V − SA‖1 > ‖U∗V − A‖1 and

SU∗

V SA−

1



. .



If : with prob. 9/10for all V , ‖SU∗V − SA‖1 > ‖U∗V − A‖1 andfor any fixed V , ‖SU∗V − SA‖1 6 β‖U∗V − A‖1

SU∗

V SA−

1



. .



If : with prob. 9/10for all V , ‖SU∗V − SA‖1 > ‖U∗V − A‖1 andfor any fixed V , ‖SU∗V − SA‖1 6 β‖U∗V − A‖1

Then : with prob. 9/10,‖U∗V ′ − A‖1 6 β‖U∗V ∗ − A‖1

SU∗

V SA−

1


Existence Result. .


Existence Result. .

. .

Given :


Existence Result. .

. .

Given : matrix A ∈ Rn×d ,


Existence Result. .

. .

Given : matrix A ∈ Rn×d ,Choose :


Existence Result. .

. .



Existence Result. .

. .


Define :


Existence Result. .

. .


Define : U∗,V ∗ := arg minU∈Rn×k ,V∈Rk×d‖UV − A‖1


Existence Result. .

. .


Define : U∗,V ∗ := arg minU∈Rn×k ,V∈Rk×d‖UV − A‖1V i := arg minV ‖SU∗V i − SAi‖2, for all i ∈ [d ]


Existence Result. .

. .



=⇒ V i = (SU∗)†SAi ,


Existence Result. .

. .



=⇒ V i = (SU∗)†SAi , =⇒ V = (SU∗)†SA


Existence Result. .

. .



=⇒ V i = (SU∗)†SAi , =⇒ V = (SU∗)†SAV ′ := arg minV ‖SU∗V − SA‖1


Existence Result. .

. .




Then :


Existence Result. .

. .




Then : ‖SU∗V − SA‖1 6√

m‖SU∗V ′ − SA‖1 (`2-relaxation)


Existence Result. .

. .






If :


Existence Result. .

. .






If : ‖U∗V ′ − A‖1 6 β‖U∗V ∗ − A‖1 (proved earlier)


Existence Result. .

. .







Then :


Existence Result. .

. .







Then : ‖U∗V − A‖1 6√

mβ‖U∗V ∗ − A‖1


Existence Result. .

. .







Then : ‖U∗V − A‖1 6√

mβ‖U∗V ∗ − A‖1

Thus, there exists a√

mβ-approximation in the row span of A.


Repeatedly Apply Multiple Regression Sketch. .



. .

Given :



. .

Given : rank-t B ∈ Rn×d , k > 1, t = poly(k)



. .

Given : rank-t B ∈ Rn×d , k > 1, t = poly(k)s, r , t1, t2 = poly(k)



. .


Choose :



. .


Choose : sketching matrices S ∈ Rs×n, R ∈ Rd×r



. .



sketching matrix T1 ∈ Rt1×n, T2 ∈ Rd×t2



. .




By S,R :



. .




By S,R : minX∈Rr×k ,Y∈Rk×s

‖BRXYSB − B‖1 6 β√

m minrank−k B ′

‖B ′ − B‖1



. .






m minrank−k B ′

‖B ′ − B‖1

By T1,T2:



. .






m minrank−k B ′

‖B ′ − B‖1

By T1,T2: ‖BRX ∗Y ∗SB − B‖1 6 γ minX∈Rr×k ,Y∈Rk×s

‖BRXYSB − B‖1



. .






m minrank−k B ′

‖B ′ − B‖1


‖BRXYSB − B‖1

where X ∗,Y ∗ := arg minX∈Rr×k ,Y∈Rk×s

‖T1BRXYSBT2 − T1BT2‖1 (∗)



. .






m minrank−k B ′

‖B ′ − B‖1


‖BRXYSB − B‖1



Thus, BRX∗,Y ∗SB gives a βγ√

m-approximation to B.



. .






m minrank−k B ′

‖B ′ − B‖1


‖BRXYSB − B‖1





We can solve (∗) by either using polynomial system solver



. .






m minrank−k B ′

‖B ′ − B‖1


‖BRXYSB − B‖1





We can solve (∗) by either using polynomial system solveror relaxing ‖ · ‖1 to ‖ · ‖F .


Reducing Arbitrary Matrix A to Rank-r Matrix B. .



. .

Given k ∈ N , r > k , A ∈ Rn×d and rank-r B,C ∈ Rn×d .



. .


If ‖B − A‖1 6 β minrank−k A′

‖A ′ − A‖1 and ‖C − B‖1 6 γ minrank−k B ′

‖B ′ − B‖1,



. .




‖B ′ − B‖1,

then ‖C − A‖1 6 O(βγ) minrank−k A′

‖A ′ − A‖1.



. .




‖B ′ − B‖1,


‖A ′ − A‖1.

Proof :



. .




‖B ′ − B‖1,


‖A ′ − A‖1.

Proof : ‖C − A‖1 6 ‖C − B‖1 + ‖B − A‖1



. .




‖B ′ − B‖1,


‖A ′ − A‖1.

Proof : ‖C − A‖1 6 ‖C − B‖1 + ‖B − A‖1

( B∗ := arg minrank−k B ′

‖B ′ − B‖1)6 γ‖B∗ − B‖1 + ‖B − A‖1



. .




‖B ′ − B‖1,


‖A ′ − A‖1.

Proof : ‖C − A‖1 6 ‖C − B‖1 + ‖B − A‖1


‖B ′ − B‖1)6 γ‖B∗ − B‖1 + ‖B − A‖1

( A∗ := arg minrank−k A′

‖A ′ − A‖1)6 γ‖A∗ − B‖1 + ‖B − A‖1



. .




‖B ′ − B‖1,


‖A ′ − A‖1.

Proof : ‖C − A‖1 6 ‖C − B‖1 + ‖B − A‖1


‖B ′ − B‖1)6 γ‖B∗ − B‖1 + ‖B − A‖1


‖A ′ − A‖1)6 γ‖A∗ − B‖1 + ‖B − A‖1

6 γ‖A∗ − A‖1 + (γ+ 1)‖B − A‖1



. .




‖B ′ − B‖1,


‖A ′ − A‖1.

Proof : ‖C − A‖1 6 ‖C − B‖1 + ‖B − A‖1


‖B ′ − B‖1)6 γ‖B∗ − B‖1 + ‖B − A‖1


‖A ′ − A‖1)6 γ‖A∗ − B‖1 + ‖B − A‖1

6 γ‖A∗ − A‖1 + (γ+ 1)‖B − A‖1

6 O(βγ)‖A∗ − A‖1


Main Results - Summary of Hard Instance andHardness

Lower bound (hard instance, hardness)

I Under the Exponential Time Hypothesis(ETH)I Hard instance for row subset selectionI Hard instance for oblivious subspace embeddings(OSE)I Hard instance for Cauchy embeddingsI Hard instance for row span



Lower bound (hard instance, hardness)I Under the Exponential Time Hypothesis(ETH)

I Hard instance for row subset selectionI Hard instance for oblivious subspace embeddings(OSE)I Hard instance for Cauchy embeddingsI Hard instance for row span



Lower bound (hard instance, hardness)I Under the Exponential Time Hypothesis(ETH)I Hard instance for row subset selection

I Hard instance for oblivious subspace embeddings(OSE)I Hard instance for Cauchy embeddingsI Hard instance for row span



Lower bound (hard instance, hardness)I Under the Exponential Time Hypothesis(ETH)I Hard instance for row subset selectionI Hard instance for oblivious subspace embeddings(OSE)

I Hard instance for Cauchy embeddingsI Hard instance for row span



Lower bound (hard instance, hardness)I Under the Exponential Time Hypothesis(ETH)I Hard instance for row subset selectionI Hard instance for oblivious subspace embeddings(OSE)I Hard instance for Cauchy embeddings

I Hard instance for row span



Lower bound (hard instance, hardness)I Under the Exponential Time Hypothesis(ETH)I Hard instance for row subset selectionI Hard instance for oblivious subspace embeddings(OSE)I Hard instance for Cauchy embeddingsI Hard instance for row span


Hardness - Under Exponential Time Hypothesis. .



. .

`1-Low Rank Approximation Hardness



. .


Given : A ∈ Rn×d , k ∈ N, α > 1



. .


Given : A ∈ Rn×d , k ∈ N, α > 1

Output : U ∈ Rn×k , V ∈ Rk×d s.t. ‖UV − A‖2F 6 α ·OPTwith prob. 9/10



. .


Given : A ∈ Rn×d , k ∈ N, α > 1


Assume : Exponential Time Hypothesis(ETH)[Impagliazzo-Paturi-Zane’98]



. .


Given : A ∈ Rn×d , k ∈ N, α > 1



for arbitrarily small constant γ > 0



. .


Given : A ∈ Rn×d , k ∈ N, α > 1



for arbitrarily small constant γ > 0for any algorithm running in (nd)O(1) time



. .


Given : A ∈ Rn×d , k ∈ N, α > 1



for arbitrarily small constant γ > 0for any algorithm running in (nd)O(1) time

Requires : α > (1 + 1log1+γ(nd)

)


Hard Instance - Row Subset Selection. .



. .

No (O(√

k))-approximation for row subset selection



. .

No (O(√


Given : A ∈ Rn×(n+k), k ∈ N, n = poly(k)



. .

No (O(√



Output : rank-k matrix A is in the row span of any n/2 rows of A s.t.‖A − A‖1 6 O(k0.5−ε) ·OPT

with positive probability.ε > 0 is a constant which can be arbitrarily small



. .

No (O(√



Output : rank-k matrix A is in the row span of any n/2 rows of A s.t.‖A − A‖1 6 O(k0.5−ε) ·OPT

with positive probability.ε > 0 is a constant which can be arbitrarily small

There is no such algorithm!


Hard Instance - Oblivious Subspace Embeddings. .



. .

No (O(√

k))-approximation for any OSE



. .

No (O(√


Let k > 1, n = poly(k), t = poly(k).



. .

No (O(√


Let k > 1, n = poly(k), t = poly(k).There exist matrices A ∈ Rn×(k+n) s.t. for any oblivious matrix S ∈ Rt×n

U

S

A A−

1



. .

No (O(√


Let k > 1, n = poly(k), t = poly(k).There exist matrices A ∈ Rn×(k+n) s.t. for any oblivious matrix S ∈ Rt×n

with probability 9/10min

U∈Rd×t‖USA − A‖1 > Ω(k0.5−ε) min


ε > 0 is a constant which can be arbitrarily small

U

S

A A−

1


Hard Instance - Cauchy Embedding. .



. .

No (O(log d))-approximation in Row Span



. .


There exist matrices A ∈ Rd×d s.t. for any o(log d) > t > 1,



. .


There exist matrices A ∈ Rd×d s.t. for any o(log d) > t > 1,for random Cauchy matrices S ∈ Rt×d , where each entryis sampled from i.i.d. Cauchy distribution C(0,γ), γ ∈ R

U

S

A A−

1



. .


There exist matrices A ∈ Rd×d s.t. for any o(log d) > t > 1,for random Cauchy matrices S ∈ Rt×d , where each entryis sampled from i.i.d. Cauchy distribution C(0,γ), γ ∈ R

with probability 9/10min

U∈Rd×t‖USA − A‖1 > Ω( log d

(t log t)) minrank−k A ′

‖A ′ − A‖1

U

S

A A−

1


Hard Instance - Row Span. .



. .

No (2-ε)-approximation in the entire row span



. .


Given : A ∈ R(d−1)×d , k ∈ N



. .


Given : A ∈ R(d−1)×d , k ∈ N

Output : rank-k matrix A is in row span of A s.t.‖A − A‖1 6 2(1 − 1

Θ(d)) ·OPTwith prob. 9/10



. .


Given : A ∈ R(d−1)×d , k ∈ N






. .


Given : A ∈ R(d−1)×d , k ∈ N




A =

1 1 0 0 · · · 01 0 1 0 · · · 01 0 0 1 · · · 0· · · · · · · · · · · · · · · · · ·1 0 0 0 · · · 1


Experimental ResultsInput matrix A = diag(n2+0.25,n1.5+0.25,B,B) ∈ R(2n+2)×(2n+2),where B ∈ Rn×n is all 1s matrix. Find rank-3 solution.

The performance of our algorithm has the best accuracy.All the algorithms(including ours) can be finished in 3 seconds,except for BDB13, KK05.

. .


Experimental ResultsInput matrix A = diag(n2+0.25,n1.5+0.25,B,B) ∈ R(2n+2)×(2n+2),where B ∈ Rn×n is all 1s matrix. Find rank-3 solution.

The performance of our algorithm has the best accuracy.All the algorithms(including ours) can be finished in 3 seconds,except for BDB13, KK05.

. .

. .

0 50 100 150 200 250 300 350 4002n+2

0e+00

1e+04

2e+04

3e+04

4e+04

5e+04

6e+04

7e+04

8e+04

` 1-n

orm

cost

OursBDB13KK05rKK05sKwak08rKwak08sDZHZ06

‖A − A‖1 vs matrix dimension

0 50 100 150 200 250 300 350 4002n+2

0

10

20

30

40

50

Tim

e


Running time vs matrix dimension


Experimental ResultsInput matrix A = diag(n2+0.25,n1.5+0.25,B,B) ∈ R(2n+2)×(2n+2),where B ∈ Rn×n is all 1s matrix. Find rank-3 solution.The performance of our algorithm has the best accuracy.

All the algorithms(including ours) can be finished in 3 seconds,except for BDB13, KK05.

. .

. .

0 50 100 150 200 250 300 350 4002n+2

0e+00

1e+04

2e+04

3e+04

4e+04

5e+04

6e+04

7e+04

8e+04

` 1-n

orm

cost



0 50 100 150 200 250 300 350 4002n+2

0

10

20

30

40

50

Tim

e




Experimental ResultsInput matrix A = diag(n2+0.25,n1.5+0.25,B,B) ∈ R(2n+2)×(2n+2),where B ∈ Rn×n is all 1s matrix. Find rank-3 solution.The performance of our algorithm has the best accuracy.All the algorithms(including ours) can be finished in 3 seconds,except for BDB13, KK05.

. .

. .

0 50 100 150 200 250 300 350 4002n+2

0e+00

1e+04

2e+04

3e+04

4e+04

5e+04

6e+04

7e+04

8e+04

` 1-n

orm

cost



0 50 100 150 200 250 300 350 4002n+2

0

10

20

30

40

50

Tim

e




Thank you!. .

. .

Questions?


new low rank approximation with entrywise 1-norm error - ibm · 2017. 2. 16. · opt ba - a 1...

Documents