efficient computation of robust low-rank matrix ...yuxiangw/docs/wibergl1_slide.pdf · efficient...
TRANSCRIPT
![Page 1: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/1.jpg)
Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data using
the L1 Norm
Anders Eriksson, Anton van den Hengel
School of Computer Science
University of Adelaide, Austrailia
Presented by
Wang Yuxiang Department of Electrical and Computer Engineering,
National University of Singapore
![Page 2: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/2.jpg)
Outline • Introduction
• Previous work
– ALP & AQP
– Alternating Least Square
– Gauss Newton/Levenberg Marquadt
– Wiberg L2 Algorithm
• Proposed algorithm
– A detour to LP
– Wiberg L1 Algorithm
• Experiments
– Synthesized data
– The Dinosaur sequence
– Comparison with Rosper
• Conclusion 2
![Page 3: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/3.jpg)
Introduction: Problem formulation
• Low-rank matrix approximation in the
presence of missing data.
3
![Page 4: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/4.jpg)
Introduction:Background
• Many applications such as SfM, shape
from varying illumination (photometric
stereo).
• Simple SVD (ML minimization of L2 Norm)
is not applicable due to missing data.
• L2 norm is sensitive to outliers.
• L1 norm is used to reduce sensitivity to
outliers.
4
![Page 5: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/5.jpg)
Introduction: Advantage of L1 Nnorm
5
Extracted from [13]
![Page 6: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/6.jpg)
Introduction: Challenges of L1
• Non-smooth (hence non-differentiable)
• Non-convex due to rank constraint and
missing data. (Is this related to adoption of
L1 norm?)
• More computational demanding than L2
(we’ll see)
6
![Page 7: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/7.jpg)
Key contribution of this paper
• Extends Wiberg algoritm to L1 norm
• Proposed algorithm is more efficient than
other algorithms that deal with same
problem.
• Address non-smoothness and
computational requirement using LP.
7
![Page 8: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/8.jpg)
Previous Works
[5] Buchanan, a quantitative survey
[16] Okatani 岡谷, Reintroduce Wiberg to
Computer Vision (more to come)
[13] Ke&Kanade, ALP and AQP (state-of-art as
claimed, explained in next slide)
[9] De la Torre&Black a robust version of PCA
based on Huber distance.
[6] Chandraker&Kriegman, certain combinatorial
optimization method to reach global optimal.
8
![Page 9: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/9.jpg)
Previous Works: Ke & Kanade
• Ke&Kanade, Robust L1 norm factorization in the
presence of outliers and missing data by alternative
convex programming, CVPR2005
• Alternative Convex Programming
9
![Page 10: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/10.jpg)
Previous Works: Ke & Kanade
• Alternated Linear Programming:
10
![Page 11: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/11.jpg)
Previous Works : Ke & Kanade
• Alternated Quadratic Programming:
Huber Norm:
11
![Page 12: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/12.jpg)
Previous Works : Ke & Kanade
Remarks:
1. Handle missing data by dropping constraints in LP and
QP formulation.
2. Result in convex sub problem, but solution might not be
good for the original problem.
12
![Page 13: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/13.jpg)
Notations:
• Small case means “vec” operator.
• W=diag(Ŵ), so Wy means the masked
vec(Y).
• ⊗: Kronecker Product
13
![Page 14: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/14.jpg)
Property of Kronecker Product
• Assume V is fixed, U is unknown,
vec(UV)=vec(IUV)=(V’ ⊗ I)vec(U);
• Assume U is fixed and V is unknown,
vec(UV)=vec(UVI)=(I⊗U)vec(V);
![Page 15: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/15.jpg)
Alternated least square (ALS)
• So is equivalent to:
=||Wy-G(U)v||22
= ||Wy-F(V)u||22
f= Fu-Wy= Gv-Wy;
1/2fTf = Φ(U,V)
Take partial derivative of Φ and assign to 0, we get:
u = (FTF)-1FT (Wy); v = (GTG)-1GT (Wy);
15
![Page 16: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/16.jpg)
Gauss-Newton Method
• Let x=[uT vT], Φ(U,V) = 1/2fTf
• Newton’s method: To attain Φ/x=0 using the step ,
satisfying:
2Φ/x2 x+ Φ/x =0 (*)
• Calculate the first and second derivative:
Φ/x = fTf/x
2Φ/x2 = (f/x)Tf/x + fT 2f/x2 (f/x)Tf/x
• Note that f/x =[f/u f/v]=[F G]
Equation (*) now becomes a Ax=b problem, which can be
solved by least square.
16
![Page 17: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/17.jpg)
Wiberg Algorithm
• Fixing part of the parameters AND apply the idea of
Newton’s method with Gauss-Newton Assumption.
• By doing so (fixing V for example), the algorithm
becomes more efficient.
• Let Φ(U,V) = (U,V(U)) =1/2gTg
where g(u)=f(u,v(u)) is a function of u only. • Now try to use Newton’s method:
2 /u2 u+ /u =0
We need to know the first and second derivative of
w.r.t. u.
17
![Page 18: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/18.jpg)
Wiberg Algorithm
• Again, using Gauss-Newton Approx, we have:
/u = gTg/u
2 /u2 = (g/u)Tg/u + gT 2g/u2 (g/u)Tg/u
• Using chain rule:
g/u = f/u + f/v v/u = F+G(v/u)
• At optimal (u,v): Φ/u = 0 AND Φ/v = 0 (KKT)
Φ/u = /u will be enforce to 0 by Gauss-Newton.
Φ/v=0 is irrelevant to u, so:
2 Φ/(vu)= (fTf/v)/ u = (fTG)/ u
= (f/v * v/u+ f/u )TG + fT(G)/ u 0
18
![Page 19: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/19.jpg)
Wiberg Algorithm
• (f/v * v/u+ f/u )TG = 0
Take Transpose both side:
GT(G v/u+F) = 0
• v/u = -(GTG)-1GTF
• Substitute into g/u = F+G(v/u) =(I- G(GTG)-1GT)F
Let QG = I- G(GTG)-1GT
• We obtained the step u for Gauss-Newton update in:
Minimize || - QGWy+QGF u ||
This is a least square problem. As derived in Okatani
Paper [16].
19
![Page 20: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/20.jpg)
Insufficient Rank of QGF
• Okatani proved that this QGF have dimension
(m-r)r instead of full rank mr.
• So further constraints that || u || is minimized
should be used to uniquely determine a u.
• Yet, in this Eriksson paper, it is not mentioned.
(corresponds to the Insufficient Rank of
Jacobian)
20
![Page 21: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/21.jpg)
Connection to Linearized Model
Consider
Minimize || - QGWy+QGF u ||
Substitute in the value of QG= I- G(GTG)-1GT
We have
Minimize||-Wy + G(GTG)-1GT Wy + QGF u ||
Recall that G(GTG)-1GT Wy = Gv*(u) and g/u = QGF
We may define function g’ = g+Wy = Gv*(u) ,
Since Wy is const. so g’/u = g/u
21
![Page 22: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/22.jpg)
Connection to Linearized Model
• Now it becomes
Minimize ||-Wy + g’(uk)+ g’/u u ||
• This corresponds to Equation (8) in the paper
and Jk = g’/u|u=uk= QGF.
(Btw there’s a missing term in Equation (8))
• We have showed that Gauss-Newton update is
equivalent to such linearized model.
22
![Page 23: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/23.jpg)
Proposed algorithm: Wiberg L1
• So using the same “Linearized Model”
argument, the L2 Wiberg model can be
extended to L1.
23
![Page 24: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/24.jpg)
Detour: Linear Programming
• Canonical form of an LP
• w.o.l.g, A (m*n) is of full rank m.
• No inequality constraints because we can
add slack variables.
24
![Page 25: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/25.jpg)
Basic Solution and Non-basic Solution
• Reorder A to get A=[B N] where B is m*m
non-singular.
• x=[xB xN] and xN =0, then x is b.s.
• If xB >=0, x is called basic feasible
solution(b.f.s.), if cTx is optimal, then
optimal basic solution.
• Fundamental Thm of LP: If an LP is
feasible, then exists a optimal b.f.s.
25
![Page 26: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/26.jpg)
Simplex Method
26
![Page 27: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/27.jpg)
Simplex Method
• Pivot operation to go from one vertex to
another until optimal (in finite steps).
• Each vertex represents a set of b.f.s., pivot
changes columns between B and N and
find a new b.f.s.
• Can be done efficiently using any
commercial solver.
27
![Page 28: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/28.jpg)
Sensitivity Analysis of LP
• How does optimal solution change if we
change constraints (A and b) or objective
function ( c ) locally?
• Thm 3.2:
28
![Page 29: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/29.jpg)
The use of sensitivity analysis in Wiberg L1
1. Fix U=Uk, calculate the optimal L1 Norm w.r.t. V
using LP, and get Vk=V*.
2. Based on the sensitivity analysis, calculate the
gradient of V* w.r.t. U at Uk, hence linearize
Φ(U) at Uk and calculate the optimal L1 Norm
w.r.t. a change step δ of U using LP again.
3. Then update U and proceed in a gradient
descent manner. 29
![Page 30: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/30.jpg)
Proposed Algorithm: Wiberg L1
• f= F(V)u-Wy= G(U)v-Wy
• Φ1(u) = vec(WUV*(U))
=F(V)u=G(U)v=G(u)v*(u)=F(v*(u))u 30
![Page 31: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/31.jpg)
LP to find v* given any U
• Given a Uk, we can solve for v* that minimize the L1
norm using the following LP. (Simplex will do.)
31
![Page 32: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/32.jpg)
Given v*, linearize the model and find the optimal u
• In order to do that, we must calculate J(U)
Φ1/u = Φ1/u + Φ1/v*v/u
=
• F and G is constant or known,
32
![Page 33: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/33.jpg)
Given v*, linearize the model and find the optimal u
33
Very high
dimension even
for small data
matrix.
Even sparse representation
doesn’t work.
![Page 34: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/34.jpg)
Minimizing L1 norm over u
• To find the optimal u, we again want to
minimize the L1 norm.
• Again, there’s a typo, a term is missing here at
(41)(42) in the paper. Also, the J(uk)u term
makes no sense at all. Correct formulation is:
min. ||Wy – Φ1(uk)-J(uk)(u)||
34
![Page 35: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/35.jpg)
LP to find u and minimize L1 norm
• Note the trust region defined here.
• Only within a small step, the linearity assumption is true.
• Again, this is solved by Simplex method.
35
![Page 36: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/36.jpg)
Wiberg L1 Summary
36
![Page 37: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/37.jpg)
Experiments
• The paper presents comparison between
proposed algorithm and ALP & AQP presented
in [13] over synthetic data.
• There is also an application in SfM using the
Dinosaur Sequence and compare to Wiberg L2.
• Results are pretty presentable in terms accuracy
of recovery and speed.
37
![Page 38: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/38.jpg)
Synthetic Data Matrix
• m = 7; n = 12; r = 3; (very small!)
• 20% Random Distributed Missing Entries
• 10% Random Distributed Sparse outlier
noise Uni(-5,5)
• Tested over 100 data matrices to obtain a
statistic.
38
![Page 39: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/39.jpg)
Histogram: Frequency vs. Error
39
![Page 40: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/40.jpg)
Speed of convergence: Residual vs. Iteration
40
![Page 41: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/41.jpg)
Speed of convergence: log error vs. Iteration
41
![Page 42: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/42.jpg)
Table of Results
• ALP has poor results in terms of error, though speed is
almost real time.
• AQP has long running time and moderate performance.
• Wiberg outperforms others in average error, and is
considerably fast. 42
![Page 43: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/43.jpg)
Structure from Motion
• 319 features are tracked over 36 views
• Uni[-50,50] outliers are artificially added
into the 10% of tracked points.
• Unknown sample rate
43
![Page 44: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/44.jpg)
Data matrix of Dinosour Sequence
44
Generated using the data provided by Oxford Visual Geometry
Group: There are 4983 feature tracks instead of 319.
http://www.robots.ox.ac.uk/~vgg/data1.html
![Page 45: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/45.jpg)
Comparison of L1 and L2 Wiberg completion results
45
![Page 46: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/46.jpg)
Result table and Reconstructed point cloud
46
![Page 47: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/47.jpg)
Critique on this SfM attempt
• Too little information is provided how they
did this SfM. What’s the sample rate? The
factorization to U and V is not unique as
mentioned earlier.
• The code provided doesn’t work for
72*319 at all. It breaks down at 20*70…
• What residual is used in Figure 4 is not
clear L1 or L2.
47
![Page 48: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/48.jpg)
Comparison with Rosper
• Rosper: Lee Choon Meng’s explicit rank
constraint low-rank optimizer
• Solving by Proximal Gradient (no A due to
non-convexity)
• Optimize W and E concurrently
48
![Page 49: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/49.jpg)
Compared algorithms
• Wiberg L1: unoptimized version of code
provided by Anders Eriksson. Slow and not
scalable. (Not working for matrices as small as
20*80)
• Original Wiberg (L2): provided by Okatani,
citation [16]. Features checking if a data matrix
is well-posed for the algorithm. (If QFG is of its
maximum rank (n-r)r )
• These two algorithms both result in non-unique
U and V. In order to compare, we only compare
W=U*V. 49
![Page 50: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/50.jpg)
Difference in Error Model
• Wiberg L1 minimizes L1 Norm, which
handles sparse outliers in data matrix.
• Wiberg L2 minimizes L2 Norm, which in
theory is optimal (MLE) if the noise is
Gaussian distributed.
• Rosper models noise more explicitly. The
first term handles dense Gaussian noise
while the second term handles sparse
outliers. 50
![Page 51: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/51.jpg)
Experiment design
• Data matrix dimension: 20*30
• Each element of U and V fall within [0,1]
• Sample rate: ~40%
• Uniformly sampled
• Noise of various scale:
– Uniform noise to represent sparse outlier
– Gaussian noise to represent dense noise
51
![Page 52: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/52.jpg)
An illustration of the sampled data matrix
52
![Page 53: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/53.jpg)
Experiment 3: 0.3 Error
53
![Page 54: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/54.jpg)
Experiment 5: 0.5 Error
54
![Page 55: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/55.jpg)
Experiment 6: 0.7 Error
55
![Page 56: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/56.jpg)
Experiment 4: 0 Error
56
![Page 57: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/57.jpg)
Different error rate
57
0
200
400
600
800
1000
1200
0 0.3 0.5 0.7
L1 Error Norm over different error rate
Wiberg L1
Wiberg L2
Rosper
![Page 58: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/58.jpg)
Different error rate
58
0
10
20
30
40
50
60
70
80
90
0 0.3 0.5 0.7
L2 Error Norm over different error rate
Wiberg L1
Wiberg L2
Rosper
![Page 59: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/59.jpg)
Result for error free experiment
59
![Page 60: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/60.jpg)
Experiment 7: 30% Large Outlier!
60
![Page 61: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/61.jpg)
Large Sparse Outlier
61
0
2000
4000
6000
8000
10000
12000
L1 Norm
Error L1 Norm under large outlier noise: 30% Uniform(-5,5)
Wiberg L1
Wiberg L2
Rosper
![Page 62: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/62.jpg)
Large Sparse Outlier
62
0
100
200
300
400
500
600
700
800
900
1000
L2 Norm
Error L2 Norm under large outlier noise: 30% Uniform(-5,5)
Wiberg L1
Wiberg L2
Rosper
![Page 63: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/63.jpg)
Experiment 9:Dense N(0,0.5)
63
![Page 64: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/64.jpg)
Experiment 10:Dense N(0,0.1)
64
![Page 65: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/65.jpg)
Dense Gaussian Noise
65
0
100
200
300
400
500
600
700
800
900
1000
N(0,0.5) N(0,0.1)
Error L1 Norm under Dense Gaussian Noise
Wiberg L1
Wiberg L2
Rosper
![Page 66: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/66.jpg)
Dense Gaussian Noise
66
0
20
40
60
80
100
120
140
N(0,0.5) N(0,0.1)
Error L2 Norm under Dense Gaussian Noise
Series1
Series2
Series3
![Page 67: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/67.jpg)
Experiment 11:Dense N(0,0.1)+0.2* Outlier Uniform(-5,5)
67
![Page 68: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/68.jpg)
Experiment 12:Dense N(0,0.1)+0.2* Outlier Uniform(-1,1)
68
![Page 69: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/69.jpg)
Dense Gaussian Noise + Sparse Outlier
69
0
500
1000
1500
2000
2500
3000
3500
4000
4500
Uni(-1,1) Uni(-5,5)
Error L1 Norm under dense N(0,0.1) AND 20% sparse outlier
Wiberg L1
Wiberg L2
Rosper
![Page 70: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/70.jpg)
Dense Gaussian Noise + Sparse Outlier
70
0
200
400
600
800
1000
1200
1400
1600
1800
Uni(-1,1) Uni(-5,5)
Error L2 Norm under dense N(0,0.1) AND 20% sparse outlier
Wiberg L1
Wiberg L2
Rosper
![Page 71: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/71.jpg)
Diagonal band shaped data matrix
71
![Page 72: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/72.jpg)
Diagonal band shaped data matrix
EXP1: 20% Uniform Error
0.45 Sample Rate [m n r]=[20 66 4]
Wiberg L1 Rosper
# of iterations 17 2027
Running Time 6 min 36 sec 9 sec
|Recovered E - E_ground_truth| 106.72 52.55
|Recovered Missing - W_ground_truth| 253.35 397.26
L1 Norm: |Wgnd - Wr| 304.35 464.8
L2 Norm: |Wgnd - Wr| 16.38 25.53
L1 Norm: |mask(Wgnd - Wr)| 51 67.54
L2 Norm: |mask(Wgnd - Wr)| 4.55 5.2
72
![Page 73: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/73.jpg)
Diagonal band shaped data matrix
73
![Page 74: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/74.jpg)
Banded Results table
EXP2: 0 Uniform Error
0.43 Sample Rate
[m n r]=[20 56 4] Wiberg L1 Rosper
# of iterations 35 5675
Running Time 6 min 59 sec 23 sec
|Recovered E - E_ground_truth| <0.01 0.015
|Recovered Missing - W_ground_truth| 61.6 315.6
L1 Norm: |Wgnd - Wr| 61.61 322.02
L2 Norm: |Wgnd - Wr| 6.22 20.87
L1 Norm: |mask(Wgnd - Wr)| <0.01 6.46
L2 Norm: |mask(Wgnd - Wr)| <0.01 0.56
74
![Page 75: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/75.jpg)
Conclusion
• L1 norm is very good in terms of detecting
outliers
• Wiberg L1 outperforms ALP and AQP in
terms of smaller residual.
• The methods of alternatively optimizing
multiple variables are likely to be faster
than optimizing all variables together.
75
![Page 76: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/76.jpg)
Conclusion
• Though claimed to be “efficient”, Wiberg
L1 is slow and not scalable. Wiberg L2
however is a fast and reliable algorithm for
many possible applications.
• Current state of factorization
methods/robust matrix completion still not
sufficient for robust application in most
computer vision applications.
76
![Page 77: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/77.jpg)
Conclusion
• Explicitly model noises in Rosper will have
positive effects on results if the noise
model is correct. Yet, it can be difficult
tuning the weighting.
• In general, Rosper is similar to Wiberg L1
in performance, and is much faster.
77
![Page 78: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/78.jpg)
Future works
• Deal with missing entries in rPCA?
• Generating the same 100 small random
matrix and run the full test for Rosper and
compare.
78
![Page 79: Efficient Computation of Robust Low-Rank Matrix ...yuxiangw/docs/WibergL1_Slide.pdf · Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf69d4b273b4348b761108/html5/thumbnails/79.jpg)
• Jacob, Martinez
• Spectrally optimal Factorization of
Incomplete Matrix
79