2
Motivation
yUx
bLy
bLUx
LUA
A
matrixany :
bSSx
SSA
SSA
e-vectorsete set ofwith compl
A
11
111
1
matrixany :
bQQx
QQA
QQA
symmetricA
T
T
T
1
11
matrix :
bUVx
UVA
VUA
A
T
T
T
1
11
matrixany :
bUVx
UVA
VUA
A
T
T
T
1
11
matrixany :
Clearly the winner
Assume A full rank
3
Ideas Behind SVD
Goal: for Am×n
find orthonormal bases for C(AT) and C(A)
rowspace
Ax=0
columnspace
ATy=0
orthonormal basis in C(AT)
orthonormal basis in C(A)
ARn Rm
There are many choices of basis in C(AT) and C(A), but we want the orthonormal ones
There are many choices of basis in C(AT) and C(A), but we want the orthonormal ones
4
SVD (2X2)
2
2
2
2
1
1
1
1
21
21
21
,
:)(in imagesr want thei we
:)(in rsunit vecto
2 Assume
Av
Av
AvAv
Av
Av
T
uu
AvAvACAND
vvAC
r
: represent the length of images; hence non-negative
: represent the length of images; hence non-negative
UAV
uuuuvvA2
121221121
I haven’t told you how to find
vi’s (p.9)
5
SVD 2x2 (cont)
TVUVUA
UAV
1
When A has complete set of e-vectors, we have
AS=S , A=SS
but S in general is not orthogonal
When A has complete set of e-vectors, we have
AS=S , A=SS
but S in general is not orthogonal
Another diagonalization using 2 sets of orthogonal bases
Another diagonalization using 2 sets of orthogonal bases
When A is symmetric, we have
A=QQT
When A is symmetric, we have
A=QQT
Compare
6
Why are orthonormal bases good? ( )-1=( )T
Implication: Matrix inversion
Ax=b TT
T
UVVUA
VUA111
system diagonal ... bUxV
bxVU
bAx
VUA
TT
T
T
7
More on U and V
TTTTT
TTTTT
UUUVVUAA
VVVUUVAA
22
21
22
21
Similarly,
V: eigenvectors of ATA
U: eigenvectors of AAT
V: eigenvectors of ATA
U: eigenvectors of AAT
[Find vi first, then use Avi to find ui
This is the key to solve SVD
8
SVD: A=UV
The singular values are the diagonal entries of the matrix and are arranged in descending order
The singular values are always real (non-negative) numbers
If A is real matrix, U and V are also real
9
Example (2x2, full rank)
2121
2121
20
022
10
01
1
0,
1
0
2
0
0
1,
0
1
0
22
21
21,
21
21,
53
35
11
22
222
111
21
T
T
VUA
uAv
uAv
vvAA
A
STEPS:1. Find e-vectors of ATA;
normalize the basis2. Compute Avi, get i
If i 0, get ui
Else find ui from N(AT)
STEPS:1. Find e-vectors of ATA;
normalize the basis2. Compute Avi, get i
If i 0, get ui
Else find ui from N(AT)
10
SVD Theory
If j=0, Avj=0vj is in N(A) The corresponding uj in N(AT)
[UTA=VT=0]
Else, vj in C(AT) The corresponding uj in C(A)
#of nonzero j = rank
rjuAv
UAV
jjj ,,2,1,
11
11
11
2
1
00
010
21
12
5
1
11
22
2
1
5
100)(
1
1
2
10)(
101
2
5
1
1
1
2
1
11
22
1
1
2
1;
1
1:basis )(,1,
11
22
122212
22212
11
111
1
T
TTT
T
VU
uAuuAANuuu
vAvANvvv
uAv
vACrA
Example (2x2, rank deficient)
Can also be obtained from
e-vectors of ATA
12
Example (cont)
11
11
2
1
00
010
21
12
5
1
11
22 TVU
TT
T
TT
vuv
uu
v
vuuVUA
11111
21
2
1121
0
00
0
Bases of N(A) and N(AT) (u2 and v2 here) do not contribute the final result. The are computed to make U and V orthogonal.
Bases of N(A) and N(AT) (u2 and v2 here) do not contribute the final result. The are computed to make U and V orthogonal.
13
Extend to Amxn
0
,,1,
1
1111r
mrrnrr
iii
uuuuvvvvA
riuAv
Basis of N(A) Basis of N(AT)
T
mmmmT
mm
TTTmm
T
Tnnnn
Tnn
TTTnn
T
Tnnnmmmnm
VUVUVUAA
VVVUVUAA
VUA
UAV
T
mmmmT
mm
TTTmm
T
Tnnnn
Tnn
TTTnn
T
Tnnnmmmnm
VUVUVUAA
VVVUVUAA
VUA
UAV
Dimension Check
14
Extend to Amxn (cont)
Trrr
T
Tm
Tr
Tr
T
rr
rmrrnrr
vuvu
v
vv
v
uuA
uuuuvvvvA
1111
1
11
1
1111
00
0
Bases of N(A) and N(AT) do not contribute
They are useful only for nullspace solutions
Summation of r rank-one matrices!
15
110
121
011
:
,2,110
01132223332
AAV
UVArA
T
)(nullspace
1
1
1
3
1,0
1
0
1
2
1,1
1
2
1
6
1,3
33
22
11
v
v
v
1,1
1
2
1
3,1
1
6
23
3
3
6
1
22
121
1
Av
Av
31
31
31
21
21
21
62
61
31
21
61
31
62
31
21
61
01
3
11
11
2
1
110
011
1
3
11
11
2
10
110
011
C(AT) N(A) C(A)
16
2
1
332211
of Expansion
707.0
0
707.0
542.0
643.0
542.0
454.0
776.0
454.0
, of vectors Eigen
A of rank values singular zero-non of Number,
0
14.0
86.28
, of values Eigen
6
10
6
106
1710
106
,
1
2
1
21
32
21
:Example
i
Tiii
TT
TT
TT
vuA
A
vuvuvu
AAAA
AAAA
AAAAA
17
Summary
SVD chooses the right basis for the 4 subspaces AV=U
v1…vr: orthonormal basis in Rn for C(AT) vr+1…vn: N(A) u1…ur: in Rm C(A) ur+1…um: N(AT)
These bases are not only , but also Avi=iui
High points of Linear Algebra Dimension, rank, orthogonality, basis, diagonalization, …
18
SVD Applications
Using SVD in computation, rather than A, has the advantage of being more robust to numerical error
Many applications: Inverse of matrix A Conditions of matrix Image compression Solve Ax=b for all cases (unique, many, no solutions; least
square solutions) rank determination, matrix approximation, …
SVD usually found by iterative methods (see Numerical Recipe, Chap.2)
19
otherwise
tifUVVUA
A
diagUVAorVUA
iA
iiTT
nTT
i
0
/1 where
dconditione-ill or singular is
/1,....,/1,/1 where
all for0 iff rnonsingula is
Inverse Matrix
10
10
11
21111
nσσA
xb
bAx
/ =:y singularit of Degree
in changes large produces in changes small when dconditione-Ill
=
equationslinear of System
matrix a ofy singularit of Degree
1
20
SVD and Ax=b (mn)
Check for existence of solution
existnot doessolution
,0but 0 If
ii
d
T
z
TT
d
dz
bUxVbxVU
21
Ax=b (inconsistent)
52
520
3
8
21
12
5
1
00
010
11
11
2
1
00
010
21
12
5
1
11
22
3
8,
11
22
2
1 bUz
z
VU
bA
T
T
No solution!
22
Ax=b (underdetermined)
21
21
2
2
2
2
0
22
11
11
2
1
0
22
0
520
4
8
21
12
5
1
00
010
11
11
2
1
00
010
21
12
5
1
11
22
4
8,
11
22
2
1
2
1
cxxx
Vzxz
z
bUz
z
VU
bA
nullparticularcomplete
particular
T
T
23
Pseudo Inverse (Sec7.4, p.395) The role of A:
Takes a vector vi from row space to iui in the column space
The role of A-1 (if it exists): Does the opposite: takes a vector ui from column
space to row space vi
ii
iii
iii
vuA
uAv
uAv
i
11
1
ii
iii
iii
vuA
uAv
uAv
i
11
1
24
Pseudo Inverse (cont)
While A-1 may not exist, a matrix that takes ui back to vi/i does exist. It is denoted as A+, the pseudo inverse
A+: dimension n by m
T
mr
r
nr
Tmmmnnn
iii
i
uuuvvv
UVA
riuArivuA
11
11
1
for 0 and for 1
25
Pseudo Inverse and Ax=b
Overdetermined case: find the solution that minimize the error r=|Ax–b|, the least square solution
A panacea for Ax=bA panacea for Ax=b
bUVbAx
bAxT
bUVbAx
bAxT
bAxAA TT ˆ Compare
26
Ex: full rank
1
1
2
0
10
01
0
0
2121
2121
)/1(diag
2121
2121
20
022
10
01
2
0,
11
22
21
221
x
bUVxbxVUbAx
VUA
bA
Ti
T
T
27
Ex: over-determined
T
T
UVA
VUA
UVArbA
11
11
2
11
3
0
10
11
01
,2,
1
2
1
,
10
11
01
31
21
61
31
62
31
21
61
23332223
Will show this need not be computed…
28
Over-determined (cont)
31
35
322
226
4
31
21
21
61
62
61
31
211
11
2
1
2111
11
2
1
1
2
1
0111
11
2
1
zyx
zyx
bUVx T
31
35
2
1
2
1
1
3
21
12
ˆ Compare
x
x
x
x
bAxAA TT
31
35
2
1
2
1
1
3
21
12
ˆ Compare
x
x
x
x
bAxAA TT
Same result!!
29
Ex: general case, no solution
1019
1019
101
51
101
51
51
52
521
521
51
52
101
212
1
3
8
3
8
0
0
3
8
00
0
11
11
2
1
00
010
21
12
5
1
11
22
3
8,
11
22
yx
yxy
x
bUVx
VU
bA
T
T
3
822
yx
yx
30
Matrix Approximation
nm
nm
vuvuvuA
AiA
imi
VUA
Tmmm
TT
i
i
Tii
:saveOperation
numbers )(matrix onerank :save Storage
distanceEuclidean of sence in the ion toapproximat rank best the:
zero) tos' last setting(by of version rank the:
222111
making small ’s to zero and back substitute(see next page for application in image compression)
31
Image Compression
As described in text p.352 For grey scale images: mn bytes
Only need to store r(m+n+1)
Trrr
TT vuvuvuA
rSVD
222111
: termst significanmost the taking,After
r = 1,3,5,10,16 (no perceivable difference afterwards)Original
6464