Singular Value Decomposition

Tr

VU

xxxxx

xxxxx

xxxxx

xxxxx

xxxxx

xxxxx

xxxxx

1

nmA is

A = (orthogonal) (diagonal) (orthogonal)TVUA

8)TVUA T

rrrTT vuvuvuA 222111

Applications of the SVD

/. Image processing

G. Strang, Linear Algebra and its Applications p444

xxxxxxxxxx

xxxxxxxxxx

xxxxxxxxxx

xxxxxxxxxx

picture matrix

Suppose a satellite takes a picture, and wants to send it to earth. The picture may contain 1000 by 1000 "pixels"—little squares each with a definite color. We can code the colors, in a range between black and white, and send back 1,000,000 numbers

Applications of the SVD

/. Image processing

It is better to find the essential information in the 1000 by 1000 matrix, and send only that.

Trrr

TT vuvuvuA 222111

G. Strang, Linear Algebra and its Applications p444

If only 60 terms are kept, we send 60 times 2000 numbers instead of a million.

Suppose we know the SVD.

The key is in the singular values.

Typically, some are significant and others are extremely small.

If we keep 60 and throw away 940, then we send only the corresponding 60 columns of U, and V.

The other 940 columns are multiplied by small singular values that are being ignored. In fact, we can do the matrix multiplication as columns times rows:

Applications of the SVD

Tvu 111 Tvu 222 Tvu 333 Tvu 444

Tvu 555 Tvu 666 Tvu 777Tvu 888

the SVD of a 32-times-32 digital image A is computed

the activities are lead by Prof. Per Christian Hansen.

Applications of the SVD

1A 2A 3A 4A

5A 6A 7A 8A

Tsss

TTs vuvuvuA 222111

rs

G. Strang “ at first you see nothing, and suddenly you recognize everything.”

Applications of the SVD

Tsss

TTs vuvuvuA 222111

rs

If A is symmetric

How to compute SVD (by hand)

Eigenvalue Decomposition

rseigenvecto ],,,[ 21 mSSSS

If A is real n-by-n matrix, then 1 SSA),,,( 21 ndiag

TQQA IQQT

Example: A

2

1

2

12

1

2

1

2

1

2

12

1

2

1

50

03

41

14

Q TQ

How to compute SVD (by hand)

TVUA

TTT UVA ))(( TTTT UVVUAA

TTT UUAA

)()( TAAA ))(( TTTT VUUVAA

TTT VVAA )()( AAA T

symmetric is TAA

symmetric is AAT

How to compute SVD (by hand)

TVUA TTT UVA

))(( TTTT UVVUAA TTT UUAA

Example:

00

11

11

A

22

22AAW T 0

22

22)det(

IW

0

4

2

1

1

1,

1

12

122

11 vv

0

2

2

1

11

1

1Avu

1

122

1 A

0

2

12

1

1u )Null(AAfor basis lorthonorma , T32 uu

000

022

022TAA

02

12

1

2u

1

0

0

3u

2

1

2

12

1

2

1

2

1

2

12

1

2

1

00

00

02

100

0

0

00

11

11Range(A) Rank(A)

Null(A)

))(( TTTT VUUVAA TTT VVAA

)()( TAAA

)()( AAA T

xxxxx

xxxxx

xxxxx

xxxxx

xxxxx

MatrixBidiagonal

Diagonal

How to compute SVD (Algorithm)

x

x

x

x

x

x

xx

xx

xx

xxHouseholder transformations

QR

transform

ations

>>[U,S,V] = svd(A)

Example:

Singular Value Decomposition

9) If A is a square matrix then

021 n 12

A

10) If A is a square matrix then

021 n 22

221

2

rFA

32A 10

FA

Tr

VU

xxxxx

xxxxx

xxxxx

xxxxx

xxxxx

xxxxx

xxxxx

1

Singular Value Decomposition

11) If A is a square symmetric matrix then the singular values of A are the absolute values of the eigenvalues of A.

Tr

VU

xxxxx

xxxxx

xxxxx

xxxxx

xxxxx

xxxxx

xxxxx

1

TT QQQQA )sign(

12) If A is a square matrix then

i

n

iA

1)det(

SVD and Eigenvalue Decomposition

SVD Eigen Decomp

Uses two different bases U, V

Uses just one (eigenvectors)

Uses orthonormal basesGenerally is not

orthogonal

All matrices(even rectangular)

Not all matrices (even square)

(only diagonalizable)

1 SSATVUA

Reduced SVD

A TVU

Reduced SVD

Singular Value Decomposition

Example : Luke Olson\.illinois 

svd_test.m

iguana.jpg

Singular Value Decomposition

Theorem: (Singular Value Decomposition) SVD

mmm RuuuU ],,,[ 21

If A is real m-by-n matrix, then there exist orthogonal matrices

nnn RvvvV ],,,[ 21

such thatTVUA

where 021 p (m,n)p min

),,( 1 pdiag

A U TV

Proof:

Singular Value Decomposition

First approximation to A is TvuA 1111

TT vuvuA 2221112 second approximation to A is

TTT vuvuvuA 222111

Trrr

TT vuvuvuA 222111

221

)(

inf

satisfies also matrix the,0 with any For :Theroem

rF

BrankRBF

BAAA

Ar

nm

Singular Value Decomposition

In later years he drove a car with the license plate:

Trefethen (Textbook author): The SVD was discovered independently by Beltrami(1873) and Jordan(1874) and again by Sylvester(1889). The SVD did not become widely known in applied mathematics until the late 1960s, when Golub and others showed that it could be computed effectively.

Cleve Moler (invented MATLAB, co-founded MathWorks)Gene Golub has done more than anyone to make the singular value decomposition one of the most powerful and widely used tools in modern matrix computation.