cs221 linear algebra

156

Upload: darwinrlo

Post on 06-May-2015

1.065 views

Category:

Technology


1 download

TRANSCRIPT

Page 1: Cs221 linear algebra

Linear Algebra Review

CS221: Introduction to Arti�cial Intelligence

Carlos Fernandez-Granda

10/11/2011

CS221 Linear Algebra Review 10/11/2011 1 / 37

Page 2: Cs221 linear algebra

Index

1 Vectors

2 Matrices

3 Matrix Decompositions

4 Application : Image Compression

CS221 Linear Algebra Review 10/11/2011 2 / 37

Page 3: Cs221 linear algebra

Vectors

1 Vectors

2 Matrices

3 Matrix Decompositions

4 Application : Image Compression

CS221 Linear Algebra Review 10/11/2011 3 / 37

Page 4: Cs221 linear algebra

Vectors

What is a vector ?

An ordered tuple of numbers :

u =

u1u2. . .un

A quantity that has a magnitude and a direction.

CS221 Linear Algebra Review 10/11/2011 4 / 37

Page 5: Cs221 linear algebra

Vectors

What is a vector ?

An ordered tuple of numbers :

u =

u1u2. . .un

A quantity that has a magnitude and a direction.

CS221 Linear Algebra Review 10/11/2011 4 / 37

Page 6: Cs221 linear algebra

Vectors

What is a vector ?

An ordered tuple of numbers :

u =

u1u2. . .un

A quantity that has a magnitude and a direction.

CS221 Linear Algebra Review 10/11/2011 4 / 37

Page 7: Cs221 linear algebra

Vectors

Vector spaces

More formally a vector is an element of a vector space.

A vector space V is a set that contains all linear combinations of itselements :

If vectors u and v ∈ V, then u + v ∈ V.If vector u ∈ V, then α u ∈ V for any scalar α.There exists 0 ∈ V such that u + 0 = u for any u ∈ V.

A subspace is a subset of a vector space that is also a vector space.

CS221 Linear Algebra Review 10/11/2011 5 / 37

Page 8: Cs221 linear algebra

Vectors

Vector spaces

More formally a vector is an element of a vector space.

A vector space V is a set that contains all linear combinations of itselements :

If vectors u and v ∈ V, then u + v ∈ V.If vector u ∈ V, then α u ∈ V for any scalar α.There exists 0 ∈ V such that u + 0 = u for any u ∈ V.

A subspace is a subset of a vector space that is also a vector space.

CS221 Linear Algebra Review 10/11/2011 5 / 37

Page 9: Cs221 linear algebra

Vectors

Vector spaces

More formally a vector is an element of a vector space.

A vector space V is a set that contains all linear combinations of itselements :

If vectors u and v ∈ V, then u + v ∈ V.

If vector u ∈ V, then α u ∈ V for any scalar α.There exists 0 ∈ V such that u + 0 = u for any u ∈ V.

A subspace is a subset of a vector space that is also a vector space.

CS221 Linear Algebra Review 10/11/2011 5 / 37

Page 10: Cs221 linear algebra

Vectors

Vector spaces

More formally a vector is an element of a vector space.

A vector space V is a set that contains all linear combinations of itselements :

If vectors u and v ∈ V, then u + v ∈ V.If vector u ∈ V, then α u ∈ V for any scalar α.

There exists 0 ∈ V such that u + 0 = u for any u ∈ V.A subspace is a subset of a vector space that is also a vector space.

CS221 Linear Algebra Review 10/11/2011 5 / 37

Page 11: Cs221 linear algebra

Vectors

Vector spaces

More formally a vector is an element of a vector space.

A vector space V is a set that contains all linear combinations of itselements :

If vectors u and v ∈ V, then u + v ∈ V.If vector u ∈ V, then α u ∈ V for any scalar α.There exists 0 ∈ V such that u + 0 = u for any u ∈ V.

A subspace is a subset of a vector space that is also a vector space.

CS221 Linear Algebra Review 10/11/2011 5 / 37

Page 12: Cs221 linear algebra

Vectors

Vector spaces

More formally a vector is an element of a vector space.

A vector space V is a set that contains all linear combinations of itselements :

If vectors u and v ∈ V, then u + v ∈ V.If vector u ∈ V, then α u ∈ V for any scalar α.There exists 0 ∈ V such that u + 0 = u for any u ∈ V.

A subspace is a subset of a vector space that is also a vector space.

CS221 Linear Algebra Review 10/11/2011 5 / 37

Page 13: Cs221 linear algebra

Vectors

Examples

Euclidean space Rn.

The span of any set of vectors {u1,u2, . . . ,un}, de�ned as :

span (u1,u2, . . . ,un) = {α1u1 + α2u2 + · · ·+ αnun | αi ∈ R}

CS221 Linear Algebra Review 10/11/2011 6 / 37

Page 14: Cs221 linear algebra

Vectors

Examples

Euclidean space Rn.

The span of any set of vectors {u1,u2, . . . ,un}, de�ned as :

span (u1,u2, . . . ,un) = {α1u1 + α2u2 + · · ·+ αnun | αi ∈ R}

CS221 Linear Algebra Review 10/11/2011 6 / 37

Page 15: Cs221 linear algebra

Vectors

Subspaces

A line through the origin in Rn(span of a vector).

A plane in Rn(span of two vectors).

CS221 Linear Algebra Review 10/11/2011 7 / 37

Page 16: Cs221 linear algebra

Vectors

Subspaces

A line through the origin in Rn(span of a vector).

A plane in Rn(span of two vectors).

CS221 Linear Algebra Review 10/11/2011 7 / 37

Page 17: Cs221 linear algebra

Vectors

Linear Independence and Basis of Vector Spaces

A vector u is linearly independent of a set of vectors {u1,u2, . . . ,un}if it does not lie in their span.

A set of vectors is linearly independent if every vector is linearlyindependent of the rest.

A basis of a vector space V is a linearly independent set of vectorswhose span is equal to V.If a vector space has a basis with d vectors, its dimension is d.

Any other basis of the same space will consist of exactly d vectors.

The dimension of a vector space can be in�nite (function spaces).

CS221 Linear Algebra Review 10/11/2011 8 / 37

Page 18: Cs221 linear algebra

Vectors

Linear Independence and Basis of Vector Spaces

A vector u is linearly independent of a set of vectors {u1,u2, . . . ,un}if it does not lie in their span.

A set of vectors is linearly independent if every vector is linearlyindependent of the rest.

A basis of a vector space V is a linearly independent set of vectorswhose span is equal to V.If a vector space has a basis with d vectors, its dimension is d.

Any other basis of the same space will consist of exactly d vectors.

The dimension of a vector space can be in�nite (function spaces).

CS221 Linear Algebra Review 10/11/2011 8 / 37

Page 19: Cs221 linear algebra

Vectors

Linear Independence and Basis of Vector Spaces

A vector u is linearly independent of a set of vectors {u1,u2, . . . ,un}if it does not lie in their span.

A set of vectors is linearly independent if every vector is linearlyindependent of the rest.

A basis of a vector space V is a linearly independent set of vectorswhose span is equal to V.

If a vector space has a basis with d vectors, its dimension is d.

Any other basis of the same space will consist of exactly d vectors.

The dimension of a vector space can be in�nite (function spaces).

CS221 Linear Algebra Review 10/11/2011 8 / 37

Page 20: Cs221 linear algebra

Vectors

Linear Independence and Basis of Vector Spaces

A vector u is linearly independent of a set of vectors {u1,u2, . . . ,un}if it does not lie in their span.

A set of vectors is linearly independent if every vector is linearlyindependent of the rest.

A basis of a vector space V is a linearly independent set of vectorswhose span is equal to V.If a vector space has a basis with d vectors, its dimension is d.

Any other basis of the same space will consist of exactly d vectors.

The dimension of a vector space can be in�nite (function spaces).

CS221 Linear Algebra Review 10/11/2011 8 / 37

Page 21: Cs221 linear algebra

Vectors

Linear Independence and Basis of Vector Spaces

A vector u is linearly independent of a set of vectors {u1,u2, . . . ,un}if it does not lie in their span.

A set of vectors is linearly independent if every vector is linearlyindependent of the rest.

A basis of a vector space V is a linearly independent set of vectorswhose span is equal to V.If a vector space has a basis with d vectors, its dimension is d.

Any other basis of the same space will consist of exactly d vectors.

The dimension of a vector space can be in�nite (function spaces).

CS221 Linear Algebra Review 10/11/2011 8 / 37

Page 22: Cs221 linear algebra

Vectors

Linear Independence and Basis of Vector Spaces

A vector u is linearly independent of a set of vectors {u1,u2, . . . ,un}if it does not lie in their span.

A set of vectors is linearly independent if every vector is linearlyindependent of the rest.

A basis of a vector space V is a linearly independent set of vectorswhose span is equal to V.If a vector space has a basis with d vectors, its dimension is d.

Any other basis of the same space will consist of exactly d vectors.

The dimension of a vector space can be in�nite (function spaces).

CS221 Linear Algebra Review 10/11/2011 8 / 37

Page 23: Cs221 linear algebra

Vectors

Norm of a Vector

A norm ||u|| measures the magnitude of a vector.

Mathematically, any function from the vector space to R thatsatis�es :

Homogeneity ||α u|| = |α| ||u||Triangle inequality ||u + v|| ≤ ||u||+ ||v||Point separation ||u|| = 0 if and only if u = 0.

Examples :

Manhattan or `1 norm : ||u||1

=∑

i|ui |.

Euclidean or `2 norm : ||u||2

=√∑

iu2i.

CS221 Linear Algebra Review 10/11/2011 9 / 37

Page 24: Cs221 linear algebra

Vectors

Norm of a Vector

A norm ||u|| measures the magnitude of a vector.

Mathematically, any function from the vector space to R thatsatis�es :

Homogeneity ||α u|| = |α| ||u||Triangle inequality ||u + v|| ≤ ||u||+ ||v||Point separation ||u|| = 0 if and only if u = 0.

Examples :

Manhattan or `1 norm : ||u||1

=∑

i|ui |.

Euclidean or `2 norm : ||u||2

=√∑

iu2i.

CS221 Linear Algebra Review 10/11/2011 9 / 37

Page 25: Cs221 linear algebra

Vectors

Norm of a Vector

A norm ||u|| measures the magnitude of a vector.

Mathematically, any function from the vector space to R thatsatis�es :

Homogeneity ||α u|| = |α| ||u||

Triangle inequality ||u + v|| ≤ ||u||+ ||v||Point separation ||u|| = 0 if and only if u = 0.

Examples :

Manhattan or `1 norm : ||u||1

=∑

i|ui |.

Euclidean or `2 norm : ||u||2

=√∑

iu2i.

CS221 Linear Algebra Review 10/11/2011 9 / 37

Page 26: Cs221 linear algebra

Vectors

Norm of a Vector

A norm ||u|| measures the magnitude of a vector.

Mathematically, any function from the vector space to R thatsatis�es :

Homogeneity ||α u|| = |α| ||u||Triangle inequality ||u + v|| ≤ ||u||+ ||v||

Point separation ||u|| = 0 if and only if u = 0.

Examples :

Manhattan or `1 norm : ||u||1

=∑

i|ui |.

Euclidean or `2 norm : ||u||2

=√∑

iu2i.

CS221 Linear Algebra Review 10/11/2011 9 / 37

Page 27: Cs221 linear algebra

Vectors

Norm of a Vector

A norm ||u|| measures the magnitude of a vector.

Mathematically, any function from the vector space to R thatsatis�es :

Homogeneity ||α u|| = |α| ||u||Triangle inequality ||u + v|| ≤ ||u||+ ||v||Point separation ||u|| = 0 if and only if u = 0.

Examples :

Manhattan or `1 norm : ||u||1

=∑

i|ui |.

Euclidean or `2 norm : ||u||2

=√∑

iu2i.

CS221 Linear Algebra Review 10/11/2011 9 / 37

Page 28: Cs221 linear algebra

Vectors

Norm of a Vector

A norm ||u|| measures the magnitude of a vector.

Mathematically, any function from the vector space to R thatsatis�es :

Homogeneity ||α u|| = |α| ||u||Triangle inequality ||u + v|| ≤ ||u||+ ||v||Point separation ||u|| = 0 if and only if u = 0.

Examples :

Manhattan or `1 norm : ||u||1

=∑

i|ui |.

Euclidean or `2 norm : ||u||2

=√∑

iu2i.

CS221 Linear Algebra Review 10/11/2011 9 / 37

Page 29: Cs221 linear algebra

Vectors

Inner Product

Inner product between u and v : 〈u, v〉 =∑n

i=1uivi .

It is the projection of one vector onto the other.

Related to the Euclidean norm : 〈u,u〉 = ||u||22.

CS221 Linear Algebra Review 10/11/2011 10 / 37

Page 30: Cs221 linear algebra

Vectors

Inner Product

Inner product between u and v : 〈u, v〉 =∑n

i=1uivi .

It is the projection of one vector onto the other.

Related to the Euclidean norm : 〈u,u〉 = ||u||22.

CS221 Linear Algebra Review 10/11/2011 10 / 37

Page 31: Cs221 linear algebra

Vectors

Inner Product

Inner product between u and v : 〈u, v〉 =∑n

i=1uivi .

It is the projection of one vector onto the other.

Related to the Euclidean norm : 〈u,u〉 = ||u||22.

CS221 Linear Algebra Review 10/11/2011 10 / 37

Page 32: Cs221 linear algebra

Vectors

Inner Product

The inner product is a measure of correlation between two vectors, scaledby the norms of the vectors :

If 〈u, v〉 > 0, u and v are aligned.

If 〈u, v〉 < 0, u and v are opposed.

If 〈u, v〉 = 0, u and v are orthogonal.

CS221 Linear Algebra Review 10/11/2011 11 / 37

Page 33: Cs221 linear algebra

Vectors

Inner Product

The inner product is a measure of correlation between two vectors, scaledby the norms of the vectors :

If 〈u, v〉 > 0, u and v are aligned.

If 〈u, v〉 < 0, u and v are opposed.

If 〈u, v〉 = 0, u and v are orthogonal.

CS221 Linear Algebra Review 10/11/2011 11 / 37

Page 34: Cs221 linear algebra

Vectors

Inner Product

The inner product is a measure of correlation between two vectors, scaledby the norms of the vectors :

If 〈u, v〉 > 0, u and v are aligned.

If 〈u, v〉 < 0, u and v are opposed.

If 〈u, v〉 = 0, u and v are orthogonal.

CS221 Linear Algebra Review 10/11/2011 11 / 37

Page 35: Cs221 linear algebra

Vectors

Inner Product

The inner product is a measure of correlation between two vectors, scaledby the norms of the vectors :

If 〈u, v〉 > 0, u and v are aligned.

If 〈u, v〉 < 0, u and v are opposed.

If 〈u, v〉 = 0, u and v are orthogonal.

CS221 Linear Algebra Review 10/11/2011 11 / 37

Page 36: Cs221 linear algebra

Vectors

Orthonormal Basis

The vectors in an orthonormal basis have unit Euclidean norm andare orthogonal to each other.

To express a vector x in an orthonormal basis, we can just take innerproducts.

Example : x = α1b1 + α2b2.

We compute 〈x,b1〉 :

〈x,b1〉 = 〈α1b1 + α2b2,b1〉= α1 〈b1,b1〉+ α2 〈b2,b1〉 By linearity.

= α1 + 0 By orthonormality.

Likewise, α2 = 〈x,b2〉.

CS221 Linear Algebra Review 10/11/2011 12 / 37

Page 37: Cs221 linear algebra

Vectors

Orthonormal Basis

The vectors in an orthonormal basis have unit Euclidean norm andare orthogonal to each other.

To express a vector x in an orthonormal basis, we can just take innerproducts.

Example : x = α1b1 + α2b2.

We compute 〈x,b1〉 :

〈x,b1〉 = 〈α1b1 + α2b2,b1〉= α1 〈b1,b1〉+ α2 〈b2,b1〉 By linearity.

= α1 + 0 By orthonormality.

Likewise, α2 = 〈x,b2〉.

CS221 Linear Algebra Review 10/11/2011 12 / 37

Page 38: Cs221 linear algebra

Vectors

Orthonormal Basis

The vectors in an orthonormal basis have unit Euclidean norm andare orthogonal to each other.

To express a vector x in an orthonormal basis, we can just take innerproducts.

Example : x = α1b1 + α2b2.

We compute 〈x,b1〉 :

〈x,b1〉 = 〈α1b1 + α2b2,b1〉= α1 〈b1,b1〉+ α2 〈b2,b1〉 By linearity.

= α1 + 0 By orthonormality.

Likewise, α2 = 〈x,b2〉.

CS221 Linear Algebra Review 10/11/2011 12 / 37

Page 39: Cs221 linear algebra

Vectors

Orthonormal Basis

The vectors in an orthonormal basis have unit Euclidean norm andare orthogonal to each other.

To express a vector x in an orthonormal basis, we can just take innerproducts.

Example : x = α1b1 + α2b2.

We compute 〈x,b1〉 :

〈x,b1〉 = 〈α1b1 + α2b2,b1〉= α1 〈b1,b1〉+ α2 〈b2,b1〉 By linearity.

= α1 + 0 By orthonormality.

Likewise, α2 = 〈x,b2〉.

CS221 Linear Algebra Review 10/11/2011 12 / 37

Page 40: Cs221 linear algebra

Vectors

Orthonormal Basis

The vectors in an orthonormal basis have unit Euclidean norm andare orthogonal to each other.

To express a vector x in an orthonormal basis, we can just take innerproducts.

Example : x = α1b1 + α2b2.

We compute 〈x,b1〉 :

〈x,b1〉 = 〈α1b1 + α2b2,b1〉= α1 〈b1,b1〉+ α2 〈b2,b1〉 By linearity.

= α1 + 0 By orthonormality.

Likewise, α2 = 〈x,b2〉.

CS221 Linear Algebra Review 10/11/2011 12 / 37

Page 41: Cs221 linear algebra

Matrices

1 Vectors

2 Matrices

3 Matrix Decompositions

4 Application : Image Compression

CS221 Linear Algebra Review 10/11/2011 13 / 37

Page 42: Cs221 linear algebra

Matrices

Linear Operator

A linear operator L : U → V is a map from a vector space U toanother vector space V that satis�es :

L (u1 + u2) = L (u1) + L (u2).L (α u) = α L (u) for any scalar α.

If the dimension n of U and m of V are �nite, L can be represented byan m × n matrix A.

A =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

CS221 Linear Algebra Review 10/11/2011 14 / 37

Page 43: Cs221 linear algebra

Matrices

Linear Operator

A linear operator L : U → V is a map from a vector space U toanother vector space V that satis�es :

L (u1 + u2) = L (u1) + L (u2).

L (α u) = α L (u) for any scalar α.

If the dimension n of U and m of V are �nite, L can be represented byan m × n matrix A.

A =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

CS221 Linear Algebra Review 10/11/2011 14 / 37

Page 44: Cs221 linear algebra

Matrices

Linear Operator

A linear operator L : U → V is a map from a vector space U toanother vector space V that satis�es :

L (u1 + u2) = L (u1) + L (u2).L (α u) = α L (u) for any scalar α.

If the dimension n of U and m of V are �nite, L can be represented byan m × n matrix A.

A =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

CS221 Linear Algebra Review 10/11/2011 14 / 37

Page 45: Cs221 linear algebra

Matrices

Linear Operator

A linear operator L : U → V is a map from a vector space U toanother vector space V that satis�es :

L (u1 + u2) = L (u1) + L (u2).L (α u) = α L (u) for any scalar α.

If the dimension n of U and m of V are �nite, L can be represented byan m × n matrix A.

A =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

CS221 Linear Algebra Review 10/11/2011 14 / 37

Page 46: Cs221 linear algebra

Matrices

Matrix Vector Multiplication

Au =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

u1u2. . .un

=

v1v2. . .vm

CS221 Linear Algebra Review 10/11/2011 15 / 37

Page 47: Cs221 linear algebra

Matrices

Matrix Vector Multiplication

Au =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

u1u2. . .un

=

v1v2. . .vm

CS221 Linear Algebra Review 10/11/2011 15 / 37

Page 48: Cs221 linear algebra

Matrices

Matrix Vector Multiplication

Au =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

u1u2. . .un

=

v1v2. . .vm

CS221 Linear Algebra Review 10/11/2011 15 / 37

Page 49: Cs221 linear algebra

Matrices

Matrix Vector Multiplication

Au =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

u1u2. . .un

=

v1v2. . .vm

CS221 Linear Algebra Review 10/11/2011 15 / 37

Page 50: Cs221 linear algebra

Matrices

Matrix Product

Applying two linear operators A and B de�nes another linear operator,which can be represented by another matrix AB.

AB =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

B11 B12 . . . B1p

B21 B22 . . . B2p

. . .Bn1 Bn2 . . . Bnp

=

AB11 AB12 . . . AB1p

AB21 AB22 . . . AB2p

. . .ABm1 ABm2 . . . ABmp

Note that if A is m × n and B is n × p, then AB is m × p.

CS221 Linear Algebra Review 10/11/2011 16 / 37

Page 51: Cs221 linear algebra

Matrices

Matrix Product

Applying two linear operators A and B de�nes another linear operator,which can be represented by another matrix AB.

AB =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

B11 B12 . . . B1p

B21 B22 . . . B2p

. . .Bn1 Bn2 . . . Bnp

=

AB11 AB12 . . . AB1p

AB21 AB22 . . . AB2p

. . .ABm1 ABm2 . . . ABmp

Note that if A is m × n and B is n × p, then AB is m × p.

CS221 Linear Algebra Review 10/11/2011 16 / 37

Page 52: Cs221 linear algebra

Matrices

Matrix Product

Applying two linear operators A and B de�nes another linear operator,which can be represented by another matrix AB.

AB =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

B11 B12 . . . B1p

B21 B22 . . . B2p

. . .Bn1 Bn2 . . . Bnp

=

AB11 AB12 . . . AB1p

AB21 AB22 . . . AB2p

. . .ABm1 ABm2 . . . ABmp

Note that if A is m × n and B is n × p, then AB is m × p.

CS221 Linear Algebra Review 10/11/2011 16 / 37

Page 53: Cs221 linear algebra

Matrices

Matrix Product

Applying two linear operators A and B de�nes another linear operator,which can be represented by another matrix AB.

AB =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

B11 B12 . . . B1p

B21 B22 . . . B2p

. . .Bn1 Bn2 . . . Bnp

=

AB11 AB12 . . . AB1p

AB21 AB22 . . . AB2p

. . .ABm1 ABm2 . . . ABmp

Note that if A is m × n and B is n × p, then AB is m × p.

CS221 Linear Algebra Review 10/11/2011 16 / 37

Page 54: Cs221 linear algebra

Matrices

Matrix Product

Applying two linear operators A and B de�nes another linear operator,which can be represented by another matrix AB.

AB =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

B11 B12 . . . B1p

B21 B22 . . . B2p

. . .Bn1 Bn2 . . . Bnp

=

AB11 AB12 . . . AB1p

AB21 AB22 . . . AB2p

. . .ABm1 ABm2 . . . ABmp

Note that if A is m × n and B is n × p, then AB is m × p.

CS221 Linear Algebra Review 10/11/2011 16 / 37

Page 55: Cs221 linear algebra

Matrices

Matrix Product

Applying two linear operators A and B de�nes another linear operator,which can be represented by another matrix AB.

AB =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

B11 B12 . . . B1p

B21 B22 . . . B2p

. . .Bn1 Bn2 . . . Bnp

=

AB11 AB12 . . . AB1p

AB21 AB22 . . . AB2p

. . .ABm1 ABm2 . . . ABmp

Note that if A is m × n and B is n × p, then AB is m × p.

CS221 Linear Algebra Review 10/11/2011 16 / 37

Page 56: Cs221 linear algebra

Matrices

Matrix Product

Applying two linear operators A and B de�nes another linear operator,which can be represented by another matrix AB.

AB =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

B11 B12 . . . B1p

B21 B22 . . . B2p

. . .Bn1 Bn2 . . . Bnp

=

AB11 AB12 . . . AB1p

AB21 AB22 . . . AB2p

. . .ABm1 ABm2 . . . ABmp

Note that if A is m × n and B is n × p, then AB is m × p.

CS221 Linear Algebra Review 10/11/2011 16 / 37

Page 57: Cs221 linear algebra

Matrices

Matrix Product

Applying two linear operators A and B de�nes another linear operator,which can be represented by another matrix AB.

AB =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

B11 B12 . . . B1p

B21 B22 . . . B2p

. . .Bn1 Bn2 . . . Bnp

=

AB11 AB12 . . . AB1p

AB21 AB22 . . . AB2p

. . .ABm1 ABm2 . . . ABmp

Note that if A is m × n and B is n × p, then AB is m × p.

CS221 Linear Algebra Review 10/11/2011 16 / 37

Page 58: Cs221 linear algebra

Matrices

Matrix Product

Applying two linear operators A and B de�nes another linear operator,which can be represented by another matrix AB.

AB =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

B11 B12 . . . B1p

B21 B22 . . . B2p

. . .Bn1 Bn2 . . . Bnp

=

AB11 AB12 . . . AB1p

AB21 AB22 . . . AB2p

. . .ABm1 ABm2 . . . ABmp

Note that if A is m × n and B is n × p, then AB is m × p.

CS221 Linear Algebra Review 10/11/2011 16 / 37

Page 59: Cs221 linear algebra

Matrices

Matrix Product

Applying two linear operators A and B de�nes another linear operator,which can be represented by another matrix AB.

AB =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

B11 B12 . . . B1p

B21 B22 . . . B2p

. . .Bn1 Bn2 . . . Bnp

=

AB11 AB12 . . . AB1p

AB21 AB22 . . . AB2p

. . .ABm1 ABm2 . . . ABmp

Note that if A is m × n and B is n × p, then AB is m × p.

CS221 Linear Algebra Review 10/11/2011 16 / 37

Page 60: Cs221 linear algebra

Matrices

Transpose of a Matrix

The transpose of a matrix is obtained by �ipping the rows andcolumns.

AT =

A11 A21 . . . An1

A12 A22 . . . An2

. . .A1m A2m . . . Anm

Some simple properties :

(AT)T

= A

(A + B)T = AT + BT

(AB)T = BTAT

The inner product for vectors can be represented as 〈u, v〉 = uTv.

CS221 Linear Algebra Review 10/11/2011 17 / 37

Page 61: Cs221 linear algebra

Matrices

Transpose of a Matrix

The transpose of a matrix is obtained by �ipping the rows andcolumns.

AT =

A11 A21 . . . An1

A12 A22 . . . An2

. . .A1m A2m . . . Anm

Some simple properties :

(AT)T

= A

(A + B)T = AT + BT

(AB)T = BTAT

The inner product for vectors can be represented as 〈u, v〉 = uTv.

CS221 Linear Algebra Review 10/11/2011 17 / 37

Page 62: Cs221 linear algebra

Matrices

Transpose of a Matrix

The transpose of a matrix is obtained by �ipping the rows andcolumns.

AT =

A11 A21 . . . An1

A12 A22 . . . An2

. . .A1m A2m . . . Anm

Some simple properties :(

AT)T

= A

(A + B)T = AT + BT

(AB)T = BTAT

The inner product for vectors can be represented as 〈u, v〉 = uTv.

CS221 Linear Algebra Review 10/11/2011 17 / 37

Page 63: Cs221 linear algebra

Matrices

Transpose of a Matrix

The transpose of a matrix is obtained by �ipping the rows andcolumns.

AT =

A11 A21 . . . An1

A12 A22 . . . An2

. . .A1m A2m . . . Anm

Some simple properties :(

AT)T

= A

(A + B)T = AT + BT

(AB)T = BTAT

The inner product for vectors can be represented as 〈u, v〉 = uTv.

CS221 Linear Algebra Review 10/11/2011 17 / 37

Page 64: Cs221 linear algebra

Matrices

Transpose of a Matrix

The transpose of a matrix is obtained by �ipping the rows andcolumns.

AT =

A11 A21 . . . An1

A12 A22 . . . An2

. . .A1m A2m . . . Anm

Some simple properties :(

AT)T

= A

(A + B)T = AT + BT

(AB)T = BTAT

The inner product for vectors can be represented as 〈u, v〉 = uTv.

CS221 Linear Algebra Review 10/11/2011 17 / 37

Page 65: Cs221 linear algebra

Matrices

Transpose of a Matrix

The transpose of a matrix is obtained by �ipping the rows andcolumns.

AT =

A11 A21 . . . An1

A12 A22 . . . An2

. . .A1m A2m . . . Anm

Some simple properties :(

AT)T

= A

(A + B)T = AT + BT

(AB)T = BTAT

The inner product for vectors can be represented as 〈u, v〉 = uTv.

CS221 Linear Algebra Review 10/11/2011 17 / 37

Page 66: Cs221 linear algebra

Matrices

Identity Operator

I =

1 0 . . . 00 1 . . . 0

. . .0 0 . . . 1

For any matrix A, AI = A.

I is the identity operator for the matrix product.

CS221 Linear Algebra Review 10/11/2011 18 / 37

Page 67: Cs221 linear algebra

Matrices

Identity Operator

I =

1 0 . . . 00 1 . . . 0

. . .0 0 . . . 1

For any matrix A, AI = A.

I is the identity operator for the matrix product.

CS221 Linear Algebra Review 10/11/2011 18 / 37

Page 68: Cs221 linear algebra

Matrices

Identity Operator

I =

1 0 . . . 00 1 . . . 0

. . .0 0 . . . 1

For any matrix A, AI = A.

I is the identity operator for the matrix product.

CS221 Linear Algebra Review 10/11/2011 18 / 37

Page 69: Cs221 linear algebra

Matrices

Column Space, Row Space and Rank

Let A be an m × n matrix.

Column space :

Span of the columns of A.Linear subspace of Rm.

Row space :

Span of the rows of A.Linear subspace of Rn.

Important fact :The column and row spaces of any matrix have the same dimension.

This dimension is the rank of the matrix.

CS221 Linear Algebra Review 10/11/2011 19 / 37

Page 70: Cs221 linear algebra

Matrices

Column Space, Row Space and Rank

Let A be an m × n matrix.

Column space :

Span of the columns of A.Linear subspace of Rm.

Row space :

Span of the rows of A.Linear subspace of Rn.

Important fact :The column and row spaces of any matrix have the same dimension.

This dimension is the rank of the matrix.

CS221 Linear Algebra Review 10/11/2011 19 / 37

Page 71: Cs221 linear algebra

Matrices

Column Space, Row Space and Rank

Let A be an m × n matrix.

Column space :

Span of the columns of A.

Linear subspace of Rm.

Row space :

Span of the rows of A.Linear subspace of Rn.

Important fact :The column and row spaces of any matrix have the same dimension.

This dimension is the rank of the matrix.

CS221 Linear Algebra Review 10/11/2011 19 / 37

Page 72: Cs221 linear algebra

Matrices

Column Space, Row Space and Rank

Let A be an m × n matrix.

Column space :

Span of the columns of A.Linear subspace of Rm.

Row space :

Span of the rows of A.Linear subspace of Rn.

Important fact :The column and row spaces of any matrix have the same dimension.

This dimension is the rank of the matrix.

CS221 Linear Algebra Review 10/11/2011 19 / 37

Page 73: Cs221 linear algebra

Matrices

Column Space, Row Space and Rank

Let A be an m × n matrix.

Column space :

Span of the columns of A.Linear subspace of Rm.

Row space :

Span of the rows of A.Linear subspace of Rn.

Important fact :The column and row spaces of any matrix have the same dimension.

This dimension is the rank of the matrix.

CS221 Linear Algebra Review 10/11/2011 19 / 37

Page 74: Cs221 linear algebra

Matrices

Column Space, Row Space and Rank

Let A be an m × n matrix.

Column space :

Span of the columns of A.Linear subspace of Rm.

Row space :

Span of the rows of A.

Linear subspace of Rn.

Important fact :The column and row spaces of any matrix have the same dimension.

This dimension is the rank of the matrix.

CS221 Linear Algebra Review 10/11/2011 19 / 37

Page 75: Cs221 linear algebra

Matrices

Column Space, Row Space and Rank

Let A be an m × n matrix.

Column space :

Span of the columns of A.Linear subspace of Rm.

Row space :

Span of the rows of A.Linear subspace of Rn.

Important fact :The column and row spaces of any matrix have the same dimension.

This dimension is the rank of the matrix.

CS221 Linear Algebra Review 10/11/2011 19 / 37

Page 76: Cs221 linear algebra

Matrices

Column Space, Row Space and Rank

Let A be an m × n matrix.

Column space :

Span of the columns of A.Linear subspace of Rm.

Row space :

Span of the rows of A.Linear subspace of Rn.

Important fact :The column and row spaces of any matrix have the same dimension.

This dimension is the rank of the matrix.

CS221 Linear Algebra Review 10/11/2011 19 / 37

Page 77: Cs221 linear algebra

Matrices

Column Space, Row Space and Rank

Let A be an m × n matrix.

Column space :

Span of the columns of A.Linear subspace of Rm.

Row space :

Span of the rows of A.Linear subspace of Rn.

Important fact :The column and row spaces of any matrix have the same dimension.

This dimension is the rank of the matrix.

CS221 Linear Algebra Review 10/11/2011 19 / 37

Page 78: Cs221 linear algebra

Matrices

Range and Null space

Let A be an m × n matrix.

Range :

Set of vectors equal to Au for some u ∈ Rn.Linear subspace of Rm.

Null space :

u belongs to the null space if Au = 0.Subspace of Rn.Every vector in the null space is orthogonal to the rows of A.The null space and row space of a matrix are orthogonal.

CS221 Linear Algebra Review 10/11/2011 20 / 37

Page 79: Cs221 linear algebra

Matrices

Range and Null space

Let A be an m × n matrix.

Range :

Set of vectors equal to Au for some u ∈ Rn.Linear subspace of Rm.

Null space :

u belongs to the null space if Au = 0.Subspace of Rn.Every vector in the null space is orthogonal to the rows of A.The null space and row space of a matrix are orthogonal.

CS221 Linear Algebra Review 10/11/2011 20 / 37

Page 80: Cs221 linear algebra

Matrices

Range and Null space

Let A be an m × n matrix.

Range :

Set of vectors equal to Au for some u ∈ Rn.

Linear subspace of Rm.

Null space :

u belongs to the null space if Au = 0.Subspace of Rn.Every vector in the null space is orthogonal to the rows of A.The null space and row space of a matrix are orthogonal.

CS221 Linear Algebra Review 10/11/2011 20 / 37

Page 81: Cs221 linear algebra

Matrices

Range and Null space

Let A be an m × n matrix.

Range :

Set of vectors equal to Au for some u ∈ Rn.Linear subspace of Rm.

Null space :

u belongs to the null space if Au = 0.Subspace of Rn.Every vector in the null space is orthogonal to the rows of A.The null space and row space of a matrix are orthogonal.

CS221 Linear Algebra Review 10/11/2011 20 / 37

Page 82: Cs221 linear algebra

Matrices

Range and Null space

Let A be an m × n matrix.

Range :

Set of vectors equal to Au for some u ∈ Rn.Linear subspace of Rm.

Null space :

u belongs to the null space if Au = 0.Subspace of Rn.Every vector in the null space is orthogonal to the rows of A.The null space and row space of a matrix are orthogonal.

CS221 Linear Algebra Review 10/11/2011 20 / 37

Page 83: Cs221 linear algebra

Matrices

Range and Null space

Let A be an m × n matrix.

Range :

Set of vectors equal to Au for some u ∈ Rn.Linear subspace of Rm.

Null space :

u belongs to the null space if Au = 0.

Subspace of Rn.Every vector in the null space is orthogonal to the rows of A.The null space and row space of a matrix are orthogonal.

CS221 Linear Algebra Review 10/11/2011 20 / 37

Page 84: Cs221 linear algebra

Matrices

Range and Null space

Let A be an m × n matrix.

Range :

Set of vectors equal to Au for some u ∈ Rn.Linear subspace of Rm.

Null space :

u belongs to the null space if Au = 0.Subspace of Rn.

Every vector in the null space is orthogonal to the rows of A.The null space and row space of a matrix are orthogonal.

CS221 Linear Algebra Review 10/11/2011 20 / 37

Page 85: Cs221 linear algebra

Matrices

Range and Null space

Let A be an m × n matrix.

Range :

Set of vectors equal to Au for some u ∈ Rn.Linear subspace of Rm.

Null space :

u belongs to the null space if Au = 0.Subspace of Rn.Every vector in the null space is orthogonal to the rows of A.The null space and row space of a matrix are orthogonal.

CS221 Linear Algebra Review 10/11/2011 20 / 37

Page 86: Cs221 linear algebra

Matrices

Range and Column Space

Another interpretation of the matrix vector product (Ai is column i of A) :

Au =

(A1 A2 . . . An

)u1u2. . .un

= u1A1 + u2A2 + · · ·+ unAn

The result is a linear combination of the columns of A.

For any matrix, the range is equal to the column space.

CS221 Linear Algebra Review 10/11/2011 21 / 37

Page 87: Cs221 linear algebra

Matrices

Range and Column Space

Another interpretation of the matrix vector product (Ai is column i of A) :

Au =

(A1 A2 . . . An

)u1u2. . .un

= u1A1 + u2A2 + · · ·+ unAn

The result is a linear combination of the columns of A.

For any matrix, the range is equal to the column space.

CS221 Linear Algebra Review 10/11/2011 21 / 37

Page 88: Cs221 linear algebra

Matrices

Range and Column Space

Another interpretation of the matrix vector product (Ai is column i of A) :

Au =

(A1 A2 . . . An

)u1u2. . .un

= u1A1 + u2A2 + · · ·+ unAn

The result is a linear combination of the columns of A.

For any matrix, the range is equal to the column space.

CS221 Linear Algebra Review 10/11/2011 21 / 37

Page 89: Cs221 linear algebra

Matrices

Matrix Inverse

For an n × n matrix A : rank + dim(null space) = n.

If dim(null space)=0 then A is full rank.

In this case, the action of the matrix is invertible.

The inversion is also linear and consequently can be represented byanother matrix A−1.

A−1 is the only matrix such that A−1A = AA−1 = I.

CS221 Linear Algebra Review 10/11/2011 22 / 37

Page 90: Cs221 linear algebra

Matrices

Matrix Inverse

For an n × n matrix A : rank + dim(null space) = n.

If dim(null space)=0 then A is full rank.

In this case, the action of the matrix is invertible.

The inversion is also linear and consequently can be represented byanother matrix A−1.

A−1 is the only matrix such that A−1A = AA−1 = I.

CS221 Linear Algebra Review 10/11/2011 22 / 37

Page 91: Cs221 linear algebra

Matrices

Matrix Inverse

For an n × n matrix A : rank + dim(null space) = n.

If dim(null space)=0 then A is full rank.

In this case, the action of the matrix is invertible.

The inversion is also linear and consequently can be represented byanother matrix A−1.

A−1 is the only matrix such that A−1A = AA−1 = I.

CS221 Linear Algebra Review 10/11/2011 22 / 37

Page 92: Cs221 linear algebra

Matrices

Matrix Inverse

For an n × n matrix A : rank + dim(null space) = n.

If dim(null space)=0 then A is full rank.

In this case, the action of the matrix is invertible.

The inversion is also linear and consequently can be represented byanother matrix A−1.

A−1 is the only matrix such that A−1A = AA−1 = I.

CS221 Linear Algebra Review 10/11/2011 22 / 37

Page 93: Cs221 linear algebra

Matrices

Matrix Inverse

For an n × n matrix A : rank + dim(null space) = n.

If dim(null space)=0 then A is full rank.

In this case, the action of the matrix is invertible.

The inversion is also linear and consequently can be represented byanother matrix A−1.

A−1 is the only matrix such that A−1A = AA−1 = I.

CS221 Linear Algebra Review 10/11/2011 22 / 37

Page 94: Cs221 linear algebra

Matrices

Orthogonal Matrices

An orthogonal matrix U satis�es UTU = I.

Equivalently, U has orthonormal columns.

Applying an orthogonal matrix to two vectors does not change theirinner product :

〈Uu,Uv〉 = (Uu)T Uv

= uTUTUv

= uTv

= 〈u, v〉

Example : Matrices that represent rotations are orthogonal.

CS221 Linear Algebra Review 10/11/2011 23 / 37

Page 95: Cs221 linear algebra

Matrices

Orthogonal Matrices

An orthogonal matrix U satis�es UTU = I.

Equivalently, U has orthonormal columns.

Applying an orthogonal matrix to two vectors does not change theirinner product :

〈Uu,Uv〉 = (Uu)T Uv

= uTUTUv

= uTv

= 〈u, v〉

Example : Matrices that represent rotations are orthogonal.

CS221 Linear Algebra Review 10/11/2011 23 / 37

Page 96: Cs221 linear algebra

Matrices

Orthogonal Matrices

An orthogonal matrix U satis�es UTU = I.

Equivalently, U has orthonormal columns.

Applying an orthogonal matrix to two vectors does not change theirinner product :

〈Uu,Uv〉 = (Uu)T Uv

= uTUTUv

= uTv

= 〈u, v〉

Example : Matrices that represent rotations are orthogonal.

CS221 Linear Algebra Review 10/11/2011 23 / 37

Page 97: Cs221 linear algebra

Matrices

Orthogonal Matrices

An orthogonal matrix U satis�es UTU = I.

Equivalently, U has orthonormal columns.

Applying an orthogonal matrix to two vectors does not change theirinner product :

〈Uu,Uv〉 = (Uu)T Uv

= uTUTUv

= uTv

= 〈u, v〉

Example : Matrices that represent rotations are orthogonal.

CS221 Linear Algebra Review 10/11/2011 23 / 37

Page 98: Cs221 linear algebra

Matrices

Trace and Determinant

The trace is the sum of the diagonal elements of a square matrix.

The determinant of a square matrix A is denoted by |A|.For a 2× 2 matrix A =

(a bc d

):

|A| = ad − bc

The absolute value of |A| is the area of the parallelogram given by therows of A.

CS221 Linear Algebra Review 10/11/2011 24 / 37

Page 99: Cs221 linear algebra

Matrices

Trace and Determinant

The trace is the sum of the diagonal elements of a square matrix.

The determinant of a square matrix A is denoted by |A|.

For a 2× 2 matrix A =(a bc d

):

|A| = ad − bc

The absolute value of |A| is the area of the parallelogram given by therows of A.

CS221 Linear Algebra Review 10/11/2011 24 / 37

Page 100: Cs221 linear algebra

Matrices

Trace and Determinant

The trace is the sum of the diagonal elements of a square matrix.

The determinant of a square matrix A is denoted by |A|.For a 2× 2 matrix A =

(a bc d

):

|A| = ad − bc

The absolute value of |A| is the area of the parallelogram given by therows of A.

CS221 Linear Algebra Review 10/11/2011 24 / 37

Page 101: Cs221 linear algebra

Matrices

Trace and Determinant

The trace is the sum of the diagonal elements of a square matrix.

The determinant of a square matrix A is denoted by |A|.For a 2× 2 matrix A =

(a bc d

):

|A| = ad − bc

The absolute value of |A| is the area of the parallelogram given by therows of A.

CS221 Linear Algebra Review 10/11/2011 24 / 37

Page 102: Cs221 linear algebra

Matrices

Trace and Determinant

The trace is the sum of the diagonal elements of a square matrix.

The determinant of a square matrix A is denoted by |A|.For a 2× 2 matrix A =

(a bc d

):

|A| = ad − bc

The absolute value of |A| is the area of the parallelogram given by therows of A.

CS221 Linear Algebra Review 10/11/2011 24 / 37

Page 103: Cs221 linear algebra

Matrices

Properties of the Determinant

The de�nition can be generalized to larger matrices.

|A| =∣∣AT

∣∣.|AB| = |A| |B||A| = 0 if and only if A is not invertible.

If A is invertible, then∣∣A−1∣∣ = 1

|A| .

CS221 Linear Algebra Review 10/11/2011 25 / 37

Page 104: Cs221 linear algebra

Matrices

Properties of the Determinant

The de�nition can be generalized to larger matrices.

|A| =∣∣AT

∣∣.

|AB| = |A| |B||A| = 0 if and only if A is not invertible.

If A is invertible, then∣∣A−1∣∣ = 1

|A| .

CS221 Linear Algebra Review 10/11/2011 25 / 37

Page 105: Cs221 linear algebra

Matrices

Properties of the Determinant

The de�nition can be generalized to larger matrices.

|A| =∣∣AT

∣∣.|AB| = |A| |B|

|A| = 0 if and only if A is not invertible.

If A is invertible, then∣∣A−1∣∣ = 1

|A| .

CS221 Linear Algebra Review 10/11/2011 25 / 37

Page 106: Cs221 linear algebra

Matrices

Properties of the Determinant

The de�nition can be generalized to larger matrices.

|A| =∣∣AT

∣∣.|AB| = |A| |B||A| = 0 if and only if A is not invertible.

If A is invertible, then∣∣A−1∣∣ = 1

|A| .

CS221 Linear Algebra Review 10/11/2011 25 / 37

Page 107: Cs221 linear algebra

Matrices

Properties of the Determinant

The de�nition can be generalized to larger matrices.

|A| =∣∣AT

∣∣.|AB| = |A| |B||A| = 0 if and only if A is not invertible.

If A is invertible, then∣∣A−1∣∣ = 1

|A| .

CS221 Linear Algebra Review 10/11/2011 25 / 37

Page 108: Cs221 linear algebra

Matrix Decompositions

1 Vectors

2 Matrices

3 Matrix Decompositions

4 Application : Image Compression

CS221 Linear Algebra Review 10/11/2011 26 / 37

Page 109: Cs221 linear algebra

Matrix Decompositions

Eigenvalues and Eigenvectors

An eigenvalue λ of a square matrix A satis�es :

Au = λu

for some vector u, which we call an eigenvector.

Geometrically, the operator A expands (λ > 1) or contracts (λ < 1)eigenvectors but does not rotate them.

If u is an eigenvector of A, it is in the null space of A− λI, which isconsequently not invertible.

The eigenvalues of A are the roots of the equation |A− λI| = 0.

Not the way eigenvalues are calculated in practice.

Eigenvalues and eigenvectors can be complex valued, even if all theentries of A are real.

CS221 Linear Algebra Review 10/11/2011 27 / 37

Page 110: Cs221 linear algebra

Matrix Decompositions

Eigenvalues and Eigenvectors

An eigenvalue λ of a square matrix A satis�es :

Au = λu

for some vector u, which we call an eigenvector.

Geometrically, the operator A expands (λ > 1) or contracts (λ < 1)eigenvectors but does not rotate them.

If u is an eigenvector of A, it is in the null space of A− λI, which isconsequently not invertible.

The eigenvalues of A are the roots of the equation |A− λI| = 0.

Not the way eigenvalues are calculated in practice.

Eigenvalues and eigenvectors can be complex valued, even if all theentries of A are real.

CS221 Linear Algebra Review 10/11/2011 27 / 37

Page 111: Cs221 linear algebra

Matrix Decompositions

Eigenvalues and Eigenvectors

An eigenvalue λ of a square matrix A satis�es :

Au = λu

for some vector u, which we call an eigenvector.

Geometrically, the operator A expands (λ > 1) or contracts (λ < 1)eigenvectors but does not rotate them.

If u is an eigenvector of A, it is in the null space of A− λI, which isconsequently not invertible.

The eigenvalues of A are the roots of the equation |A− λI| = 0.

Not the way eigenvalues are calculated in practice.

Eigenvalues and eigenvectors can be complex valued, even if all theentries of A are real.

CS221 Linear Algebra Review 10/11/2011 27 / 37

Page 112: Cs221 linear algebra

Matrix Decompositions

Eigenvalues and Eigenvectors

An eigenvalue λ of a square matrix A satis�es :

Au = λu

for some vector u, which we call an eigenvector.

Geometrically, the operator A expands (λ > 1) or contracts (λ < 1)eigenvectors but does not rotate them.

If u is an eigenvector of A, it is in the null space of A− λI, which isconsequently not invertible.

The eigenvalues of A are the roots of the equation |A− λI| = 0.

Not the way eigenvalues are calculated in practice.

Eigenvalues and eigenvectors can be complex valued, even if all theentries of A are real.

CS221 Linear Algebra Review 10/11/2011 27 / 37

Page 113: Cs221 linear algebra

Matrix Decompositions

Eigenvalues and Eigenvectors

An eigenvalue λ of a square matrix A satis�es :

Au = λu

for some vector u, which we call an eigenvector.

Geometrically, the operator A expands (λ > 1) or contracts (λ < 1)eigenvectors but does not rotate them.

If u is an eigenvector of A, it is in the null space of A− λI, which isconsequently not invertible.

The eigenvalues of A are the roots of the equation |A− λI| = 0.

Not the way eigenvalues are calculated in practice.

Eigenvalues and eigenvectors can be complex valued, even if all theentries of A are real.

CS221 Linear Algebra Review 10/11/2011 27 / 37

Page 114: Cs221 linear algebra

Matrix Decompositions

Eigenvalues and Eigenvectors

An eigenvalue λ of a square matrix A satis�es :

Au = λu

for some vector u, which we call an eigenvector.

Geometrically, the operator A expands (λ > 1) or contracts (λ < 1)eigenvectors but does not rotate them.

If u is an eigenvector of A, it is in the null space of A− λI, which isconsequently not invertible.

The eigenvalues of A are the roots of the equation |A− λI| = 0.

Not the way eigenvalues are calculated in practice.

Eigenvalues and eigenvectors can be complex valued, even if all theentries of A are real.

CS221 Linear Algebra Review 10/11/2011 27 / 37

Page 115: Cs221 linear algebra

Matrix Decompositions

Eigendecomposition of a Matrix

Let A be an n × n square matrix with n linearly independenteigenvectors p1 . . .pn and eigenvalues λ1 . . . λn.

We de�ne the matrices :

P =

(p1 . . . pn

)

Λ =

λ1 0 . . . 00 λ2 . . . 0

. . .0 0 . . . λn

A satis�es AP = PΛ.

P is full rank. Inverting it yields the eigendecomposition of A :

A = PΛP−1

CS221 Linear Algebra Review 10/11/2011 28 / 37

Page 116: Cs221 linear algebra

Matrix Decompositions

Eigendecomposition of a Matrix

Let A be an n × n square matrix with n linearly independenteigenvectors p1 . . .pn and eigenvalues λ1 . . . λn.

We de�ne the matrices :

P =

(p1 . . . pn

)

Λ =

λ1 0 . . . 00 λ2 . . . 0

. . .0 0 . . . λn

A satis�es AP = PΛ.

P is full rank. Inverting it yields the eigendecomposition of A :

A = PΛP−1

CS221 Linear Algebra Review 10/11/2011 28 / 37

Page 117: Cs221 linear algebra

Matrix Decompositions

Eigendecomposition of a Matrix

Let A be an n × n square matrix with n linearly independenteigenvectors p1 . . .pn and eigenvalues λ1 . . . λn.

We de�ne the matrices :

P =

(p1 . . . pn

)

Λ =

λ1 0 . . . 00 λ2 . . . 0

. . .0 0 . . . λn

A satis�es AP = PΛ.

P is full rank. Inverting it yields the eigendecomposition of A :

A = PΛP−1

CS221 Linear Algebra Review 10/11/2011 28 / 37

Page 118: Cs221 linear algebra

Matrix Decompositions

Eigendecomposition of a Matrix

Let A be an n × n square matrix with n linearly independenteigenvectors p1 . . .pn and eigenvalues λ1 . . . λn.

We de�ne the matrices :

P =

(p1 . . . pn

)

Λ =

λ1 0 . . . 00 λ2 . . . 0

. . .0 0 . . . λn

A satis�es AP = PΛ.

P is full rank. Inverting it yields the eigendecomposition of A :

A = PΛP−1

CS221 Linear Algebra Review 10/11/2011 28 / 37

Page 119: Cs221 linear algebra

Matrix Decompositions

Properties of the Eigendecomposition

Not all matrices are diagonalizable (have an eigendecomposition).Example : ( 1 1

0 1).

Trace(A) = Trace(Λ) =∑n

i=1λi .

|A| = |Λ| =∏n

i=1λi .

The rank of A is equal to the number of nonzero eigenvalues.

If λ is a nonzero eigenvalue of A, 1

λ is an eigenvalue of A−1 with thesame eigenvector.

The eigendecomposition makes it possible to compute matrix powerse�ciently :Am =

(PΛP−1

)m= PΛP−1 PΛP−1 . . .PΛP−1 = PΛmP−1.

CS221 Linear Algebra Review 10/11/2011 29 / 37

Page 120: Cs221 linear algebra

Matrix Decompositions

Properties of the Eigendecomposition

Not all matrices are diagonalizable (have an eigendecomposition).Example : ( 1 1

0 1).

Trace(A) = Trace(Λ) =∑n

i=1λi .

|A| = |Λ| =∏n

i=1λi .

The rank of A is equal to the number of nonzero eigenvalues.

If λ is a nonzero eigenvalue of A, 1

λ is an eigenvalue of A−1 with thesame eigenvector.

The eigendecomposition makes it possible to compute matrix powerse�ciently :Am =

(PΛP−1

)m= PΛP−1 PΛP−1 . . .PΛP−1 = PΛmP−1.

CS221 Linear Algebra Review 10/11/2011 29 / 37

Page 121: Cs221 linear algebra

Matrix Decompositions

Properties of the Eigendecomposition

Not all matrices are diagonalizable (have an eigendecomposition).Example : ( 1 1

0 1).

Trace(A) = Trace(Λ) =∑n

i=1λi .

|A| = |Λ| =∏n

i=1λi .

The rank of A is equal to the number of nonzero eigenvalues.

If λ is a nonzero eigenvalue of A, 1

λ is an eigenvalue of A−1 with thesame eigenvector.

The eigendecomposition makes it possible to compute matrix powerse�ciently :Am =

(PΛP−1

)m= PΛP−1 PΛP−1 . . .PΛP−1 = PΛmP−1.

CS221 Linear Algebra Review 10/11/2011 29 / 37

Page 122: Cs221 linear algebra

Matrix Decompositions

Properties of the Eigendecomposition

Not all matrices are diagonalizable (have an eigendecomposition).Example : ( 1 1

0 1).

Trace(A) = Trace(Λ) =∑n

i=1λi .

|A| = |Λ| =∏n

i=1λi .

The rank of A is equal to the number of nonzero eigenvalues.

If λ is a nonzero eigenvalue of A, 1

λ is an eigenvalue of A−1 with thesame eigenvector.

The eigendecomposition makes it possible to compute matrix powerse�ciently :Am =

(PΛP−1

)m= PΛP−1 PΛP−1 . . .PΛP−1 = PΛmP−1.

CS221 Linear Algebra Review 10/11/2011 29 / 37

Page 123: Cs221 linear algebra

Matrix Decompositions

Properties of the Eigendecomposition

Not all matrices are diagonalizable (have an eigendecomposition).Example : ( 1 1

0 1).

Trace(A) = Trace(Λ) =∑n

i=1λi .

|A| = |Λ| =∏n

i=1λi .

The rank of A is equal to the number of nonzero eigenvalues.

If λ is a nonzero eigenvalue of A, 1

λ is an eigenvalue of A−1 with thesame eigenvector.

The eigendecomposition makes it possible to compute matrix powerse�ciently :Am =

(PΛP−1

)m= PΛP−1 PΛP−1 . . .PΛP−1 = PΛmP−1.

CS221 Linear Algebra Review 10/11/2011 29 / 37

Page 124: Cs221 linear algebra

Matrix Decompositions

Properties of the Eigendecomposition

Not all matrices are diagonalizable (have an eigendecomposition).Example : ( 1 1

0 1).

Trace(A) = Trace(Λ) =∑n

i=1λi .

|A| = |Λ| =∏n

i=1λi .

The rank of A is equal to the number of nonzero eigenvalues.

If λ is a nonzero eigenvalue of A, 1

λ is an eigenvalue of A−1 with thesame eigenvector.

The eigendecomposition makes it possible to compute matrix powerse�ciently :Am =

(PΛP−1

)m= PΛP−1 PΛP−1 . . .PΛP−1 = PΛmP−1.

CS221 Linear Algebra Review 10/11/2011 29 / 37

Page 125: Cs221 linear algebra

Matrix Decompositions

Matlab Example

CS221 Linear Algebra Review 10/11/2011 30 / 37

Page 126: Cs221 linear algebra

Matrix Decompositions

Eigendecomposition of a Symmetric Matrix

If A = AT then A is symmetric.

The eigenvalues of symmetric matrices are real.

The eigenvectors of symmetric matrices are orthonormal.

Consequently, the eigendecomposition becomes :A = UΛUT for Λ real and U orthogonal.

The eigenvectors of A are an orthonormal basis for the column space(and the row space, since they are equal).

CS221 Linear Algebra Review 10/11/2011 31 / 37

Page 127: Cs221 linear algebra

Matrix Decompositions

Eigendecomposition of a Symmetric Matrix

If A = AT then A is symmetric.

The eigenvalues of symmetric matrices are real.

The eigenvectors of symmetric matrices are orthonormal.

Consequently, the eigendecomposition becomes :A = UΛUT for Λ real and U orthogonal.

The eigenvectors of A are an orthonormal basis for the column space(and the row space, since they are equal).

CS221 Linear Algebra Review 10/11/2011 31 / 37

Page 128: Cs221 linear algebra

Matrix Decompositions

Eigendecomposition of a Symmetric Matrix

If A = AT then A is symmetric.

The eigenvalues of symmetric matrices are real.

The eigenvectors of symmetric matrices are orthonormal.

Consequently, the eigendecomposition becomes :A = UΛUT for Λ real and U orthogonal.

The eigenvectors of A are an orthonormal basis for the column space(and the row space, since they are equal).

CS221 Linear Algebra Review 10/11/2011 31 / 37

Page 129: Cs221 linear algebra

Matrix Decompositions

Eigendecomposition of a Symmetric Matrix

If A = AT then A is symmetric.

The eigenvalues of symmetric matrices are real.

The eigenvectors of symmetric matrices are orthonormal.

Consequently, the eigendecomposition becomes :A = UΛUT for Λ real and U orthogonal.

The eigenvectors of A are an orthonormal basis for the column space(and the row space, since they are equal).

CS221 Linear Algebra Review 10/11/2011 31 / 37

Page 130: Cs221 linear algebra

Matrix Decompositions

Eigendecomposition of a Symmetric Matrix

If A = AT then A is symmetric.

The eigenvalues of symmetric matrices are real.

The eigenvectors of symmetric matrices are orthonormal.

Consequently, the eigendecomposition becomes :A = UΛUT for Λ real and U orthogonal.

The eigenvectors of A are an orthonormal basis for the column space(and the row space, since they are equal).

CS221 Linear Algebra Review 10/11/2011 31 / 37

Page 131: Cs221 linear algebra

Matrix Decompositions

Eigendecomposition of a Symmetric Matrix

The action of a symmetric matrix on a vector Au = UΛUTu can bedecomposed into :

1 Projection of u onto the column space of A (multiplication by UT ).2 Scaling of each coe�cient 〈Ui ,u〉 by the corresponding eigenvalue

(multiplication by Λ).3 Linear combination of the eigenvectors scaled by the resulting

coe�cient (multiplication by U).

Summarizing :

Au =n∑

i=1

λi 〈Ui ,u〉Ui

It would be great to generalize this to all matrices...

CS221 Linear Algebra Review 10/11/2011 32 / 37

Page 132: Cs221 linear algebra

Matrix Decompositions

Eigendecomposition of a Symmetric Matrix

The action of a symmetric matrix on a vector Au = UΛUTu can bedecomposed into :

1 Projection of u onto the column space of A (multiplication by UT ).

2 Scaling of each coe�cient 〈Ui ,u〉 by the corresponding eigenvalue(multiplication by Λ).

3 Linear combination of the eigenvectors scaled by the resultingcoe�cient (multiplication by U).

Summarizing :

Au =n∑

i=1

λi 〈Ui ,u〉Ui

It would be great to generalize this to all matrices...

CS221 Linear Algebra Review 10/11/2011 32 / 37

Page 133: Cs221 linear algebra

Matrix Decompositions

Eigendecomposition of a Symmetric Matrix

The action of a symmetric matrix on a vector Au = UΛUTu can bedecomposed into :

1 Projection of u onto the column space of A (multiplication by UT ).2 Scaling of each coe�cient 〈Ui ,u〉 by the corresponding eigenvalue

(multiplication by Λ).

3 Linear combination of the eigenvectors scaled by the resultingcoe�cient (multiplication by U).

Summarizing :

Au =n∑

i=1

λi 〈Ui ,u〉Ui

It would be great to generalize this to all matrices...

CS221 Linear Algebra Review 10/11/2011 32 / 37

Page 134: Cs221 linear algebra

Matrix Decompositions

Eigendecomposition of a Symmetric Matrix

The action of a symmetric matrix on a vector Au = UΛUTu can bedecomposed into :

1 Projection of u onto the column space of A (multiplication by UT ).2 Scaling of each coe�cient 〈Ui ,u〉 by the corresponding eigenvalue

(multiplication by Λ).3 Linear combination of the eigenvectors scaled by the resulting

coe�cient (multiplication by U).

Summarizing :

Au =n∑

i=1

λi 〈Ui ,u〉Ui

It would be great to generalize this to all matrices...

CS221 Linear Algebra Review 10/11/2011 32 / 37

Page 135: Cs221 linear algebra

Matrix Decompositions

Eigendecomposition of a Symmetric Matrix

The action of a symmetric matrix on a vector Au = UΛUTu can bedecomposed into :

1 Projection of u onto the column space of A (multiplication by UT ).2 Scaling of each coe�cient 〈Ui ,u〉 by the corresponding eigenvalue

(multiplication by Λ).3 Linear combination of the eigenvectors scaled by the resulting

coe�cient (multiplication by U).

Summarizing :

Au =n∑

i=1

λi 〈Ui ,u〉Ui

It would be great to generalize this to all matrices...

CS221 Linear Algebra Review 10/11/2011 32 / 37

Page 136: Cs221 linear algebra

Matrix Decompositions

Eigendecomposition of a Symmetric Matrix

The action of a symmetric matrix on a vector Au = UΛUTu can bedecomposed into :

1 Projection of u onto the column space of A (multiplication by UT ).2 Scaling of each coe�cient 〈Ui ,u〉 by the corresponding eigenvalue

(multiplication by Λ).3 Linear combination of the eigenvectors scaled by the resulting

coe�cient (multiplication by U).

Summarizing :

Au =n∑

i=1

λi 〈Ui ,u〉Ui

It would be great to generalize this to all matrices...

CS221 Linear Algebra Review 10/11/2011 32 / 37

Page 137: Cs221 linear algebra

Matrix Decompositions

Singular Value Decomposition

Every matrix has a singular value decomposition :

A = UΣVT

The columns of U are an orthonormal basis for the column space ofA.

The columns of V are an orthonormal basis for the row space of A.

Σ is diagonal. The entries on its diagonal σi = Σii are positive realnumbers, called the singular values of A.

The action of A on a vector u can be decomposed into :

Au =n∑

i=1

σi 〈Vi ,u〉Ui

CS221 Linear Algebra Review 10/11/2011 33 / 37

Page 138: Cs221 linear algebra

Matrix Decompositions

Singular Value Decomposition

Every matrix has a singular value decomposition :

A = UΣVT

The columns of U are an orthonormal basis for the column space ofA.

The columns of V are an orthonormal basis for the row space of A.

Σ is diagonal. The entries on its diagonal σi = Σii are positive realnumbers, called the singular values of A.

The action of A on a vector u can be decomposed into :

Au =n∑

i=1

σi 〈Vi ,u〉Ui

CS221 Linear Algebra Review 10/11/2011 33 / 37

Page 139: Cs221 linear algebra

Matrix Decompositions

Singular Value Decomposition

Every matrix has a singular value decomposition :

A = UΣVT

The columns of U are an orthonormal basis for the column space ofA.

The columns of V are an orthonormal basis for the row space of A.

Σ is diagonal. The entries on its diagonal σi = Σii are positive realnumbers, called the singular values of A.

The action of A on a vector u can be decomposed into :

Au =n∑

i=1

σi 〈Vi ,u〉Ui

CS221 Linear Algebra Review 10/11/2011 33 / 37

Page 140: Cs221 linear algebra

Matrix Decompositions

Singular Value Decomposition

Every matrix has a singular value decomposition :

A = UΣVT

The columns of U are an orthonormal basis for the column space ofA.

The columns of V are an orthonormal basis for the row space of A.

Σ is diagonal. The entries on its diagonal σi = Σii are positive realnumbers, called the singular values of A.

The action of A on a vector u can be decomposed into :

Au =n∑

i=1

σi 〈Vi ,u〉Ui

CS221 Linear Algebra Review 10/11/2011 33 / 37

Page 141: Cs221 linear algebra

Matrix Decompositions

Singular Value Decomposition

Every matrix has a singular value decomposition :

A = UΣVT

The columns of U are an orthonormal basis for the column space ofA.

The columns of V are an orthonormal basis for the row space of A.

Σ is diagonal. The entries on its diagonal σi = Σii are positive realnumbers, called the singular values of A.

The action of A on a vector u can be decomposed into :

Au =n∑

i=1

σi 〈Vi ,u〉Ui

CS221 Linear Algebra Review 10/11/2011 33 / 37

Page 142: Cs221 linear algebra

Matrix Decompositions

Properties of the Singular Value Decomposition

The eigenvalues of the symmetric matrix ATA are equal to thesquare of the singular values of A :

ATA = VΣUTUΣVT = VΣ2VT

The rank of a matrix is equal to the number of nonzero singularvalues.

The largest singular value σ1 is the solution to the optimizationproblem :

σ1 = maxx 6=0

||Ax||2

||x||2

It can be veri�ed that the largest singular value satis�es the propertiesof a norm, it is called the spectral norm of the matrix.

In statistics analyzing data with the singular value decomposition iscalled Principal Component Analysis.

CS221 Linear Algebra Review 10/11/2011 34 / 37

Page 143: Cs221 linear algebra

Matrix Decompositions

Properties of the Singular Value Decomposition

The eigenvalues of the symmetric matrix ATA are equal to thesquare of the singular values of A :

ATA = VΣUTUΣVT = VΣ2VT

The rank of a matrix is equal to the number of nonzero singularvalues.

The largest singular value σ1 is the solution to the optimizationproblem :

σ1 = maxx 6=0

||Ax||2

||x||2

It can be veri�ed that the largest singular value satis�es the propertiesof a norm, it is called the spectral norm of the matrix.

In statistics analyzing data with the singular value decomposition iscalled Principal Component Analysis.

CS221 Linear Algebra Review 10/11/2011 34 / 37

Page 144: Cs221 linear algebra

Matrix Decompositions

Properties of the Singular Value Decomposition

The eigenvalues of the symmetric matrix ATA are equal to thesquare of the singular values of A :

ATA = VΣUTUΣVT = VΣ2VT

The rank of a matrix is equal to the number of nonzero singularvalues.

The largest singular value σ1 is the solution to the optimizationproblem :

σ1 = maxx 6=0

||Ax||2

||x||2

It can be veri�ed that the largest singular value satis�es the propertiesof a norm, it is called the spectral norm of the matrix.

In statistics analyzing data with the singular value decomposition iscalled Principal Component Analysis.

CS221 Linear Algebra Review 10/11/2011 34 / 37

Page 145: Cs221 linear algebra

Matrix Decompositions

Properties of the Singular Value Decomposition

The eigenvalues of the symmetric matrix ATA are equal to thesquare of the singular values of A :

ATA = VΣUTUΣVT = VΣ2VT

The rank of a matrix is equal to the number of nonzero singularvalues.

The largest singular value σ1 is the solution to the optimizationproblem :

σ1 = maxx 6=0

||Ax||2

||x||2

It can be veri�ed that the largest singular value satis�es the propertiesof a norm, it is called the spectral norm of the matrix.

In statistics analyzing data with the singular value decomposition iscalled Principal Component Analysis.

CS221 Linear Algebra Review 10/11/2011 34 / 37

Page 146: Cs221 linear algebra

Matrix Decompositions

Properties of the Singular Value Decomposition

The eigenvalues of the symmetric matrix ATA are equal to thesquare of the singular values of A :

ATA = VΣUTUΣVT = VΣ2VT

The rank of a matrix is equal to the number of nonzero singularvalues.

The largest singular value σ1 is the solution to the optimizationproblem :

σ1 = maxx 6=0

||Ax||2

||x||2

It can be veri�ed that the largest singular value satis�es the propertiesof a norm, it is called the spectral norm of the matrix.

In statistics analyzing data with the singular value decomposition iscalled Principal Component Analysis.

CS221 Linear Algebra Review 10/11/2011 34 / 37

Page 147: Cs221 linear algebra

Application : Image Compression

1 Vectors

2 Matrices

3 Matrix Decompositions

4 Application : Image Compression

CS221 Linear Algebra Review 10/11/2011 35 / 37

Page 148: Cs221 linear algebra

Application : Image Compression

Compression using the Singular Value Decomposition

The singular value decomposition can be viewed as a sum of rank 1matrices :

If some of the singular values are very small, we can discard them.

This is a form of lossy compression.

CS221 Linear Algebra Review 10/11/2011 36 / 37

Page 149: Cs221 linear algebra

Application : Image Compression

Compression using the Singular Value Decomposition

The singular value decomposition can be viewed as a sum of rank 1matrices :

If some of the singular values are very small, we can discard them.

This is a form of lossy compression.

CS221 Linear Algebra Review 10/11/2011 36 / 37

Page 150: Cs221 linear algebra

Application : Image Compression

Compression using the Singular Value Decomposition

The singular value decomposition can be viewed as a sum of rank 1matrices :

If some of the singular values are very small, we can discard them.

This is a form of lossy compression.

CS221 Linear Algebra Review 10/11/2011 36 / 37

Page 151: Cs221 linear algebra

Application : Image Compression

Compression using the Singular Value Decomposition

Truncating the singular value decomposition allows us to represent thematrix with less parameters.

For a 512× 512 matrix :

Full representation : 512× 512 =262 144

Rank 10 approximation : 512× 10 + 10 + 512× 10 =10 250

Rank 40 approximation : 512× 40 + 40 + 512× 40 =41 000

Rank 80 approximation : 512× 80 + 80 + 512× 80 =82 000

CS221 Linear Algebra Review 10/11/2011 37 / 37

Page 152: Cs221 linear algebra

Application : Image Compression

Compression using the Singular Value Decomposition

Truncating the singular value decomposition allows us to represent thematrix with less parameters.

For a 512× 512 matrix :

Full representation : 512× 512 =262 144

Rank 10 approximation : 512× 10 + 10 + 512× 10 =10 250

Rank 40 approximation : 512× 40 + 40 + 512× 40 =41 000

Rank 80 approximation : 512× 80 + 80 + 512× 80 =82 000

CS221 Linear Algebra Review 10/11/2011 37 / 37

Page 153: Cs221 linear algebra

Application : Image Compression

Compression using the Singular Value Decomposition

Truncating the singular value decomposition allows us to represent thematrix with less parameters.

For a 512× 512 matrix :

Full representation : 512× 512 =262 144

Rank 10 approximation : 512× 10 + 10 + 512× 10 =10 250

Rank 40 approximation : 512× 40 + 40 + 512× 40 =41 000

Rank 80 approximation : 512× 80 + 80 + 512× 80 =82 000

CS221 Linear Algebra Review 10/11/2011 37 / 37

Page 154: Cs221 linear algebra

Application : Image Compression

Compression using the Singular Value Decomposition

Truncating the singular value decomposition allows us to represent thematrix with less parameters.

For a 512× 512 matrix :

Full representation : 512× 512 =262 144

Rank 10 approximation : 512× 10 + 10 + 512× 10 =10 250

Rank 40 approximation : 512× 40 + 40 + 512× 40 =41 000

Rank 80 approximation : 512× 80 + 80 + 512× 80 =82 000

CS221 Linear Algebra Review 10/11/2011 37 / 37

Page 155: Cs221 linear algebra

Application : Image Compression

Compression using the Singular Value Decomposition

Truncating the singular value decomposition allows us to represent thematrix with less parameters.

For a 512× 512 matrix :

Full representation : 512× 512 =262 144

Rank 10 approximation : 512× 10 + 10 + 512× 10 =10 250

Rank 40 approximation : 512× 40 + 40 + 512× 40 =41 000

Rank 80 approximation : 512× 80 + 80 + 512× 80 =82 000

CS221 Linear Algebra Review 10/11/2011 37 / 37

Page 156: Cs221 linear algebra

Application : Image Compression

Compression using the Singular Value Decomposition

Truncating the singular value decomposition allows us to represent thematrix with less parameters.

For a 512× 512 matrix :

Full representation : 512× 512 =262 144

Rank 10 approximation : 512× 10 + 10 + 512× 10 =10 250

Rank 40 approximation : 512× 40 + 40 + 512× 40 =41 000

Rank 80 approximation : 512× 80 + 80 + 512× 80 =82 000

CS221 Linear Algebra Review 10/11/2011 37 / 37