# cs221 linear algebra

TRANSCRIPT

Linear Algebra Review

CS221: Introduction to Arti�cial Intelligence

Carlos Fernandez-Granda

10/11/2011

CS221 Linear Algebra Review 10/11/2011 1 / 37

Index

1 Vectors

2 Matrices

3 Matrix Decompositions

4 Application : Image Compression

CS221 Linear Algebra Review 10/11/2011 2 / 37

Vectors

1 Vectors

2 Matrices

3 Matrix Decompositions

4 Application : Image Compression

CS221 Linear Algebra Review 10/11/2011 3 / 37

Vectors

What is a vector ?

An ordered tuple of numbers :

u =

u1u2. . .un

A quantity that has a magnitude and a direction.

CS221 Linear Algebra Review 10/11/2011 4 / 37

Vectors

What is a vector ?

An ordered tuple of numbers :

u =

u1u2. . .un

A quantity that has a magnitude and a direction.

CS221 Linear Algebra Review 10/11/2011 4 / 37

Vectors

What is a vector ?

An ordered tuple of numbers :

u =

u1u2. . .un

A quantity that has a magnitude and a direction.

CS221 Linear Algebra Review 10/11/2011 4 / 37

Vectors

Vector spaces

More formally a vector is an element of a vector space.

A vector space V is a set that contains all linear combinations of itselements :

If vectors u and v ∈ V, then u + v ∈ V.If vector u ∈ V, then α u ∈ V for any scalar α.There exists 0 ∈ V such that u + 0 = u for any u ∈ V.

A subspace is a subset of a vector space that is also a vector space.

CS221 Linear Algebra Review 10/11/2011 5 / 37

Vectors

Vector spaces

More formally a vector is an element of a vector space.

A vector space V is a set that contains all linear combinations of itselements :

If vectors u and v ∈ V, then u + v ∈ V.If vector u ∈ V, then α u ∈ V for any scalar α.There exists 0 ∈ V such that u + 0 = u for any u ∈ V.

A subspace is a subset of a vector space that is also a vector space.

CS221 Linear Algebra Review 10/11/2011 5 / 37

Vectors

Vector spaces

More formally a vector is an element of a vector space.

A vector space V is a set that contains all linear combinations of itselements :

If vectors u and v ∈ V, then u + v ∈ V.

If vector u ∈ V, then α u ∈ V for any scalar α.There exists 0 ∈ V such that u + 0 = u for any u ∈ V.

A subspace is a subset of a vector space that is also a vector space.

CS221 Linear Algebra Review 10/11/2011 5 / 37

Vectors

Vector spaces

More formally a vector is an element of a vector space.

A vector space V is a set that contains all linear combinations of itselements :

If vectors u and v ∈ V, then u + v ∈ V.If vector u ∈ V, then α u ∈ V for any scalar α.

There exists 0 ∈ V such that u + 0 = u for any u ∈ V.A subspace is a subset of a vector space that is also a vector space.

CS221 Linear Algebra Review 10/11/2011 5 / 37

Vectors

Vector spaces

More formally a vector is an element of a vector space.

A vector space V is a set that contains all linear combinations of itselements :

If vectors u and v ∈ V, then u + v ∈ V.If vector u ∈ V, then α u ∈ V for any scalar α.There exists 0 ∈ V such that u + 0 = u for any u ∈ V.

A subspace is a subset of a vector space that is also a vector space.

CS221 Linear Algebra Review 10/11/2011 5 / 37

Vectors

Vector spaces

More formally a vector is an element of a vector space.

A vector space V is a set that contains all linear combinations of itselements :

A subspace is a subset of a vector space that is also a vector space.

CS221 Linear Algebra Review 10/11/2011 5 / 37

Vectors

Examples

Euclidean space Rn.

The span of any set of vectors {u1,u2, . . . ,un}, de�ned as :

span (u1,u2, . . . ,un) = {α1u1 + α2u2 + · · ·+ αnun | αi ∈ R}

CS221 Linear Algebra Review 10/11/2011 6 / 37

Vectors

Examples

Euclidean space Rn.

The span of any set of vectors {u1,u2, . . . ,un}, de�ned as :

span (u1,u2, . . . ,un) = {α1u1 + α2u2 + · · ·+ αnun | αi ∈ R}

CS221 Linear Algebra Review 10/11/2011 6 / 37

Vectors

Subspaces

A line through the origin in Rn(span of a vector).

A plane in Rn(span of two vectors).

CS221 Linear Algebra Review 10/11/2011 7 / 37

Vectors

Subspaces

A line through the origin in Rn(span of a vector).

A plane in Rn(span of two vectors).

CS221 Linear Algebra Review 10/11/2011 7 / 37

Vectors

Linear Independence and Basis of Vector Spaces

A vector u is linearly independent of a set of vectors {u1,u2, . . . ,un}if it does not lie in their span.

A set of vectors is linearly independent if every vector is linearlyindependent of the rest.

A basis of a vector space V is a linearly independent set of vectorswhose span is equal to V.If a vector space has a basis with d vectors, its dimension is d.

Any other basis of the same space will consist of exactly d vectors.

The dimension of a vector space can be in�nite (function spaces).

CS221 Linear Algebra Review 10/11/2011 8 / 37

Vectors

Linear Independence and Basis of Vector Spaces

A vector u is linearly independent of a set of vectors {u1,u2, . . . ,un}if it does not lie in their span.

A set of vectors is linearly independent if every vector is linearlyindependent of the rest.

A basis of a vector space V is a linearly independent set of vectorswhose span is equal to V.If a vector space has a basis with d vectors, its dimension is d.

Any other basis of the same space will consist of exactly d vectors.

The dimension of a vector space can be in�nite (function spaces).

CS221 Linear Algebra Review 10/11/2011 8 / 37

Vectors

Linear Independence and Basis of Vector Spaces

A vector u is linearly independent of a set of vectors {u1,u2, . . . ,un}if it does not lie in their span.

A set of vectors is linearly independent if every vector is linearlyindependent of the rest.

A basis of a vector space V is a linearly independent set of vectorswhose span is equal to V.

If a vector space has a basis with d vectors, its dimension is d.

Any other basis of the same space will consist of exactly d vectors.

The dimension of a vector space can be in�nite (function spaces).

CS221 Linear Algebra Review 10/11/2011 8 / 37

Vectors

Linear Independence and Basis of Vector Spaces

A set of vectors is linearly independent if every vector is linearlyindependent of the rest.

A basis of a vector space V is a linearly independent set of vectorswhose span is equal to V.If a vector space has a basis with d vectors, its dimension is d.

Any other basis of the same space will consist of exactly d vectors.

The dimension of a vector space can be in�nite (function spaces).

CS221 Linear Algebra Review 10/11/2011 8 / 37

Vectors

Linear Independence and Basis of Vector Spaces

A set of vectors is linearly independent if every vector is linearlyindependent of the rest.

Any other basis of the same space will consist of exactly d vectors.

The dimension of a vector space can be in�nite (function spaces).

CS221 Linear Algebra Review 10/11/2011 8 / 37

Vectors

Linear Independence and Basis of Vector Spaces

A set of vectors is linearly independent if every vector is linearlyindependent of the rest.

Any other basis of the same space will consist of exactly d vectors.

The dimension of a vector space can be in�nite (function spaces).

CS221 Linear Algebra Review 10/11/2011 8 / 37

Vectors

Norm of a Vector

A norm ||u|| measures the magnitude of a vector.

Mathematically, any function from the vector space to R thatsatis�es :

Homogeneity ||α u|| = |α| ||u||Triangle inequality ||u + v|| ≤ ||u||+ ||v||Point separation ||u|| = 0 if and only if u = 0.

Examples :

Manhattan or `1 norm : ||u||1

=∑

i|ui |.

Euclidean or `2 norm : ||u||2

=√∑

iu2i.

CS221 Linear Algebra Review 10/11/2011 9 / 37

Vectors

Norm of a Vector

A norm ||u|| measures the magnitude of a vector.

Mathematically, any function from the vector space to R thatsatis�es :

Homogeneity ||α u|| = |α| ||u||Triangle inequality ||u + v|| ≤ ||u||+ ||v||Point separation ||u|| = 0 if and only if u = 0.

Examples :

Manhattan or `1 norm : ||u||1

=∑

i|ui |.

Euclidean or `2 norm : ||u||2

=√∑

iu2i.

CS221 Linear Algebra Review 10/11/2011 9 / 37

Vectors

Norm of a Vector

A norm ||u|| measures the magnitude of a vector.

Mathematically, any function from the vector space to R thatsatis�es :

Homogeneity ||α u|| = |α| ||u||

Triangle inequality ||u + v|| ≤ ||u||+ ||v||Point separation ||u|| = 0 if and only if u = 0.

Examples :

Manhattan or `1 norm : ||u||1

=∑

i|ui |.

Euclidean or `2 norm : ||u||2

=√∑

iu2i.

CS221 Linear Algebra Review 10/11/2011 9 / 37

Vectors

Norm of a Vector

A norm ||u|| measures the magnitude of a vector.

Mathematically, any function from the vector space to R thatsatis�es :

Homogeneity ||α u|| = |α| ||u||Triangle inequality ||u + v|| ≤ ||u||+ ||v||

Point separation ||u|| = 0 if and only if u = 0.

Examples :

Manhattan or `1 norm : ||u||1

=∑

i|ui |.

Euclidean or `2 norm : ||u||2

=√∑

iu2i.

CS221 Linear Algebra Review 10/11/2011 9 / 37

Vectors

Norm of a Vector

A norm ||u|| measures the magnitude of a vector.

Mathematically, any function from the vector space to R thatsatis�es :

Homogeneity ||α u|| = |α| ||u||Triangle inequality ||u + v|| ≤ ||u||+ ||v||Point separation ||u|| = 0 if and only if u = 0.

Examples :

Manhattan or `1 norm : ||u||1

=∑

i|ui |.

Euclidean or `2 norm : ||u||2

=√∑

iu2i.

CS221 Linear Algebra Review 10/11/2011 9 / 37

Vectors

Norm of a Vector

A norm ||u|| measures the magnitude of a vector.

Mathematically, any function from the vector space to R thatsatis�es :

Examples :

Manhattan or `1 norm : ||u||1

=∑

i|ui |.

Euclidean or `2 norm : ||u||2

=√∑

iu2i.

CS221 Linear Algebra Review 10/11/2011 9 / 37

Vectors

Inner Product

Inner product between u and v : 〈u, v〉 =∑n

i=1uivi .

It is the projection of one vector onto the other.

Related to the Euclidean norm : 〈u,u〉 = ||u||22.

CS221 Linear Algebra Review 10/11/2011 10 / 37

Vectors

Inner Product

Inner product between u and v : 〈u, v〉 =∑n

i=1uivi .

It is the projection of one vector onto the other.

Related to the Euclidean norm : 〈u,u〉 = ||u||22.

CS221 Linear Algebra Review 10/11/2011 10 / 37

Vectors

Inner Product

Inner product between u and v : 〈u, v〉 =∑n

i=1uivi .

It is the projection of one vector onto the other.

Related to the Euclidean norm : 〈u,u〉 = ||u||22.

CS221 Linear Algebra Review 10/11/2011 10 / 37

Vectors

Inner Product

The inner product is a measure of correlation between two vectors, scaledby the norms of the vectors :

If 〈u, v〉 > 0, u and v are aligned.

If 〈u, v〉 < 0, u and v are opposed.

If 〈u, v〉 = 0, u and v are orthogonal.

CS221 Linear Algebra Review 10/11/2011 11 / 37

Vectors

Inner Product

The inner product is a measure of correlation between two vectors, scaledby the norms of the vectors :

If 〈u, v〉 > 0, u and v are aligned.

If 〈u, v〉 < 0, u and v are opposed.

If 〈u, v〉 = 0, u and v are orthogonal.

CS221 Linear Algebra Review 10/11/2011 11 / 37

Vectors

Inner Product

The inner product is a measure of correlation between two vectors, scaledby the norms of the vectors :

If 〈u, v〉 > 0, u and v are aligned.

If 〈u, v〉 < 0, u and v are opposed.

If 〈u, v〉 = 0, u and v are orthogonal.

CS221 Linear Algebra Review 10/11/2011 11 / 37

Vectors

Inner Product

If 〈u, v〉 > 0, u and v are aligned.

If 〈u, v〉 < 0, u and v are opposed.

If 〈u, v〉 = 0, u and v are orthogonal.

CS221 Linear Algebra Review 10/11/2011 11 / 37

Vectors

Orthonormal Basis

The vectors in an orthonormal basis have unit Euclidean norm andare orthogonal to each other.

To express a vector x in an orthonormal basis, we can just take innerproducts.

Example : x = α1b1 + α2b2.

We compute 〈x,b1〉 :

〈x,b1〉 = 〈α1b1 + α2b2,b1〉= α1 〈b1,b1〉+ α2 〈b2,b1〉 By linearity.

= α1 + 0 By orthonormality.

Likewise, α2 = 〈x,b2〉.

CS221 Linear Algebra Review 10/11/2011 12 / 37

Vectors

Orthonormal Basis

The vectors in an orthonormal basis have unit Euclidean norm andare orthogonal to each other.

To express a vector x in an orthonormal basis, we can just take innerproducts.

Example : x = α1b1 + α2b2.

We compute 〈x,b1〉 :

〈x,b1〉 = 〈α1b1 + α2b2,b1〉= α1 〈b1,b1〉+ α2 〈b2,b1〉 By linearity.

= α1 + 0 By orthonormality.

Likewise, α2 = 〈x,b2〉.

CS221 Linear Algebra Review 10/11/2011 12 / 37

Vectors

Orthonormal Basis

The vectors in an orthonormal basis have unit Euclidean norm andare orthogonal to each other.

To express a vector x in an orthonormal basis, we can just take innerproducts.

Example : x = α1b1 + α2b2.

We compute 〈x,b1〉 :

〈x,b1〉 = 〈α1b1 + α2b2,b1〉= α1 〈b1,b1〉+ α2 〈b2,b1〉 By linearity.

= α1 + 0 By orthonormality.

Likewise, α2 = 〈x,b2〉.

CS221 Linear Algebra Review 10/11/2011 12 / 37

Vectors

Orthonormal Basis

The vectors in an orthonormal basis have unit Euclidean norm andare orthogonal to each other.

To express a vector x in an orthonormal basis, we can just take innerproducts.

Example : x = α1b1 + α2b2.

We compute 〈x,b1〉 :

〈x,b1〉 = 〈α1b1 + α2b2,b1〉= α1 〈b1,b1〉+ α2 〈b2,b1〉 By linearity.

= α1 + 0 By orthonormality.

Likewise, α2 = 〈x,b2〉.

CS221 Linear Algebra Review 10/11/2011 12 / 37

Vectors

Orthonormal Basis

The vectors in an orthonormal basis have unit Euclidean norm andare orthogonal to each other.

To express a vector x in an orthonormal basis, we can just take innerproducts.

Example : x = α1b1 + α2b2.

We compute 〈x,b1〉 :

〈x,b1〉 = 〈α1b1 + α2b2,b1〉= α1 〈b1,b1〉+ α2 〈b2,b1〉 By linearity.

= α1 + 0 By orthonormality.

Likewise, α2 = 〈x,b2〉.

CS221 Linear Algebra Review 10/11/2011 12 / 37

Matrices

1 Vectors

2 Matrices

3 Matrix Decompositions

4 Application : Image Compression

CS221 Linear Algebra Review 10/11/2011 13 / 37

Matrices

Linear Operator

A linear operator L : U → V is a map from a vector space U toanother vector space V that satis�es :

L (u1 + u2) = L (u1) + L (u2).L (α u) = α L (u) for any scalar α.

If the dimension n of U and m of V are �nite, L can be represented byan m × n matrix A.

A =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

CS221 Linear Algebra Review 10/11/2011 14 / 37

Matrices

Linear Operator

A linear operator L : U → V is a map from a vector space U toanother vector space V that satis�es :

L (u1 + u2) = L (u1) + L (u2).

L (α u) = α L (u) for any scalar α.

If the dimension n of U and m of V are �nite, L can be represented byan m × n matrix A.

A =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

CS221 Linear Algebra Review 10/11/2011 14 / 37

Matrices

Linear Operator

A linear operator L : U → V is a map from a vector space U toanother vector space V that satis�es :

L (u1 + u2) = L (u1) + L (u2).L (α u) = α L (u) for any scalar α.

If the dimension n of U and m of V are �nite, L can be represented byan m × n matrix A.

A =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

CS221 Linear Algebra Review 10/11/2011 14 / 37

Matrices

Linear Operator

A linear operator L : U → V is a map from a vector space U toanother vector space V that satis�es :

L (u1 + u2) = L (u1) + L (u2).L (α u) = α L (u) for any scalar α.

If the dimension n of U and m of V are �nite, L can be represented byan m × n matrix A.

A =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

CS221 Linear Algebra Review 10/11/2011 14 / 37

Matrices

Matrix Vector Multiplication

Au =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

u1u2. . .un

=

v1v2. . .vm

CS221 Linear Algebra Review 10/11/2011 15 / 37

Matrices

Matrix Vector Multiplication

Au =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

u1u2. . .un

=

v1v2. . .vm

CS221 Linear Algebra Review 10/11/2011 15 / 37

Matrices

Matrix Vector Multiplication

Au =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

u1u2. . .un

=

v1v2. . .vm

CS221 Linear Algebra Review 10/11/2011 15 / 37

Matrices

Matrix Vector Multiplication

Au =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

u1u2. . .un

=

v1v2. . .vm

CS221 Linear Algebra Review 10/11/2011 15 / 37

Matrices

Matrix Product

Applying two linear operators A and B de�nes another linear operator,which can be represented by another matrix AB.

AB =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

B11 B12 . . . B1p

B21 B22 . . . B2p

. . .Bn1 Bn2 . . . Bnp

=

AB11 AB12 . . . AB1p

AB21 AB22 . . . AB2p

. . .ABm1 ABm2 . . . ABmp

Note that if A is m × n and B is n × p, then AB is m × p.

CS221 Linear Algebra Review 10/11/2011 16 / 37

Matrices

Matrix Product

Applying two linear operators A and B de�nes another linear operator,which can be represented by another matrix AB.

AB =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

B11 B12 . . . B1p

B21 B22 . . . B2p

. . .Bn1 Bn2 . . . Bnp

=

AB11 AB12 . . . AB1p

AB21 AB22 . . . AB2p

. . .ABm1 ABm2 . . . ABmp

Note that if A is m × n and B is n × p, then AB is m × p.

CS221 Linear Algebra Review 10/11/2011 16 / 37

Matrices

Matrix Product

Applying two linear operators A and B de�nes another linear operator,which can be represented by another matrix AB.

AB =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

B11 B12 . . . B1p

B21 B22 . . . B2p

. . .Bn1 Bn2 . . . Bnp

=

AB11 AB12 . . . AB1p

AB21 AB22 . . . AB2p

. . .ABm1 ABm2 . . . ABmp

Note that if A is m × n and B is n × p, then AB is m × p.

CS221 Linear Algebra Review 10/11/2011 16 / 37

Matrices

Matrix Product

AB =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

B11 B12 . . . B1p

B21 B22 . . . B2p

. . .Bn1 Bn2 . . . Bnp

=

AB11 AB12 . . . AB1p

AB21 AB22 . . . AB2p

. . .ABm1 ABm2 . . . ABmp

Note that if A is m × n and B is n × p, then AB is m × p.

CS221 Linear Algebra Review 10/11/2011 16 / 37

Matrices

Matrix Product

AB =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

B11 B12 . . . B1p

B21 B22 . . . B2p

. . .Bn1 Bn2 . . . Bnp

=

AB11 AB12 . . . AB1p

AB21 AB22 . . . AB2p

. . .ABm1 ABm2 . . . ABmp

Note that if A is m × n and B is n × p, then AB is m × p.

CS221 Linear Algebra Review 10/11/2011 16 / 37

Matrices

Matrix Product

AB =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

B11 B12 . . . B1p

B21 B22 . . . B2p

. . .Bn1 Bn2 . . . Bnp

=

AB11 AB12 . . . AB1p

AB21 AB22 . . . AB2p

. . .ABm1 ABm2 . . . ABmp

Note that if A is m × n and B is n × p, then AB is m × p.

CS221 Linear Algebra Review 10/11/2011 16 / 37

Matrices

Matrix Product

AB =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

B11 B12 . . . B1p

B21 B22 . . . B2p

. . .Bn1 Bn2 . . . Bnp

=

AB11 AB12 . . . AB1p

AB21 AB22 . . . AB2p

. . .ABm1 ABm2 . . . ABmp

Note that if A is m × n and B is n × p, then AB is m × p.

CS221 Linear Algebra Review 10/11/2011 16 / 37

Matrices

Matrix Product

AB =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

B11 B12 . . . B1p

B21 B22 . . . B2p

. . .Bn1 Bn2 . . . Bnp

=

AB11 AB12 . . . AB1p

AB21 AB22 . . . AB2p

. . .ABm1 ABm2 . . . ABmp

Note that if A is m × n and B is n × p, then AB is m × p.

CS221 Linear Algebra Review 10/11/2011 16 / 37

Matrices

Matrix Product

AB =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

B11 B12 . . . B1p

B21 B22 . . . B2p

. . .Bn1 Bn2 . . . Bnp

=

AB11 AB12 . . . AB1p

AB21 AB22 . . . AB2p

. . .ABm1 ABm2 . . . ABmp

Note that if A is m × n and B is n × p, then AB is m × p.

CS221 Linear Algebra Review 10/11/2011 16 / 37

Matrices

Matrix Product

AB =

A11 A12 . . . A1n

A21 A22 . . . A2n

. . .Am1 Am2 . . . Amn

B11 B12 . . . B1p

B21 B22 . . . B2p

. . .Bn1 Bn2 . . . Bnp

=

AB11 AB12 . . . AB1p

AB21 AB22 . . . AB2p

. . .ABm1 ABm2 . . . ABmp

Note that if A is m × n and B is n × p, then AB is m × p.

CS221 Linear Algebra Review 10/11/2011 16 / 37

Matrices

Transpose of a Matrix

The transpose of a matrix is obtained by �ipping the rows andcolumns.

AT =

A11 A21 . . . An1

A12 A22 . . . An2

. . .A1m A2m . . . Anm

Some simple properties :

(AT)T

= A

(A + B)T = AT + BT

(AB)T = BTAT

The inner product for vectors can be represented as 〈u, v〉 = uTv.

CS221 Linear Algebra Review 10/11/2011 17 / 37

Matrices

Transpose of a Matrix

The transpose of a matrix is obtained by �ipping the rows andcolumns.

AT =

A11 A21 . . . An1

A12 A22 . . . An2

. . .A1m A2m . . . Anm

Some simple properties :

(AT)T

= A

(A + B)T = AT + BT

(AB)T = BTAT

The inner product for vectors can be represented as 〈u, v〉 = uTv.

CS221 Linear Algebra Review 10/11/2011 17 / 37

Matrices

Transpose of a Matrix

The transpose of a matrix is obtained by �ipping the rows andcolumns.

AT =

A11 A21 . . . An1

A12 A22 . . . An2

. . .A1m A2m . . . Anm

Some simple properties :(

AT)T

= A

(A + B)T = AT + BT

(AB)T = BTAT

The inner product for vectors can be represented as 〈u, v〉 = uTv.

CS221 Linear Algebra Review 10/11/2011 17 / 37

Matrices

Transpose of a Matrix

The transpose of a matrix is obtained by �ipping the rows andcolumns.

AT =

A11 A21 . . . An1

A12 A22 . . . An2

. . .A1m A2m . . . Anm

Some simple properties :(

AT)T

= A

(A + B)T = AT + BT

(AB)T = BTAT

The inner product for vectors can be represented as 〈u, v〉 = uTv.

CS221 Linear Algebra Review 10/11/2011 17 / 37

Matrices

Transpose of a Matrix

The transpose of a matrix is obtained by �ipping the rows andcolumns.

AT =

A11 A21 . . . An1

A12 A22 . . . An2

. . .A1m A2m . . . Anm

Some simple properties :(

AT)T

= A

(A + B)T = AT + BT

(AB)T = BTAT

The inner product for vectors can be represented as 〈u, v〉 = uTv.

CS221 Linear Algebra Review 10/11/2011 17 / 37

Matrices

Transpose of a Matrix

The transpose of a matrix is obtained by �ipping the rows andcolumns.

AT =

A11 A21 . . . An1

A12 A22 . . . An2

. . .A1m A2m . . . Anm

Some simple properties :(

AT)T

= A

(A + B)T = AT + BT

(AB)T = BTAT

The inner product for vectors can be represented as 〈u, v〉 = uTv.

CS221 Linear Algebra Review 10/11/2011 17 / 37

Matrices

Identity Operator

I =

1 0 . . . 00 1 . . . 0

. . .0 0 . . . 1

For any matrix A, AI = A.

I is the identity operator for the matrix product.

CS221 Linear Algebra Review 10/11/2011 18 / 37

Matrices

Identity Operator

I =

1 0 . . . 00 1 . . . 0

. . .0 0 . . . 1

For any matrix A, AI = A.

I is the identity operator for the matrix product.

CS221 Linear Algebra Review 10/11/2011 18 / 37

Matrices

Identity Operator

I =

1 0 . . . 00 1 . . . 0

. . .0 0 . . . 1

For any matrix A, AI = A.

I is the identity operator for the matrix product.

CS221 Linear Algebra Review 10/11/2011 18 / 37

Matrices

Column Space, Row Space and Rank

Let A be an m × n matrix.

Column space :

Span of the columns of A.Linear subspace of Rm.

Row space :

Span of the rows of A.Linear subspace of Rn.

Important fact :The column and row spaces of any matrix have the same dimension.

This dimension is the rank of the matrix.

CS221 Linear Algebra Review 10/11/2011 19 / 37

Matrices

Column Space, Row Space and Rank

Let A be an m × n matrix.

Column space :

Span of the columns of A.Linear subspace of Rm.

Row space :

Span of the rows of A.Linear subspace of Rn.

Important fact :The column and row spaces of any matrix have the same dimension.

This dimension is the rank of the matrix.

CS221 Linear Algebra Review 10/11/2011 19 / 37

Matrices

Column Space, Row Space and Rank

Let A be an m × n matrix.

Column space :

Span of the columns of A.

Linear subspace of Rm.

Row space :

Span of the rows of A.Linear subspace of Rn.

Important fact :The column and row spaces of any matrix have the same dimension.

This dimension is the rank of the matrix.

CS221 Linear Algebra Review 10/11/2011 19 / 37

Matrices

Column Space, Row Space and Rank

Let A be an m × n matrix.

Column space :

Span of the columns of A.Linear subspace of Rm.

Row space :

Span of the rows of A.Linear subspace of Rn.

Important fact :The column and row spaces of any matrix have the same dimension.

This dimension is the rank of the matrix.

CS221 Linear Algebra Review 10/11/2011 19 / 37

Matrices

Column Space, Row Space and Rank

Let A be an m × n matrix.

Column space :

Span of the columns of A.Linear subspace of Rm.

Row space :

Span of the rows of A.Linear subspace of Rn.

Important fact :The column and row spaces of any matrix have the same dimension.

This dimension is the rank of the matrix.

CS221 Linear Algebra Review 10/11/2011 19 / 37

Matrices

Column Space, Row Space and Rank

Let A be an m × n matrix.

Column space :

Span of the columns of A.Linear subspace of Rm.

Row space :

Span of the rows of A.

Linear subspace of Rn.

Important fact :The column and row spaces of any matrix have the same dimension.

This dimension is the rank of the matrix.

CS221 Linear Algebra Review 10/11/2011 19 / 37

Matrices

Column Space, Row Space and Rank

Let A be an m × n matrix.

Column space :

Span of the columns of A.Linear subspace of Rm.

Row space :

Span of the rows of A.Linear subspace of Rn.

Important fact :The column and row spaces of any matrix have the same dimension.

This dimension is the rank of the matrix.

CS221 Linear Algebra Review 10/11/2011 19 / 37

Matrices

Column Space, Row Space and Rank

Let A be an m × n matrix.

Column space :

Span of the columns of A.Linear subspace of Rm.

Row space :

Span of the rows of A.Linear subspace of Rn.

Important fact :The column and row spaces of any matrix have the same dimension.

This dimension is the rank of the matrix.

CS221 Linear Algebra Review 10/11/2011 19 / 37

Matrices

Column Space, Row Space and Rank

Let A be an m × n matrix.

Column space :

Span of the columns of A.Linear subspace of Rm.

Row space :

Span of the rows of A.Linear subspace of Rn.

Important fact :The column and row spaces of any matrix have the same dimension.

This dimension is the rank of the matrix.

CS221 Linear Algebra Review 10/11/2011 19 / 37

Matrices

Range and Null space

Let A be an m × n matrix.

Range :

Set of vectors equal to Au for some u ∈ Rn.Linear subspace of Rm.

Null space :

u belongs to the null space if Au = 0.Subspace of Rn.Every vector in the null space is orthogonal to the rows of A.The null space and row space of a matrix are orthogonal.

CS221 Linear Algebra Review 10/11/2011 20 / 37

Matrices

Range and Null space

Let A be an m × n matrix.

Range :

Set of vectors equal to Au for some u ∈ Rn.Linear subspace of Rm.

Null space :

u belongs to the null space if Au = 0.Subspace of Rn.Every vector in the null space is orthogonal to the rows of A.The null space and row space of a matrix are orthogonal.

CS221 Linear Algebra Review 10/11/2011 20 / 37

Matrices

Range and Null space

Let A be an m × n matrix.

Range :

Set of vectors equal to Au for some u ∈ Rn.

Linear subspace of Rm.

Null space :

u belongs to the null space if Au = 0.Subspace of Rn.Every vector in the null space is orthogonal to the rows of A.The null space and row space of a matrix are orthogonal.

CS221 Linear Algebra Review 10/11/2011 20 / 37

Matrices

Range and Null space

Let A be an m × n matrix.

Range :

Set of vectors equal to Au for some u ∈ Rn.Linear subspace of Rm.

Null space :

CS221 Linear Algebra Review 10/11/2011 20 / 37

Matrices

Range and Null space

Let A be an m × n matrix.

Range :

Set of vectors equal to Au for some u ∈ Rn.Linear subspace of Rm.

Null space :

CS221 Linear Algebra Review 10/11/2011 20 / 37

Matrices

Range and Null space

Let A be an m × n matrix.

Range :

Set of vectors equal to Au for some u ∈ Rn.Linear subspace of Rm.

Null space :

u belongs to the null space if Au = 0.

Subspace of Rn.Every vector in the null space is orthogonal to the rows of A.The null space and row space of a matrix are orthogonal.

CS221 Linear Algebra Review 10/11/2011 20 / 37

Matrices

Range and Null space

Let A be an m × n matrix.

Range :

Set of vectors equal to Au for some u ∈ Rn.Linear subspace of Rm.

Null space :

u belongs to the null space if Au = 0.Subspace of Rn.

Every vector in the null space is orthogonal to the rows of A.The null space and row space of a matrix are orthogonal.

CS221 Linear Algebra Review 10/11/2011 20 / 37

Matrices

Range and Null space

Let A be an m × n matrix.

Range :

Set of vectors equal to Au for some u ∈ Rn.Linear subspace of Rm.

Null space :

CS221 Linear Algebra Review 10/11/2011 20 / 37

Matrices

Range and Column Space

Another interpretation of the matrix vector product (Ai is column i of A) :

Au =

(A1 A2 . . . An

)u1u2. . .un

= u1A1 + u2A2 + · · ·+ unAn

The result is a linear combination of the columns of A.

For any matrix, the range is equal to the column space.

CS221 Linear Algebra Review 10/11/2011 21 / 37

Matrices

Range and Column Space

Another interpretation of the matrix vector product (Ai is column i of A) :

Au =

(A1 A2 . . . An

)u1u2. . .un

= u1A1 + u2A2 + · · ·+ unAn

The result is a linear combination of the columns of A.

For any matrix, the range is equal to the column space.

CS221 Linear Algebra Review 10/11/2011 21 / 37

Matrices

Range and Column Space

Another interpretation of the matrix vector product (Ai is column i of A) :

Au =

(A1 A2 . . . An

)u1u2. . .un

= u1A1 + u2A2 + · · ·+ unAn

The result is a linear combination of the columns of A.

For any matrix, the range is equal to the column space.

CS221 Linear Algebra Review 10/11/2011 21 / 37

Matrices

Matrix Inverse

For an n × n matrix A : rank + dim(null space) = n.

If dim(null space)=0 then A is full rank.

In this case, the action of the matrix is invertible.

The inversion is also linear and consequently can be represented byanother matrix A−1.

A−1 is the only matrix such that A−1A = AA−1 = I.

CS221 Linear Algebra Review 10/11/2011 22 / 37

Matrices

Matrix Inverse

For an n × n matrix A : rank + dim(null space) = n.

If dim(null space)=0 then A is full rank.

In this case, the action of the matrix is invertible.

The inversion is also linear and consequently can be represented byanother matrix A−1.

A−1 is the only matrix such that A−1A = AA−1 = I.

CS221 Linear Algebra Review 10/11/2011 22 / 37

Matrices

Matrix Inverse

For an n × n matrix A : rank + dim(null space) = n.

If dim(null space)=0 then A is full rank.

In this case, the action of the matrix is invertible.

The inversion is also linear and consequently can be represented byanother matrix A−1.

A−1 is the only matrix such that A−1A = AA−1 = I.

CS221 Linear Algebra Review 10/11/2011 22 / 37

Matrices

Matrix Inverse

For an n × n matrix A : rank + dim(null space) = n.

If dim(null space)=0 then A is full rank.

In this case, the action of the matrix is invertible.

The inversion is also linear and consequently can be represented byanother matrix A−1.

A−1 is the only matrix such that A−1A = AA−1 = I.

CS221 Linear Algebra Review 10/11/2011 22 / 37

Matrices

Matrix Inverse

For an n × n matrix A : rank + dim(null space) = n.

If dim(null space)=0 then A is full rank.

In this case, the action of the matrix is invertible.

The inversion is also linear and consequently can be represented byanother matrix A−1.

A−1 is the only matrix such that A−1A = AA−1 = I.

CS221 Linear Algebra Review 10/11/2011 22 / 37

Matrices

Orthogonal Matrices

An orthogonal matrix U satis�es UTU = I.

Equivalently, U has orthonormal columns.

Applying an orthogonal matrix to two vectors does not change theirinner product :

〈Uu,Uv〉 = (Uu)T Uv

= uTUTUv

= uTv

= 〈u, v〉

Example : Matrices that represent rotations are orthogonal.

CS221 Linear Algebra Review 10/11/2011 23 / 37

Matrices

Orthogonal Matrices

An orthogonal matrix U satis�es UTU = I.

Equivalently, U has orthonormal columns.

Applying an orthogonal matrix to two vectors does not change theirinner product :

〈Uu,Uv〉 = (Uu)T Uv

= uTUTUv

= uTv

= 〈u, v〉

Example : Matrices that represent rotations are orthogonal.

CS221 Linear Algebra Review 10/11/2011 23 / 37

Matrices

Orthogonal Matrices

An orthogonal matrix U satis�es UTU = I.

Equivalently, U has orthonormal columns.

Applying an orthogonal matrix to two vectors does not change theirinner product :

〈Uu,Uv〉 = (Uu)T Uv

= uTUTUv

= uTv

= 〈u, v〉

Example : Matrices that represent rotations are orthogonal.

CS221 Linear Algebra Review 10/11/2011 23 / 37

Matrices

Orthogonal Matrices

An orthogonal matrix U satis�es UTU = I.

Equivalently, U has orthonormal columns.

Applying an orthogonal matrix to two vectors does not change theirinner product :

〈Uu,Uv〉 = (Uu)T Uv

= uTUTUv

= uTv

= 〈u, v〉

Example : Matrices that represent rotations are orthogonal.

CS221 Linear Algebra Review 10/11/2011 23 / 37

Matrices

Trace and Determinant

The trace is the sum of the diagonal elements of a square matrix.

The determinant of a square matrix A is denoted by |A|.For a 2× 2 matrix A =

(a bc d

):

|A| = ad − bc

The absolute value of |A| is the area of the parallelogram given by therows of A.

CS221 Linear Algebra Review 10/11/2011 24 / 37

Matrices

Trace and Determinant

The trace is the sum of the diagonal elements of a square matrix.

The determinant of a square matrix A is denoted by |A|.

For a 2× 2 matrix A =(a bc d

):

|A| = ad − bc

The absolute value of |A| is the area of the parallelogram given by therows of A.

CS221 Linear Algebra Review 10/11/2011 24 / 37

Matrices

Trace and Determinant

The trace is the sum of the diagonal elements of a square matrix.

The determinant of a square matrix A is denoted by |A|.For a 2× 2 matrix A =

(a bc d

):

|A| = ad − bc

The absolute value of |A| is the area of the parallelogram given by therows of A.

CS221 Linear Algebra Review 10/11/2011 24 / 37

Matrices

Trace and Determinant

The trace is the sum of the diagonal elements of a square matrix.

The determinant of a square matrix A is denoted by |A|.For a 2× 2 matrix A =

(a bc d

):

|A| = ad − bc

The absolute value of |A| is the area of the parallelogram given by therows of A.

CS221 Linear Algebra Review 10/11/2011 24 / 37

Matrices

Trace and Determinant

The trace is the sum of the diagonal elements of a square matrix.

The determinant of a square matrix A is denoted by |A|.For a 2× 2 matrix A =

(a bc d

):

|A| = ad − bc

The absolute value of |A| is the area of the parallelogram given by therows of A.

CS221 Linear Algebra Review 10/11/2011 24 / 37

Matrices

Properties of the Determinant

The de�nition can be generalized to larger matrices.

|A| =∣∣AT

∣∣.|AB| = |A| |B||A| = 0 if and only if A is not invertible.

If A is invertible, then∣∣A−1∣∣ = 1

|A| .

CS221 Linear Algebra Review 10/11/2011 25 / 37

Matrices

Properties of the Determinant

The de�nition can be generalized to larger matrices.

|A| =∣∣AT

∣∣.

|AB| = |A| |B||A| = 0 if and only if A is not invertible.

If A is invertible, then∣∣A−1∣∣ = 1

|A| .

CS221 Linear Algebra Review 10/11/2011 25 / 37

Matrices

Properties of the Determinant

The de�nition can be generalized to larger matrices.

|A| =∣∣AT

∣∣.|AB| = |A| |B|

|A| = 0 if and only if A is not invertible.

If A is invertible, then∣∣A−1∣∣ = 1

|A| .

CS221 Linear Algebra Review 10/11/2011 25 / 37

Matrices

Properties of the Determinant

The de�nition can be generalized to larger matrices.

|A| =∣∣AT

∣∣.|AB| = |A| |B||A| = 0 if and only if A is not invertible.

If A is invertible, then∣∣A−1∣∣ = 1

|A| .

CS221 Linear Algebra Review 10/11/2011 25 / 37

Matrices

Properties of the Determinant

The de�nition can be generalized to larger matrices.

|A| =∣∣AT

∣∣.|AB| = |A| |B||A| = 0 if and only if A is not invertible.

If A is invertible, then∣∣A−1∣∣ = 1

|A| .

CS221 Linear Algebra Review 10/11/2011 25 / 37

Matrix Decompositions

1 Vectors

2 Matrices

3 Matrix Decompositions

4 Application : Image Compression

CS221 Linear Algebra Review 10/11/2011 26 / 37

Matrix Decompositions

Eigenvalues and Eigenvectors

An eigenvalue λ of a square matrix A satis�es :

Au = λu

for some vector u, which we call an eigenvector.

Geometrically, the operator A expands (λ > 1) or contracts (λ < 1)eigenvectors but does not rotate them.

If u is an eigenvector of A, it is in the null space of A− λI, which isconsequently not invertible.

The eigenvalues of A are the roots of the equation |A− λI| = 0.

Not the way eigenvalues are calculated in practice.

Eigenvalues and eigenvectors can be complex valued, even if all theentries of A are real.

CS221 Linear Algebra Review 10/11/2011 27 / 37

Matrix Decompositions

Eigenvalues and Eigenvectors

An eigenvalue λ of a square matrix A satis�es :

Au = λu

for some vector u, which we call an eigenvector.

Geometrically, the operator A expands (λ > 1) or contracts (λ < 1)eigenvectors but does not rotate them.

If u is an eigenvector of A, it is in the null space of A− λI, which isconsequently not invertible.

The eigenvalues of A are the roots of the equation |A− λI| = 0.

Not the way eigenvalues are calculated in practice.

Eigenvalues and eigenvectors can be complex valued, even if all theentries of A are real.

CS221 Linear Algebra Review 10/11/2011 27 / 37

Matrix Decompositions

Eigenvalues and Eigenvectors

An eigenvalue λ of a square matrix A satis�es :

Au = λu

for some vector u, which we call an eigenvector.

Geometrically, the operator A expands (λ > 1) or contracts (λ < 1)eigenvectors but does not rotate them.

If u is an eigenvector of A, it is in the null space of A− λI, which isconsequently not invertible.

The eigenvalues of A are the roots of the equation |A− λI| = 0.

Not the way eigenvalues are calculated in practice.

Eigenvalues and eigenvectors can be complex valued, even if all theentries of A are real.

CS221 Linear Algebra Review 10/11/2011 27 / 37

Matrix Decompositions

Eigenvalues and Eigenvectors

An eigenvalue λ of a square matrix A satis�es :

Au = λu

for some vector u, which we call an eigenvector.

If u is an eigenvector of A, it is in the null space of A− λI, which isconsequently not invertible.

The eigenvalues of A are the roots of the equation |A− λI| = 0.

Not the way eigenvalues are calculated in practice.

Eigenvalues and eigenvectors can be complex valued, even if all theentries of A are real.

CS221 Linear Algebra Review 10/11/2011 27 / 37

Matrix Decompositions

Eigenvalues and Eigenvectors

An eigenvalue λ of a square matrix A satis�es :

Au = λu

for some vector u, which we call an eigenvector.

If u is an eigenvector of A, it is in the null space of A− λI, which isconsequently not invertible.

The eigenvalues of A are the roots of the equation |A− λI| = 0.

Not the way eigenvalues are calculated in practice.

Eigenvalues and eigenvectors can be complex valued, even if all theentries of A are real.

CS221 Linear Algebra Review 10/11/2011 27 / 37

Matrix Decompositions

Eigenvalues and Eigenvectors

An eigenvalue λ of a square matrix A satis�es :

Au = λu

for some vector u, which we call an eigenvector.

If u is an eigenvector of A, it is in the null space of A− λI, which isconsequently not invertible.

The eigenvalues of A are the roots of the equation |A− λI| = 0.

Not the way eigenvalues are calculated in practice.

Eigenvalues and eigenvectors can be complex valued, even if all theentries of A are real.

CS221 Linear Algebra Review 10/11/2011 27 / 37

Matrix Decompositions

Eigendecomposition of a Matrix

Let A be an n × n square matrix with n linearly independenteigenvectors p1 . . .pn and eigenvalues λ1 . . . λn.

We de�ne the matrices :

P =

(p1 . . . pn

)

Λ =

λ1 0 . . . 00 λ2 . . . 0

. . .0 0 . . . λn

A satis�es AP = PΛ.

P is full rank. Inverting it yields the eigendecomposition of A :

A = PΛP−1

CS221 Linear Algebra Review 10/11/2011 28 / 37

Matrix Decompositions

Eigendecomposition of a Matrix

Let A be an n × n square matrix with n linearly independenteigenvectors p1 . . .pn and eigenvalues λ1 . . . λn.

We de�ne the matrices :

P =

(p1 . . . pn

)

Λ =

λ1 0 . . . 00 λ2 . . . 0

. . .0 0 . . . λn

A satis�es AP = PΛ.

P is full rank. Inverting it yields the eigendecomposition of A :

A = PΛP−1

CS221 Linear Algebra Review 10/11/2011 28 / 37

Matrix Decompositions

Eigendecomposition of a Matrix

Let A be an n × n square matrix with n linearly independenteigenvectors p1 . . .pn and eigenvalues λ1 . . . λn.

We de�ne the matrices :

P =

(p1 . . . pn

)

Λ =

λ1 0 . . . 00 λ2 . . . 0

. . .0 0 . . . λn

A satis�es AP = PΛ.

P is full rank. Inverting it yields the eigendecomposition of A :

A = PΛP−1

CS221 Linear Algebra Review 10/11/2011 28 / 37

Matrix Decompositions

Eigendecomposition of a Matrix

We de�ne the matrices :

P =

(p1 . . . pn

)

Λ =

λ1 0 . . . 00 λ2 . . . 0

. . .0 0 . . . λn

A satis�es AP = PΛ.

P is full rank. Inverting it yields the eigendecomposition of A :

A = PΛP−1

CS221 Linear Algebra Review 10/11/2011 28 / 37

Matrix Decompositions

Properties of the Eigendecomposition

Not all matrices are diagonalizable (have an eigendecomposition).Example : ( 1 1

0 1).

Trace(A) = Trace(Λ) =∑n

i=1λi .

|A| = |Λ| =∏n

i=1λi .

The rank of A is equal to the number of nonzero eigenvalues.

If λ is a nonzero eigenvalue of A, 1

λ is an eigenvalue of A−1 with thesame eigenvector.

The eigendecomposition makes it possible to compute matrix powerse�ciently :Am =

(PΛP−1

)m= PΛP−1 PΛP−1 . . .PΛP−1 = PΛmP−1.

CS221 Linear Algebra Review 10/11/2011 29 / 37

Matrix Decompositions

Properties of the Eigendecomposition

Not all matrices are diagonalizable (have an eigendecomposition).Example : ( 1 1

0 1).

Trace(A) = Trace(Λ) =∑n

i=1λi .

|A| = |Λ| =∏n

i=1λi .

The rank of A is equal to the number of nonzero eigenvalues.

If λ is a nonzero eigenvalue of A, 1

λ is an eigenvalue of A−1 with thesame eigenvector.

The eigendecomposition makes it possible to compute matrix powerse�ciently :Am =

(PΛP−1

)m= PΛP−1 PΛP−1 . . .PΛP−1 = PΛmP−1.

CS221 Linear Algebra Review 10/11/2011 29 / 37

Matrix Decompositions

Properties of the Eigendecomposition

Not all matrices are diagonalizable (have an eigendecomposition).Example : ( 1 1

0 1).

Trace(A) = Trace(Λ) =∑n

i=1λi .

|A| = |Λ| =∏n

i=1λi .

The rank of A is equal to the number of nonzero eigenvalues.

If λ is a nonzero eigenvalue of A, 1

λ is an eigenvalue of A−1 with thesame eigenvector.

The eigendecomposition makes it possible to compute matrix powerse�ciently :Am =

(PΛP−1

)m= PΛP−1 PΛP−1 . . .PΛP−1 = PΛmP−1.

CS221 Linear Algebra Review 10/11/2011 29 / 37

Matrix Decompositions

Properties of the Eigendecomposition

Not all matrices are diagonalizable (have an eigendecomposition).Example : ( 1 1

0 1).

Trace(A) = Trace(Λ) =∑n

i=1λi .

|A| = |Λ| =∏n

i=1λi .

The rank of A is equal to the number of nonzero eigenvalues.

If λ is a nonzero eigenvalue of A, 1

λ is an eigenvalue of A−1 with thesame eigenvector.

The eigendecomposition makes it possible to compute matrix powerse�ciently :Am =

(PΛP−1

)m= PΛP−1 PΛP−1 . . .PΛP−1 = PΛmP−1.

CS221 Linear Algebra Review 10/11/2011 29 / 37

Matrix Decompositions

Properties of the Eigendecomposition

Not all matrices are diagonalizable (have an eigendecomposition).Example : ( 1 1

0 1).

Trace(A) = Trace(Λ) =∑n

i=1λi .

|A| = |Λ| =∏n

i=1λi .

The rank of A is equal to the number of nonzero eigenvalues.

If λ is a nonzero eigenvalue of A, 1

λ is an eigenvalue of A−1 with thesame eigenvector.

The eigendecomposition makes it possible to compute matrix powerse�ciently :Am =

(PΛP−1

)m= PΛP−1 PΛP−1 . . .PΛP−1 = PΛmP−1.

CS221 Linear Algebra Review 10/11/2011 29 / 37

Matrix Decompositions

Properties of the Eigendecomposition

Not all matrices are diagonalizable (have an eigendecomposition).Example : ( 1 1

0 1).

Trace(A) = Trace(Λ) =∑n

i=1λi .

|A| = |Λ| =∏n

i=1λi .

The rank of A is equal to the number of nonzero eigenvalues.

If λ is a nonzero eigenvalue of A, 1

λ is an eigenvalue of A−1 with thesame eigenvector.

The eigendecomposition makes it possible to compute matrix powerse�ciently :Am =

(PΛP−1

)m= PΛP−1 PΛP−1 . . .PΛP−1 = PΛmP−1.

CS221 Linear Algebra Review 10/11/2011 29 / 37

Matrix Decompositions

Matlab Example

CS221 Linear Algebra Review 10/11/2011 30 / 37

Matrix Decompositions

Eigendecomposition of a Symmetric Matrix

If A = AT then A is symmetric.

The eigenvalues of symmetric matrices are real.

The eigenvectors of symmetric matrices are orthonormal.

Consequently, the eigendecomposition becomes :A = UΛUT for Λ real and U orthogonal.

The eigenvectors of A are an orthonormal basis for the column space(and the row space, since they are equal).

CS221 Linear Algebra Review 10/11/2011 31 / 37

Matrix Decompositions

Eigendecomposition of a Symmetric Matrix

If A = AT then A is symmetric.

The eigenvalues of symmetric matrices are real.

The eigenvectors of symmetric matrices are orthonormal.

Consequently, the eigendecomposition becomes :A = UΛUT for Λ real and U orthogonal.

The eigenvectors of A are an orthonormal basis for the column space(and the row space, since they are equal).

CS221 Linear Algebra Review 10/11/2011 31 / 37

Matrix Decompositions

Eigendecomposition of a Symmetric Matrix

If A = AT then A is symmetric.

The eigenvalues of symmetric matrices are real.

The eigenvectors of symmetric matrices are orthonormal.

Consequently, the eigendecomposition becomes :A = UΛUT for Λ real and U orthogonal.

The eigenvectors of A are an orthonormal basis for the column space(and the row space, since they are equal).

CS221 Linear Algebra Review 10/11/2011 31 / 37

Matrix Decompositions

Eigendecomposition of a Symmetric Matrix

If A = AT then A is symmetric.

The eigenvalues of symmetric matrices are real.

The eigenvectors of symmetric matrices are orthonormal.

Consequently, the eigendecomposition becomes :A = UΛUT for Λ real and U orthogonal.

CS221 Linear Algebra Review 10/11/2011 31 / 37

Matrix Decompositions

Eigendecomposition of a Symmetric Matrix

If A = AT then A is symmetric.

The eigenvalues of symmetric matrices are real.

The eigenvectors of symmetric matrices are orthonormal.

Consequently, the eigendecomposition becomes :A = UΛUT for Λ real and U orthogonal.

CS221 Linear Algebra Review 10/11/2011 31 / 37

Matrix Decompositions

Eigendecomposition of a Symmetric Matrix

The action of a symmetric matrix on a vector Au = UΛUTu can bedecomposed into :

1 Projection of u onto the column space of A (multiplication by UT ).2 Scaling of each coe�cient 〈Ui ,u〉 by the corresponding eigenvalue

(multiplication by Λ).3 Linear combination of the eigenvectors scaled by the resulting

coe�cient (multiplication by U).

Summarizing :

Au =n∑

i=1

λi 〈Ui ,u〉Ui

It would be great to generalize this to all matrices...

CS221 Linear Algebra Review 10/11/2011 32 / 37

Matrix Decompositions

Eigendecomposition of a Symmetric Matrix

The action of a symmetric matrix on a vector Au = UΛUTu can bedecomposed into :

1 Projection of u onto the column space of A (multiplication by UT ).

2 Scaling of each coe�cient 〈Ui ,u〉 by the corresponding eigenvalue(multiplication by Λ).

3 Linear combination of the eigenvectors scaled by the resultingcoe�cient (multiplication by U).

Summarizing :

Au =n∑

i=1

λi 〈Ui ,u〉Ui

It would be great to generalize this to all matrices...

CS221 Linear Algebra Review 10/11/2011 32 / 37

Matrix Decompositions

Eigendecomposition of a Symmetric Matrix

The action of a symmetric matrix on a vector Au = UΛUTu can bedecomposed into :

1 Projection of u onto the column space of A (multiplication by UT ).2 Scaling of each coe�cient 〈Ui ,u〉 by the corresponding eigenvalue

(multiplication by Λ).

3 Linear combination of the eigenvectors scaled by the resultingcoe�cient (multiplication by U).

Summarizing :

Au =n∑

i=1

λi 〈Ui ,u〉Ui

It would be great to generalize this to all matrices...

CS221 Linear Algebra Review 10/11/2011 32 / 37

Matrix Decompositions

Eigendecomposition of a Symmetric Matrix

The action of a symmetric matrix on a vector Au = UΛUTu can bedecomposed into :

1 Projection of u onto the column space of A (multiplication by UT ).2 Scaling of each coe�cient 〈Ui ,u〉 by the corresponding eigenvalue

(multiplication by Λ).3 Linear combination of the eigenvectors scaled by the resulting

coe�cient (multiplication by U).

Summarizing :

Au =n∑

i=1

λi 〈Ui ,u〉Ui

It would be great to generalize this to all matrices...

CS221 Linear Algebra Review 10/11/2011 32 / 37

Matrix Decompositions

Eigendecomposition of a Symmetric Matrix

The action of a symmetric matrix on a vector Au = UΛUTu can bedecomposed into :

(multiplication by Λ).3 Linear combination of the eigenvectors scaled by the resulting

coe�cient (multiplication by U).

Summarizing :

Au =n∑

i=1

λi 〈Ui ,u〉Ui

It would be great to generalize this to all matrices...

CS221 Linear Algebra Review 10/11/2011 32 / 37

Matrix Decompositions

Eigendecomposition of a Symmetric Matrix

The action of a symmetric matrix on a vector Au = UΛUTu can bedecomposed into :

(multiplication by Λ).3 Linear combination of the eigenvectors scaled by the resulting

coe�cient (multiplication by U).

Summarizing :

Au =n∑

i=1

λi 〈Ui ,u〉Ui

It would be great to generalize this to all matrices...

CS221 Linear Algebra Review 10/11/2011 32 / 37

Matrix Decompositions

Singular Value Decomposition

Every matrix has a singular value decomposition :

A = UΣVT

The columns of U are an orthonormal basis for the column space ofA.

The columns of V are an orthonormal basis for the row space of A.

Σ is diagonal. The entries on its diagonal σi = Σii are positive realnumbers, called the singular values of A.

The action of A on a vector u can be decomposed into :

Au =n∑

i=1

σi 〈Vi ,u〉Ui

CS221 Linear Algebra Review 10/11/2011 33 / 37

Matrix Decompositions

Singular Value Decomposition

Every matrix has a singular value decomposition :

A = UΣVT

The columns of U are an orthonormal basis for the column space ofA.

The columns of V are an orthonormal basis for the row space of A.

Σ is diagonal. The entries on its diagonal σi = Σii are positive realnumbers, called the singular values of A.

The action of A on a vector u can be decomposed into :

Au =n∑

i=1

σi 〈Vi ,u〉Ui

CS221 Linear Algebra Review 10/11/2011 33 / 37

Matrix Decompositions

Singular Value Decomposition

Every matrix has a singular value decomposition :

A = UΣVT

The columns of U are an orthonormal basis for the column space ofA.

The columns of V are an orthonormal basis for the row space of A.

Σ is diagonal. The entries on its diagonal σi = Σii are positive realnumbers, called the singular values of A.

The action of A on a vector u can be decomposed into :

Au =n∑

i=1

σi 〈Vi ,u〉Ui

CS221 Linear Algebra Review 10/11/2011 33 / 37

Matrix Decompositions

Singular Value Decomposition

Every matrix has a singular value decomposition :

A = UΣVT

The columns of U are an orthonormal basis for the column space ofA.

The columns of V are an orthonormal basis for the row space of A.

The action of A on a vector u can be decomposed into :

Au =n∑

i=1

σi 〈Vi ,u〉Ui

CS221 Linear Algebra Review 10/11/2011 33 / 37

Matrix Decompositions

Singular Value Decomposition

Every matrix has a singular value decomposition :

A = UΣVT

The columns of U are an orthonormal basis for the column space ofA.

The columns of V are an orthonormal basis for the row space of A.

The action of A on a vector u can be decomposed into :

Au =n∑

i=1

σi 〈Vi ,u〉Ui

CS221 Linear Algebra Review 10/11/2011 33 / 37

Matrix Decompositions

Properties of the Singular Value Decomposition

The eigenvalues of the symmetric matrix ATA are equal to thesquare of the singular values of A :

ATA = VΣUTUΣVT = VΣ2VT

The rank of a matrix is equal to the number of nonzero singularvalues.

The largest singular value σ1 is the solution to the optimizationproblem :

σ1 = maxx 6=0

||Ax||2

||x||2

It can be veri�ed that the largest singular value satis�es the propertiesof a norm, it is called the spectral norm of the matrix.

In statistics analyzing data with the singular value decomposition iscalled Principal Component Analysis.

CS221 Linear Algebra Review 10/11/2011 34 / 37

Matrix Decompositions

Properties of the Singular Value Decomposition

The eigenvalues of the symmetric matrix ATA are equal to thesquare of the singular values of A :

ATA = VΣUTUΣVT = VΣ2VT

The rank of a matrix is equal to the number of nonzero singularvalues.

The largest singular value σ1 is the solution to the optimizationproblem :

σ1 = maxx 6=0

||Ax||2

||x||2

It can be veri�ed that the largest singular value satis�es the propertiesof a norm, it is called the spectral norm of the matrix.

In statistics analyzing data with the singular value decomposition iscalled Principal Component Analysis.

CS221 Linear Algebra Review 10/11/2011 34 / 37

Matrix Decompositions

Properties of the Singular Value Decomposition

The eigenvalues of the symmetric matrix ATA are equal to thesquare of the singular values of A :

ATA = VΣUTUΣVT = VΣ2VT

The rank of a matrix is equal to the number of nonzero singularvalues.

The largest singular value σ1 is the solution to the optimizationproblem :

σ1 = maxx 6=0

||Ax||2

||x||2

It can be veri�ed that the largest singular value satis�es the propertiesof a norm, it is called the spectral norm of the matrix.

In statistics analyzing data with the singular value decomposition iscalled Principal Component Analysis.

CS221 Linear Algebra Review 10/11/2011 34 / 37

Matrix Decompositions

Properties of the Singular Value Decomposition

The eigenvalues of the symmetric matrix ATA are equal to thesquare of the singular values of A :

ATA = VΣUTUΣVT = VΣ2VT

The rank of a matrix is equal to the number of nonzero singularvalues.

The largest singular value σ1 is the solution to the optimizationproblem :

σ1 = maxx 6=0

||Ax||2

||x||2

CS221 Linear Algebra Review 10/11/2011 34 / 37

Matrix Decompositions

Properties of the Singular Value Decomposition

The eigenvalues of the symmetric matrix ATA are equal to thesquare of the singular values of A :

ATA = VΣUTUΣVT = VΣ2VT

The rank of a matrix is equal to the number of nonzero singularvalues.

The largest singular value σ1 is the solution to the optimizationproblem :

σ1 = maxx 6=0

||Ax||2

||x||2

CS221 Linear Algebra Review 10/11/2011 34 / 37

Application : Image Compression

1 Vectors

2 Matrices

3 Matrix Decompositions

4 Application : Image Compression

CS221 Linear Algebra Review 10/11/2011 35 / 37

Application : Image Compression

Compression using the Singular Value Decomposition

The singular value decomposition can be viewed as a sum of rank 1matrices :

If some of the singular values are very small, we can discard them.

This is a form of lossy compression.

CS221 Linear Algebra Review 10/11/2011 36 / 37

Application : Image Compression

Compression using the Singular Value Decomposition

The singular value decomposition can be viewed as a sum of rank 1matrices :

If some of the singular values are very small, we can discard them.

This is a form of lossy compression.

CS221 Linear Algebra Review 10/11/2011 36 / 37

Application : Image Compression

Compression using the Singular Value Decomposition

The singular value decomposition can be viewed as a sum of rank 1matrices :

If some of the singular values are very small, we can discard them.

This is a form of lossy compression.

CS221 Linear Algebra Review 10/11/2011 36 / 37

Application : Image Compression

Compression using the Singular Value Decomposition

Truncating the singular value decomposition allows us to represent thematrix with less parameters.

For a 512× 512 matrix :

Full representation : 512× 512 =262 144

Rank 10 approximation : 512× 10 + 10 + 512× 10 =10 250

Rank 40 approximation : 512× 40 + 40 + 512× 40 =41 000

Rank 80 approximation : 512× 80 + 80 + 512× 80 =82 000

CS221 Linear Algebra Review 10/11/2011 37 / 37

Application : Image Compression

Compression using the Singular Value Decomposition

Truncating the singular value decomposition allows us to represent thematrix with less parameters.

For a 512× 512 matrix :

Full representation : 512× 512 =262 144

Rank 10 approximation : 512× 10 + 10 + 512× 10 =10 250

Rank 40 approximation : 512× 40 + 40 + 512× 40 =41 000

Rank 80 approximation : 512× 80 + 80 + 512× 80 =82 000

CS221 Linear Algebra Review 10/11/2011 37 / 37

Application : Image Compression

Compression using the Singular Value Decomposition

Truncating the singular value decomposition allows us to represent thematrix with less parameters.

For a 512× 512 matrix :

Full representation : 512× 512 =262 144

Rank 10 approximation : 512× 10 + 10 + 512× 10 =10 250

Rank 40 approximation : 512× 40 + 40 + 512× 40 =41 000

Rank 80 approximation : 512× 80 + 80 + 512× 80 =82 000

CS221 Linear Algebra Review 10/11/2011 37 / 37

Application : Image Compression

Compression using the Singular Value Decomposition

Truncating the singular value decomposition allows us to represent thematrix with less parameters.

For a 512× 512 matrix :

Full representation : 512× 512 =262 144

Rank 10 approximation : 512× 10 + 10 + 512× 10 =10 250

Rank 40 approximation : 512× 40 + 40 + 512× 40 =41 000

Rank 80 approximation : 512× 80 + 80 + 512× 80 =82 000

CS221 Linear Algebra Review 10/11/2011 37 / 37

Application : Image Compression

Compression using the Singular Value Decomposition

Truncating the singular value decomposition allows us to represent thematrix with less parameters.

For a 512× 512 matrix :

Full representation : 512× 512 =262 144

Rank 10 approximation : 512× 10 + 10 + 512× 10 =10 250

Rank 40 approximation : 512× 40 + 40 + 512× 40 =41 000

Rank 80 approximation : 512× 80 + 80 + 512× 80 =82 000

CS221 Linear Algebra Review 10/11/2011 37 / 37

Application : Image Compression

Compression using the Singular Value Decomposition

Truncating the singular value decomposition allows us to represent thematrix with less parameters.

For a 512× 512 matrix :

Full representation : 512× 512 =262 144

Rank 10 approximation : 512× 10 + 10 + 512× 10 =10 250

Rank 40 approximation : 512× 40 + 40 + 512× 40 =41 000

Rank 80 approximation : 512× 80 + 80 + 512× 80 =82 000

CS221 Linear Algebra Review 10/11/2011 37 / 37