review of linear algebra (1)

8/19/2019 Review of Linear Algebra (1)

1/45

Chapter 1: Linear Algebra

- Vector space

-Linear transformation

- Equivalances for invertibility matrices


2/45

Scalar

• A quantity (variable), described by a singlereal number

Linear Algebra & Matrices, MfD 2010

e.g. Intensity of each

voxel in an MRI scan


3/45

Vector • Not a physics vector (magnitude, direction)


xn

x x21

i.e. A column

of numbers

VECTOR=

EXAMPLE:


4/45

Matrices• Rectangular display of vectors in rows and columns

• Can inform about the same vector intensity at different

times or different voxels at the same time

• Vector is just a n x 1 matrix


333231232221

131211

x x x x x x

x x x


5/45

Matrices

Square (3 x 3)

333231

232221

131211

d d d

d d d

d d d

D


Matrix locations/size defined as rows x columns (R x C)

d i j : ith row, jth column

333231232221

131211

x x x x x x

x x x

Rectangular (3 x 2)

333231

232221

131211

x x x

x x x

x x x

333231

232221

131211

x x x

x x x

x x x

333231

232221

131211

x x x

x x x

x x x

333231

232221

131211

x x x

x x x

x x x

963

852

741

A

6

5

4

3

2

1

A

3 dimensional (3 x 3 x 5)


6/45

Basic concepts

• Vector in Rn is an ordered

set of n real numbers.

– e.g. v = (1,6,3,4) is in R4

– “(1,6,3,4)” is a column

vector: – as opposed to a row

vector:

• m-by-n matrix is an object

with m rows and n columns,

each entry fill with a real

number:

4

3

6

1

4361

239

6784821


7/45

Matrices in MATLAB

963

5

4

11

11

0

0

0


963

852

741

X X=[1 4 7;2 5 8;3 6 9]Matrix(X)

;=end of a row

Description Type into MATLAB Meaning

2nd Element of 3rd column X(2,3) 8

X(3, :)3rd row

(X(row,column))Reference matrix values

Note the : refers to all of row or column and , is the divider between rows and columns

Elements 2&3 of column 2 X( [1 2], 2)

Special types of matrix

All zeros size 3x1

All ones size 2x2

zeros(3,1)

ones(2,2)


8/45

Transposition

column row row column


9

4

3T d

476

145

321

A

413

742

651

T

A

2

1

1

b 211T b 943d


9/45

Basic concepts

• Transpose: reflect vector/matrix on line:

bab

a T

d b

ca

d c

ba T

– Note: (Ax)T=xT AT (We’ll define multiplication soon…)

• Vector norms:

– Lp norm of v = (v1,…,vk) is (Σi |vi|p)1/p

– Common norms: L1, L2

– Linfinity = maxi |vi|

• Length of a vector v is L2(v)


10/45

Matrix Calculations

Addition – Commutative: A+B=B+A

– Associative: (A+B)+C=A+(B+C)

Subtraction

- By adding a negative matrix

6543

15320412

1301

5242

BA



11/45

Scalar multiplication

Scalar * matrix = scalar multiplication



12/45

Matrix Multiplication“When A is a m x n matrix & B is a k x l matrix, AB is only possible

if n =k . The result will be an m x l matrix ”

A 1 A 2 A 3

A 4 A 5 A 6

A 7 A 8 A 9

A 10 A 11 A 12

m

n

x

B13 B14

B15 B16

B17 B18

l

k

Simp ly put, can ONLY perform A *B IF:

Number of columns in A = Number of rows in B

= m x l matrix


Hint:

1) If you see this message in MATLAB:

??? Error using ==> mtimes

Inner matrix dimensions must agree -Then columns in A is not equal to rows in B


13/45

Matrix multiplication

• Multiplication method:Sum over product of respective rows and columns

Matlab does all this for you!Simply type: C = A * B


Hints:

1) You can work out the size of the output (2x2). In MATLAB, if you pre-allocate a

matrix this size (e.g. C=zeros(2,2)) then the calculation is quicker


14/45

Matrix multiplication

• Matrix multiplication is NOT commutativei.e the order matters!

– AB≠BA

• Matrix multiplication IS associative

– A(BC)=(AB)C

• Matrix multiplication IS distributive

– A(B+C)=AB+AC

– (A+B)C=AC+BCLinear Algebra & Matrices, MfD 2010


15/45

Identity matrix A special matrix which plays a similar role as the number 1 in number

multiplication?

For any nxn matrix A , we have A I n = I n A = A

For any nxm matrix A , we have I n A = A , and A I

m= A (so 2 possible matrices)


If the answers always A, why use an identity matrix?

Can’t divide matrices, therefore to solve may problems have to use the inverse. The

identity is important in these types of calculations.


16/45

Identity matrix

1 2 3 1 0 0 1+0+0 0+2+0 0+0+3

4 5 6 X 0 1 0 = 4+0+0 0+5+0 0+0+6

7 8 9 0 0 1 7+0+0 0+8+0 0+0+9

Worked

example A I 3 = A

for a 3x3 matrix:

• In Matlab: eye(r, c) produces an r x c identity matrix



17/45

Basic concepts

• Vector products:

– Dot product:

– Outer product:

22112

1

21 vuvuv

vuuvuvu T

2212

2111

21

2

1

vuvu

vuvuvv

u

uuvT


18/45

Matrices as linear transformations

5

5

1

1

50

05(stretching)

1

1

1

1

01

10(rotation)


19/45

Matrices as linear transformations

10

01

0110 (reflection)

0

1

1

1

00

01(projection)

(shearing)

y

cy x

y

xc

10

1


20/45

Matrices as sets of constraints

22

1

z y x

z y x

2

1

112

111

z

y

x


21/45

Special matrices

f

ed

cba

00

0

c

ba

00

0000

ji

h g f

ed c

ba

00

0

0

00

f ed

cb

a

0

00

diagonal upper-triangular

tri-diagonal lower-triangular

100

010

001

I (identity matrix)


22/45

Vector spaces

• Formally, a vector space is a set of vectors which isclosed under addition and multiplication by real numbers.

• A subspace is a subset of a vector space which is a

vector space itself, e.g. the plane z=0 is a subspace of

R3 (It is essentially R2.).

• We’ll be looking at Rn and subspaces of Rn

Our notion of planes in R3

may be extended to

hyperplanes in Rn

(ofdimension n-1)

Note: subspaces must

include the origin (zero

vector).


23/45

Linear system & subspaces

• Linear systems define certainsubspaces

• Ax = b is solvable iff b may be

written as a linear combination

of the columns of A• The set of possible vectors b

forms a subspace called the

column space of A

3

2

1

31

32

01

b

b

b

v

u

3

2

1

3

3

0

1

2

1

b

b

b

vu(1,2,1)

(0,3,3)


24/45

Linear system & subspaces

Null space: {(c,c,-c)}

0

0

0

31

32

01

vu

0

00

431

532101

z

y x

The set of solutions to Ax = 0 forms a subspace

called the null space of A.

Null space: {(0,0)}


25/45

Linear independence and basis

0

0

0

|||

|||

3

2

1

321

c

c

c

vvv

Recall nullspace contained

only (u,v)=(0,0).

i.e. the columns are linearly

independent.

• Vectors v1,…,vk are linearly independent ifc1v1+…+ckvk = 0 implies c1=…=ck=0

(0,1)

(1,0)

(1,1)

(2,2)

0

0

0

31

32

01

v

u

i.e. the nullspace is the origin


26/45


1

0

0

2

0

1

0

2

0

0

1

2

2

2

2

• If all vectors in a vector space may be

expressed as linear combinations of v1,…,vk,

then v1,…,vk span the space.

(0,0,1)

(0,1,0)

(1,0,0)

(.1,.2,1)

(.3,1,0)

(.9,.2,0)

1

2.

1.

2

0

1

3.

29.1

0

2.

9.

57.1

2

2

2


27/45


• A basis is a set of linearly independent vectors which span thespace.

• The dimension of a space is the # of “degrees of freedom” of the

space; it is the number of vectors in any basis for the space.

• A basis is a maximal set of linearly independent vectors and a

minimal set of spanning vectors.

1

0

0

2

0

1

0

2

0

0

1

2

2

2

2

(0,0,1)

(0,1,0)

(1,0,0)

(.1,.2,1)

(.3,1,0)

(.9,.2,0)

1

2.

1.

2

0

1

3.

29.1

0

2.

9.

57.1

2

2

2


28/45


• Two vectors are orthogonal if their dot product is 0.• An orthogonal basis consists of orthogonal vectors.

• An orthonormal basis consists of orthogonal vectors

of unit length.

1

0

0

2

0

1

0

2

0

0

1

2

2

2

2

(0,0,1)

(0,1,0)

(1,0,0)

(.1,.2,1)

(.3,1,0)

(.9,.2,0)

1

2.

1.

2

0

1

3.

29.1

0

2.

9.

57.1

2

2

2


29/45

About subspaces

• The rank of A is the dimension of the column space of A.

• It also equals the dimension of the row space of A (the

subspace of vectors which may be written as linear

combinations of the rows of A).

3132

01 (1,3) = (2,3) – (1,0)

Only 2 linearly independentrows, so rank = 2.


30/45

About subspacesFundamental Theorem of Linear Algebra:

If A is m x n with rank r,

Column space(A) has dimension r

Nullspace(A) has dimension n-r (= nullity of A)

Row space(A) = Column space(AT) has dimension r Left nullspace(A) = Nullspace(AT) has dimension m - r

Rank-Nullity Theorem: rank + nullity = n => prove it!

(0,0,1)

(0,1,0)

(1,0,0)

00

10

01m = 3

n = 2

r = 2


31/45

Non-square matrices

1

1

1

32

10

01

2

1

x

x

32

10

01m = 3

n = 2

r = 2

System Ax=b may

not have a solution

(x has 2 variablesbut 3 constraints).

310

201

m = 2

n = 3

r = 2System Ax=b is

underdetermined

(x has 3 variables

and 2 constraints).

1

1

310

201

3

2

1

x

x

x


32/45

Basis transformations

• Before talking about basis transformations,

we need to recall matrix inversion and

projections.


33/45

Matrix inversion

• To solve Ax=b, we can write a closed-form solution if

we can find a matrix A-1

s.t. AA-1 =A-1 A=I (identity matrix)

• Then Ax=b iff x=A-1

b:x = Ix = A-1 Ax = A-1b

• A is non-singular iff A-1 exists iff Ax=b has a unique

solution.

• Note: If A-1,B-1 exist, then (AB)-1 = B-1 A-1,

and (AT)-1 = (A-1)T


34/45

Projections

2

2

2

000

010

001

0

2

2

(0,0,1)

(0,1,0)

(1,0,0)

(2,2,2)

a = (1,0)

b = (2,2)

0

2a

aa

bac

T

T


35/45


2

29.1

57.1

100

2.12.

1.3.9.

2

2

2

2

2

2

100

010

001

2

2

2

(0,0,1)

(0,1,0)

(1,0,0)

(.1,.2,1)

(.3,1,0)

(.9,.2,0)

We may write v=(2,2,2) in terms of an alternate basis:

Components of (1.57,1.29,2) are projections of v onto new basis vectors,

normalized so new v still has same length.


36/45


Given vector v written in standard basis, rewrite as vQin terms of basis Q.

If columns of Q are orthonormal, vQ = QTv

Otherwise, vQ = (QTQ)QTv


37/45

Special matrices

• Matrix A is symmetric if A = AT

• A is positive definite if xT Ax>0 for all non-zero x ( positive semi-

definite if inequality is not strict)

222

100

010

001

cba

c

b

a

cba

222

100

010

001

cba

c

b

a

cba


38/45

Special matrices

• Matrix A is symmetric if A = AT

• A is positive definite if xT Ax>0 for all non-zero x ( positive semi-

definite if inequality is not strict)

• Useful fact: Any matrix of form AT A is positive semi-definite.

To see this, xT(AT A)x = (xT AT)(Ax) = (Ax)T(Ax) ≥ 0 => prove

it!


39/45

Determinants

• If det(A) = 0, then A is

singular.

• If det(A) ≠ 0, then A is

invertible.• To compute:

– Simple example:

– Matlab: det(A)

bcad d c

ba

det


40/45

Determinants

• m-by-n matrix A is rank-deficient if it has

rank r < m (≤ n)

• Thm: rank(A) < r iff

det(A) = 0 for all t-by-t submatrices,

r ≤ t ≤ m


41/45

Eigenvalues & eigenvectors

• How can we characterize matrices?

• The solutions to Ax = λx in the form of eigenpairs

(λ,x) = (eigenvalue,eigenvector) where x is non-zero

• To solve this, (A – λI)x = 0• λ is an eigenvalue iff det(A – λI) = 0


42/45


(A – λI)x = 0

λ is an eigenvalue iff det(A – λI) = 0

Example:

2/100

64/30541

A

)2/1)(4/3)(1(

2/100

64/30541

)det(

I A

2/1,4/3,1


43/45


10

02 A Eigenvalues λ = 2, 1 with

eigenvectors (1,0), (0,1)

(0,1)

(2,0)

Eigenvectors of a linear transformation A are not rotated (but will be

scaled by the corresponding eigenvalue) when A is applied.

v Av

(1,0)


44/45

Solving Ax=b

x + 2y + z = 0y - z = 2

x +2z= 1

-------------

x + 2y + z = 0y - z = 2

-2y + z= 1

-------------

x + 2y + z = 0

y - z = 2

- z = 5

5100

2110

0121

1120

2110

0121

1201

2110

0121 Write system ofequations in matrixform.

Subtract first row fromlast row.

Add 2 copies of secondrow to last row.

Now solve by back-substitution: z = -5, y = 2-z = 7, x = -2y-z = -9


45/45

Solving Ax=b & condition numbers

• Matlab: linsolve(A,b)

• How stable is the solution?

• If A or b are changed slightly, how much does it

effect x?• The condition number c of A measures this:

c = λmax/ λmin

• Values of c near 1 are good.

review of linear algebra (1)

Documents