Review of Matrix Operations

Vector: a sequence of elements (the order is important)

e.g., x = (2, 1) denotes a vector

length = sqrt(2*2+1*1)

orientation angle = a

x = (x1, x2, ……, xn), an n dimensional vector

a point in an n dimensional space

column vector: row vector

8

5

2

1

x

a

Txy )8521(

xx TT )(

X (2, 1)

transpose

norms of a vector: (magnitude)

vector operations:

1 11

2 1/ 2

2 12

1

norm:

norm: ( )

norm: max

n

i i

n

i i

ii n

L x x

L x x

L x x

1 2

1 1

2 2

1 2 1 21

( , ,...... ) : a scaler, : a column vector

inner (dot) product

, are column vectors of same dimension

( , ...... ) ( , ... )

T

n

nT

n i i ni

n n

rx rx rx rx r x

x y n

y x

y xx y x y x x x x y y y y

y x

M MTy x

Cross product:

defines another vector orthogonal to the plan

formed by x and y.

1

2 2

1 21 1

( , ...... ) 0

n nT

n i i ii i

n

x

xx x x x x x x x

x

M

yx

(dot product of weight vector and input vector)net w x

Matrix:

the element on the ith row and jth column a diagonal element (if m = n) a weight in a weight matrix W

each row or column is a vectorjth column vectorith row vector

11 12 1

1 2

......

( )

......

n

m n i j m n

m m mn

a a a

A a

a a a

M

1

x 1

:

:

( ...... )

j

i

m n n

m

a

a

a

A a a

a

M

:::

ij

ii

ij

waa

a column vector of dimension m is a matrix of m x 1

transpose:

jth column becomes jth row

square matrix:

identity matrix:

mnnn

mT

nm

aaa

aaa

A

......

......

21

12111

nnA

otherwise0

if1

1......00

0......100.....01

jiaI ji

symmetric matrix: m = n and

matrix operations:

The result is a row vector, each element of which is an inner product of and a column vector

jiijiiT aaijoraaiorAA ,,

)(),......( 1 jin rarararA

),......(),......)(......(

1

11

nTT

nmnmT

axaxaaxxAx

Tx ja

product of two matrices:

vector outer product:

jiijpmpnnm baCCBA where

nmnnnm AIA

1 1 1 2 11

1

1 2

, ,......

......

, , ......

n

Ti n

m m m nm

x y x y x yx

x y x y y

x y x y x yx

M

M

M

• Two vectors

are said to be orthogonal to each other if

• A set of vectors of dimension n are said to be linearly independent of each other if there does not exist a set of real numbers which are not all zero such that

otherwise, these vectors are linearly dependent and each one can be expressed as a linear combination of the others

T T1 1( ,..., ) and ( ,..., )n nx x x y y y

Linear Algebra

ni ii yxyx 1 .0

)()1( ,..., kxx

kaa ,...,1(1) ( )

1 0kka x a x L

j( i ) (1) (k ) ( j)1 k

j i

i i i

aa ax x x x

a a a L

• Vector x != 0 is an eigenvector of matrix A if there exists a constant such that Ax = x is called a eigenvalue of A (wrt x)

– A matrix A may have more than one eigenvectors, each with its own eigenvalues

• Ex. has 3 eigenvalues/eigenvectors

• Matrix B is called the inverse matrix of square matrix A if AB = I (I is the identity matrix)

– Denote B as A-1

– Not every square matrix has inverse (e.g., when one of the row can be expressed as a linear combination of other rows)

• Every matrix A has a unique pseudo-inverse A*, which satisfies the following properties

AA*A = A; A*AA* = A*; A*A = (A*A)T; AA* = (AA*)T

Ex. A = (2 1 -2), A* = (2/9 1/9 -2/9) T

Calculus and Differential Equations• the derivative of , with respect to time

• System of differential equations

solution:

difficult to solve unless are simple

( )ix t& ，

1 1

2 32 2

( ) sin( ) ( ) cos( ) has a solution

( ) ( ) / 3

x t t x t t

x t t x t t

&

&

ix t

1( ( ), ( ))nx t x tL)(tfi

1 1( ) ( )

( ) ( )n n

x t f t

x t f t

&

M&

dynamic system:

– change of may potentially affect other x

– all continue to change (the system evolves)

– reaches equilibrium when

– stability/attraction: special equilibrium point

(minimal energy state)

– pattern of at a stable state often represents a solution of the problem

1 1 1

1

( ) ( , ..... )

( ) ( , ...... )

n

n n n

x t f x x

x t f x x

&

M&

ix0 ix i &

)......,( 1 nxx

ix

• Multi-variable calculus:

partial derivative: gives the direction and speed of

change of y with respect to . Ex.

1 2( , , , )ny f x x x K

)(

3

)(2

2

)(1

1

)(221

321

321

321

321

2

)cos(

)sin(

xxx

xxx

xxx

xxx

ex

y

exx

y

exx

y

exxy

ix

the total derivative of

gives the direction and speed of change of y, with respect to t

Gradient of f :

Chain-rule: z is a function of y, y is a function of x, x is a function of t

11

( , , )1

( )( )( )

( )( )

n

Tf

df f fndt x x

n

x tx ty t

x tx t

L

L

&&&

&&

)......,(1 n

f x

f

x

f

dt

dx

dx

dy

dy

dz

dt

dz

1 2( ) ( ( ), ( ), , ( ))ny t f x t x t x t K