math321 lec2 solutions to systems of linear equations, eigenvalues and eigenvectors (1).pdf

7/28/2019 MATH321 Lec2 Solutions to Systems of Linear Equations, Eigenvalues and Eigenvectors (1).pdf

http://slidepdf.com/reader/full/math321-lec2-solutions-to-systems-of-linear-equations-eigenvalues-and-eigenvectors 1/45

ECE-MATH321 engradc (c)2011

Solutions to Systems

of Linear Equations Angelo R. dela Cruz, ECE




Topicsz Rank of a Matrix

z Homogeneous and Non-Homogeneousz Cramer’s Rule

z Inverse Method

z QR-Decomposition

z Gaussian Elimination

z Lower-Upper (LU) Decompositionz Cholesky Factorization

z Eigenvalues and Eigenvectors




Rank of a Matrix, r

z It is the greatest value of r for which there

exists an r x r submatrix of A with nonvanishing determinants.

z The rank of an m x n matrix can at most beequal to the smaller of the numbers m and n

but may be less.

mnr nmr <≤<≤ OR




Rank of a Matrix, r

z An n x n matrix has a rank less than n if and

only if A is singular matrix.

z Full rank can be achieved if and only if A is a

square and non-singular matrix.

0if =< Anr

0if ≠= Anr




Examplez Determine the rank of the matrices given

below

⎥⎦

⎤

⎢⎣

⎡

−−−

−=

8246

4123a.) A

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−−

−−

−−

=

3121

95636342

b.) A




System of Linear Equations:

General Formz n unknowns with m equations

z Where aij

are coefficients of the unknowns xk

.

mnmnmm

nn

nn

nn

b xa xa xa

b xa xa xa

b xa xa xa

b xa xa xa

=+++

=+++

=+++=+++

...

......

...

...

...

2211

33232131

22222121

11212111




General Formz Using matrix form (m x n)

⎥⎥⎥

⎥

⎦

⎤

⎢⎢⎢

⎢

⎣

⎡

=

⎥⎥⎥

⎥

⎦

⎤

⎢⎢⎢

⎢

⎣

⎡

⎥⎥⎥

⎥

⎦

⎤

⎢⎢⎢

⎢

⎣

⎡

mnmnmm

n

n

b

b

b

x

x

x

aaa

aaa

aaa

::

...

::::...

...

2

1

2

1

21

22221

11211

bxA=




Homogeneous and

Non-Homogeneous

z If vector b is a null vector i.e.

A x = 0

thus, The given linear equations are

homogeneous

z If vector b is not a null vector thus, the given

linear equations are non-homogeneous




Solution to Homogeneous

Linear Equationsz If the system of linear equations is

homogeneous i.e. b = 0, and A is

non-singular square matrix (full-rank),

therefore the system has only trivialsolution i.e.

x1 = x2 = x3 = … = xn = 0




Solution to Homogeneous


homogeneous i.e. b = 0, and A is

not a full rank matrix whose r < m, consider

only r equations whose coefficient matrix hasa rank r and omitting the other m–r

equations. There will be an infinite solutionto the system




Examplesz Determine the solution of the following:

a. 2x – 4y = 0, x + y = 0

b. x + y + z = 0

2x – y – 5z = 0

x – y – 5z = 0




Examplesz Determine the solution of the following:

a. x + y + z = 0, 2x – y – 7z = 0,

x – y – 5z = 0

b. 2x + y + z = 0

4x + 3y – z = 0




Solution to Non-Homogeneous


non-homogeneous i.e. b ≠ 0, and A isnot full-rank matrix: Let

⎥⎥⎥

⎥

⎦

⎤

⎢⎢⎢

⎢

⎣

⎡

=

⎥⎥⎥

⎥

⎦

⎤

⎢⎢⎢

⎢

⎣

⎡

=

nmnmm

n

n

mnmm

n

n

baaa

baaa

baaa

aaa

aaa

aaa

...

:::::...

...

,

...

::::...

...

21

222221

111211

21

22221

11211

BA





Linear Equationsz Rules:

1. If rank(A) = rank(B), exact solution exista. If rank(A) < m, consider only r equations whose

coefficient matrix has rank r . Omit the other m-r

equations

b. Transpose n-r unknowns to the right member of

each equation, retaining the r unknowns in the left

member whose coefficient matrix has a rank r .

c. Solve the system using Cramer’s rule




Cramer’s Rule

nnnn

n

n

aba

aba

aba

x D

....

:::::

:::::

....

....

)(

1

2221

1111

2 =

nnnn

n

n

aab

aab

aab

x D

....

:::::

:::::

....

....

)(

2

2222

1121

1 =

nnn

n

baa

baa

baa

x D

....

:::::

:::::

....

....

)(

21

22221

11211

= AAA)(...,,)(,)( 2

21

1n

n x D x x D x x D x ===




QR-Decomposition Steps

[ ] [ ]

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=

33

2322

131211

3232

00

0

r

r r

r r r

qqqaaa 11

Consider a 4x3 matrix A

Where:

ak Æ 4x1 column vectors of A

qk Æ 4x1 orthonormal vectors

R Æ 3x3 upper triangular matrix




QR-Decomposition Steps

( )

( )( )

3

33333

3223

3223313

2

22222

2212

11

ˆˆ,ˆ

,

ˆ,

ˆ

ˆ,ˆ

ˆ,

ˆ

ˆ

,ˆ

ˆ

qqqq

aq

aqqIqqIqaq

q

qqq

aqqIqaq

q

q

qq

aq

111

1121

1

111

11

==

=−−==

==

−==

==

=

r

r

r

r

r

r

T

T T T

T T




QR-Decomposition General Steps

( )

ji j

T

iij

k kk

k

k k

k k k k k k k

k

T

k k

T T

k

T

k k

T T

k

r

r

r r r r

≠

−−

−−

−−

=

=

=

−−−−−=

−−−−=

−−−=

aq

q

q

qq

qqqqa

aqqqqqqI

aqqIqqIqqIq

1

11

11

ˆ

ˆ

ˆ

...

...

...ˆ

1,13,32,2,1

1122

1122




QR-Decomposition

bQIRx

bQQRxQ

bQRx

bAx

T

T T

=

==

=

bQRx T =

Note that if Q is orthonormal matrix,

QT Q = I

Note that since R is upper triangular matrix,

x can be solved using backward substi tution

Case 1: Overdetermined System of Linear Equations

The Least-Square Method




QR-Decomposition

yQx

byR

QR A

=

=

=T

T

yÆ can be solved using forward substitution

Case 2: Underdetermined System of Linear Equations

The Least-Norm Method




Examplesz Solve the following systems of linear

equations

a. 2x – 5y + 13z = –24

3x + y – 6z = –2

b. x – y + z = 4, x + y – z = 0

2x – 3y + z = 4, 3x + 2y – 3z = –1

S l ti t N H





Linear Equationsz Rules:

2. If rank(A) ≠ rank(B), the system has no exactsolution and the system can be solved by

estimation if m > n (more equations than

unknowns)

Typical solution to best estimate the unknownsis Method of Least Squares using QR-

Decomposition method




Method of Least Squares

z Since the solution is not exact, The residual

error (vector), e is the distance vector of theactual vs. estimate and the loss function, ϕ

(scalar) is the square norm of the error:

ee0xAbe

T φ =

≠−=min

ˆ




QR-Decomposition Example

Obtain the best solution given an overdeterminedsystem of equations

⎥⎥

⎥⎥

⎦

⎤

⎢⎢

⎢⎢

⎣

⎡

=

⎥⎥

⎥

⎦

⎤

⎢⎢

⎢

⎣

⎡

⎥⎥

⎥⎥

⎦

⎤

⎢⎢

⎢⎢

⎣

⎡

−

−−

00

0

1

011241

241

411

3

2

1

x

x

x

Solution to Non Homogeneous






non-homogeneous i.e. b ≠ 0, and A isa full-rank matrix r = n

z

Can be solved by:z Inverse method

z Gaussian elimination

z LU-decompositionz QR-Decomposition

z Cholesky Factorization⎥⎥

⎥⎥

⎦

⎤

⎢⎢

⎢⎢

⎣

⎡

=

nnnn

n

n

aaa

aaa

aaa

...

::::

...

...

21

22221

11211

A




Inverse Methodz Since Ax = b

Premultiply both sides of the equation by A-1

bAxbAxI

bAxAA

-1

-1

-1-1

==

=




Example1. Solve for the unknowns, x and y using

Inverse method and compare it to thesolution using Cramer’s rule

2x + 3y = 5

3x – y = 4




Gaussian Eliminationz This method involves replacing equations by

obtaining its triangular matrix form

z The unknowns are computed using backward

substitution

nnnn

nn

nn

c xu

c xu xu

c xu xu xu

=

=++

=+++

:............

...

...

22222

11212111




Examplez Determine the solution using Gaussian

elimination method

3 x + 4 y + 2 z – w = 24 x + 2 y – 3 z + 2w = 4

–5 x + 3 y – 4 z + 3w = 4

3 x + y + z + 2w = 2

Lower-Upper (LU)




Lower-Upper (LU)

Decomposition Methodz From the form Ax = b

z Decompose A = LU where L is a lower

triangular matrix and U is an upper triangular

matrix

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

=

nn

n

n

u

uu

uuu

...00

::::...0

...

222

11211

U

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

=

1...

::::0...1

0...01

21

21

nn l l

l L

Lower-Upper (LU)




Lower-Upper (LU)

Decomposition Methodz After decomposing A = LU, therefore

LUx = b.

z Let y = Ux, thus Ly = b where y can be

solved by forward substitution

z After solving y, since y = Ux, x can be

solved by backward substitution




Examplez Determine the solution using

LU-decomposition and QR-decompositionmethod

3 x + 4 y + 2 z – w = 2

4 x + 2 y – 3 z + 2w = 4

–5 x + 3 y – 4 z + 3w = 4

3 x + y + z + 2w = 2




Cholesky-Decomposition

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

=

44

4333

423222

41312111

44434241

333231

2221

11

000

00

0

0

00000

l

l l

l l l l l l l

l l l l

l l l

l l l

A

A = LLT for 4x4 matrix

If A is real symmetric positive definite square matrix

A is posit ive definite square matrix iff its eigenvalues are real > 0





jil l al

l

l al

j

k

k jk i ji

j j

ji

j

k

k j j j j j

>⎟⎟ ⎠

⎞⎜⎜⎝

⎛ −=

−=

∑

∑−

=

−

=

for ,1

1

1

,,,

,

,

1

1

2,,,

The elements of L can be computed easily using the

Cholesky algorithm shown below





⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−

−

=

33

3222

312111

333231

2221

11

00

00

00

522

2102

224

l

l l

l l l

l l l

l l

l

A

Example: Obtain the Cholesky Decomposition of A




Eigenvalues and Eigenvectorsz Given a square matrix A, a non-trivial vector

v and a scalar λ that satisfy the relationship

Av = λv

z Where λÆ eigenvalue of AvÆ corresponding eigenvector




Eigenvalues and EigenvectorsAv – λv = 0

Note Iv = v

Av – λIv = 0

(A – λI) v = 0

z Note that v = 0 is a possible solution but it is

a trivial solution!




Eigenvalues and Eigenvectorsz To get a non-trivial solution

det (A – λI) = 0

z This leads to a polynomial equation f (λ) = 0of degree equal to rank of A

z The roots of f (λ) are the eigenvalues of A




Eigenvectorsz The eigenvector v(i), that corresponds to

eigenvalue λi is computed as

(A – λiI) v(i)

= 0

z The non-trivial solution of v(i)

is theeigenvector of the corresponding eigenvalue

λi




Eigenvectorsz The eigenvector v(i), of the corresponding

eigenvalue λi is not unique, thus a

normalization factor η is needed to have a

common eigenvector 2)(2)(

2

2)(

1

...

vvv

i

n

ii+++=η

)()( 1ˆ ii

vv η =




Eigen-Decomposition

[ ] [ ] 1

2

22

11

2

000

000

00

−

⎥⎥

⎥⎥

⎦

⎤

⎢⎢

⎢⎢

⎣

⎡

= n

nn

n

..

:..:

..

v...vvv...vvA 11

λ

λ

λ

A = QΛQ –1

Where:

QÆ eigenvector matrix arrange in column vectors

ΛÆ diagonal matrix whose elements are the eigenvalues of A




Examplez Solve for the eigenvalues and the

eigenvector of the greatest positiveeigenvalue. Decompose A = QΛQ –1

⎥⎥

⎥

⎦

⎤

⎢⎢

⎢

⎣

⎡

−=

423

362

235

A

math321 lec2 solutions to systems of linear equations, eigenvalues and eigenvectors (1).pdf

Documents