engg 407_p12_l20_lecture_03

ENGG 407

Numerical Methods in Engineering

P12L20

Lecture #3

Dr. Sameh Nassar

ENGG 407 L20 Spring 2012 Lecture #3 Dr. Sameh Nassar

Recall: ENGG 407 W12 Topics

1. Introduction and Mathematical Background (Ch. #1, #2)

2. “Roots of ” Nonlinear Equations (Ch. #3)

3. Linear Equations and Systems (Ch. #4)

© Sameh Nassar

4. Interpolation, Least-squares Estimation and Curve Fitting (Ch. #5)

5. Numerical Differentiation (Ch. #6)

6. Numerical Integration (Ch. #7)

7. Ordinary Differential Equations: Initial Value Problems (Ch. #8)

8. Ordinary Differential Equations: Boundary Value Problems (Ch. #9)


Linear Algebraic Equations

• Simultaneous linear algebraic equations:

1n1n313212111

bxa....xaxaxa

bxa....xaxaxa

=++++

=++++ a’s:

Constant

Coefficients

© Sameh Nassar

nnnn3n32n21n1

2n2n323222121

bxa....xaxaxa

bxa....xaxaxa

=++++

=++++

MM

MM

Coefficients

b’s:

Constants


• In Matrix form:

{B}[A]{x} = or simply bAx =


© Sameh Nassar

[A] n*n square matrix of coefficients

{B} n*1 column vector of constants

{x} n*1 column vector of unknowns


2n232221

1n131211

a......aaa

a......aaa

2

1

x

x

2

1

b

b


© Sameh Nassar

=

nnn3n2n1

3n333231

2n232221

a......aaa

a......aaa

MMMMM

A

=

n

3

2

x

x

M

x

=

n

3

2

b

b

M

b


Small Number of Equations

Methods:

• Small sets of simultaneous equations means in general that: n ≤ 3.

© Sameh Nassar

Methods:

(1) Graphical method

(2) Cramer’s rule

(3) Elimination of unknowns

Something you know or

may tried before


Large Number of Equations

Methods:

(1) Gauss Elimination

• Means that we have to use “Numerical Methods”.

© Sameh Nassar


(2) Gauss-Jordan Elimination

(3) LU Decomposition

(4) Jacobi

(4) Gauss-Seidel

Direct (Elimination) Methods

Iterative methods


Graphical Method (1/2)

Case of 2 Equations (for example), i.e. n =2:

© Sameh Nassar


Graphical Methods (2/2)

)2(2x2x

)1(18x2x3

21

21

=+−

=+

© Sameh Nassar

)a2(1x2

1x

)a1(9x2

3x

12

12

+=

+−=


Cramer’s Rule


© Sameh Nassar

||

aab

aab

aab

x 33323

23222

13121

1 A=

||

baa

baa

baa

x 33231

22221

11211

3 A=

||

aba

aba

aba

x 33331

23221

13111

2 A=

Where | | is the corresponding Determinant


Elimination of Unknowns


)2(2x2x

)1(18x2x3

21

21

=+−

=+

© Sameh Nassar

)b2(16x4218x3x 111 −=−⇒=+−− 4x1 =

)a2(2)9x2

3(2x

)a1(9x2

3x

11

12

=+−+−

+−=

394*2

39x

2

3x 12 =+−=+−=

x2 elimination

x1 back-substitution


Problems with Some Linear Equations

© Sameh Nassar

No SolutionInfinite No. of Solutions

Singular Systems

Ill-Conditioned Systems


Gauss Elimination Method

• It is an extension of the method ofElimination of Unknowns to large sets of equations by developing a systematic scheme or algorithm to eliminate

Naïve Gauss Elimination (or simply Gauss Elimination)

© Sameh Nassar

scheme or algorithm to eliminate unknowns and then back substitute.

• Therefore, as in the shown case of the solution of 2 equations, the technique for n equations consists of two stages (phases):

- Forward elimination of unknowns

- Back substitution




= 2

1

b

b

b

© Sameh Nassar

=

3

2

b

bb

3333231

2232221

1131211

baaa

baaa

baaa

M

M

M

Augmented Matrix



3333231

2232221

1131211

baaa

baaa

baaa

M

M

M

111

212 R)

a

a(R −

131

3 R)a

a(R −

Phase 1:Forward Elimination

© Sameh Nassar

3333231

333

22322

1131211

ba00

baa0

baaa

""

'''

M

M

M

33332

22322

1131211

baa0

baa0

baaa

'''

'''

M

M

M

111

3 a

222

323 R)

a

a(R

''−



33

3

a

b

""=3x

Phase 2: Back Substitution

1131211 baaa M

© Sameh Nassar

33a"

11

12131

a

aab 231

xxx

−−=

22

232

a

ab

''' 3

2

xx

−=

333

22322

ba00

baa0

""

'''

M

M



Pivot Equation

Pivot Coefficientor

© Sameh Nassar

or Pivot Element

Division of a row Ri by its pivot element (aii , a’ii ,

etc.), is called: Row Normalization



"""

''''

n

n

11n131211

baa

baaa

baaaa

M

M

M

......00

......0

......

3333

222322

(1) Forward Elimination

In General:

© Sameh Nassar

−− 1)(n1)(n

nnn ba M

MMMMMM

......000

......

1,....,2,11 −−=−

= −+=

−− ∑nnifor

a

xab

xii

n

ijjiji

i 1)(i

1)(i1)(i

1)-(n

1)-(n

nn

nn a

bx =

(2) Back Substitution “Upper” Triangular Matrix


Example #1

Use Gauss Elimination to solve:

© Sameh Nassar

Check your answers by substituting them into the

original equations.


Example #1 (Sol. “1/3”)

−−5.62*)

10(4 =−−

© Sameh Nassar

5.02*)8

10(2 −=−

0.52*)8

12(2 =−−

−−

62212

44210

2228

M

M

M

Augmented Matrix

12 R)8

10(R −

13 R)8

12(R −

5.62*)8

10(4 =−−

5.62*)8

(4 =−−

0.12*)8

12(2 −=−

0.92*)8

12(6 =−−



−−−

5.65.65.00

2228

M

M

© Sameh Nassar

85.6*)5.0

0.1(5 −=−−−

45.6*)5.0

0.1(9 −=−−−

− 9510 M23 R)5.0

0.1(R−−−

−−−

−−

0.40.800

5.65.65.00

2228

M

M

M



5.08

4x3 =

−−=

−−−

−−

0.40.800

5.65.65.00

2228

M

M

M

5.65.0

5.0*5.65.6x2 −=

−−=

© Sameh Nassar

5.18

5.6*2)5.0*2(2x1 =−−−−−=

6)5.0(2)5.6(2)5.1(12

4)5.0(4)5.6(2)5.1(10

2)5.0(2)5.6(2)5.1(8

=+−+

=+−+

−=−−+

Check:



(1) Round-off Errors:become important when large numbers of equations are solved. Everyresult is dependent on previous results. Hence, an error in the earlysteps will tend to propagate.

Problems Encountered

© Sameh Nassar

(2) Ill-Conditioned Systems:are those where small changes in coefficients results in large changesin the solution.

(3) Division by Zero:For example if a11 is zero, the elimination procedure fails. On the otherhand, although there may be no zeros on the main diagonal in theoriginal matrix, the elimination process may create zero aii elements on

the main diagonal in the final matrix.

Similar problems may occur if a11 (or any resulted aii) is close to zero.



(1) Use of More Significant Figures:To help in solving the problems of:• Round-off errors• Ill-conditioned systems

Overcoming Encountered Problems (1/2)

© Sameh Nassar

• Ill-conditioned systems

(2) Pivoting:Before each row is normalized, the largest available coefficient in thecolumn below the pivot element is determined. The rows can then beswitched so that the largest element is the pivot element. Thisapproach is called “Partial Pivoting”.

Ex:

−−

62212

44210

2228

M

M

M

−− 2228

44210

62212

M

M

M



(2) “Partial” Pivoting:• Avoids division by zero• Minimizes Round-off errors• Serves as a partial remedy

Overcoming Encountered Problems (2/2)

© Sameh Nassar

• Serves as a partial remedy for Ill-conditioning.

(3) Scaling:When the magnitude of the elements of one or more of the equationsare greatly different from the magnitudes of the elements of the otherequations, this leads to significant round-off errors .

Scaling is accomplished by dividing the elements of each row by thelargest element in the row (thus, the resulted largest element is 1.0).

After scaling, pivoting is employed. (This is known as scaled pivoting).


Scaling and Partial Pivoting (1/2)

−

−

01.0152.03.0

67.09.115.0

44.05.03.01.0

M

M

M

−−

01.0152.03.0

44.05.03.01.0

67.09.115.0

M

M

MPivoting only,

no scaling

9.1byRdivide

5.0byRdivide 1

Scaling

−

0.352610.52630.2632

88.016.02.0

M

M

−

67.09.115.0

44.05.03.01.0

M

M

© Sameh Nassar

−

0.01-152.03.0

0.352610.52630.2632

88.016.02.0

M

M

M

− 88.016.02.0

0.352610.52630.2632

0.01-152.03.0

M

M

MScaling, thenpivoting using

scaled coefficients

Scaling, then pivotingusing original coefficients(Recommended approach)

−

−

01.0152.03.0

67.09.115.0

44.05.03.01.0

M

M

M

−

−

44.05.03.01.0

67.09.115.0

01.0152.03.0

M

M

M

1byRdivide

9.1byRdivide

3

2Scaling

0.01-152.03.0

0.352610.52630.2632

M

M

− 01.0152.03.0

67.09.115.0

M

M


Scaling and Partial Pivoting (2/2)

• As shown, scaling itself will sometimes introduce round-off errors.

• Thus, in most Gauss elimination applications, scaling is employed

© Sameh Nassar

applications, scaling is employed to calculate scaled values of the coefficients solely as a criterion for pivoting but the original coefficient values are retained for the actual elimination and substitution computations; details on previous slide!


Example #2

Given the equations:

347xx3x

38x6x2x

321

321

−=+−−

−=−−

© Sameh Nassar

(a) Solve by Gauss elimination with partial pivoting only (i.e. no scaling). Show all steps of the computations.

(b) Substitute your results into the original equations to check your answers.

202xx8x 321 −=−+−



202xx8x

347xx3x

38x6x2x

321

321

321

−=−+−

−=+−−

−=−−

−−−−−−−−−

20218

34713

38162

M

M

MAugmented

Matrix

5.2620*)3

(34 −=−−−−

© Sameh Nassar

−−−−−−−−−

38162

34713

20218

M

M

M

-1.3751*)8

3(1 =−−−−

5.12*)8

2(1 −=−−

−−

12 )8

3( RR−−−

13 )8

2( RR−

−

75.72*)8

3(7 =−−−−

-5.751*)8

2(6 =−

−−

5.2620*)8

(34 −=−−

−−

4320*)8

2(38 −=−−

−−



−−−−−−−−

435.175.50

5.2675.7375.10

20218

M

M

M

-16.217443*)75.5

375.1(5.26 =−

−−−−

© Sameh Nassar

−−−−−−−−

5.2675.7375.10

435.175.50

20218

M

M

M

8.1086965.1*)75.5

375.1(75.7 =−

−−−

23 )75.5

375.1( RR

−−−

−−−−−−

16.2174-8.10869600

435.175.50

20218

M

M

M



28.108696

16.2174-x3 −==

875.5

2*)5.1(432 =

−−−−−=x

−−−−−−

16.2174-8.10869600

435.175.50

20218

M

M

M

© Sameh Nassar

75.5−

48

8*12*)2(20x1 =

−−−−−−=

Check:

20(-2)*284*8

34(*784*3

38(8*64*2

−=−+−

−=−+−−

−=−−−

)2

)2


Gauss-Jordan Elimination

• Is a variation of Gauss Elimination.

• The major difference is that when an unknown is eliminated in the Gauss-Jordan Elimination method, it is eliminated from all other equations rather than just the subsequent ones as in

© Sameh Nassar

all other equations rather than just the subsequent ones as in Gauss elimination.

• In addition, all rows are normalized by dividing them by their pivot elements.

• Thus, the elimination step results in an Identity matrixrather than an upper triangular matrix.

• Hence, it is not necessary to employ back substitution.


Gauss-Jordan Elimination

© Sameh Nassar

Case of n = 3


Example #3

Use Gauss-Jordan elimination to solve:

5

1

−=++

=−+ 321

42x2xx

xx2x

© Sameh Nassar

Do not employ pivoting. Check your answers by substituting

them into the original equations.

53

5

=++

−=++

321

321

xxx

42x2xx



−−

5113

4225

1112

M

M

M

53

5

1

=++

−=++

=−+

321

321

321

xxx

42x2xx

xx2x

−−

5113

4225

5.05.05.01

M

M

MR1/2

© Sameh Nassar

−−

5113

4225

5.05.05.01

M

M

M

-0.55.0*52 =−

5.25.0*31 =−−

12 *5 RR −

13 *3 RR −

5.45.0*52 =−−

.50-5.0*31 =−

5.65.0*54 −=−−

5.35.0*35 =−



−−−

−

5.35.25.00

5.65.45.00

5.05.05.01

M

M

M

−−

−

5.35.25.00

13910

5.05.05.01

M

M

M

R2/-0.5

49*)5.0(5.0 =−−−

© Sameh Nassar

−−

−

5.35.25.00

13910

5.05.05.01

M

M

M

-29*)5.0(5.2 =−−−

1013*)5.0(5.3 =−−

23 *)5.0( RR −−

−−

102-00

13910

6401

M

M

M

21 *5.0 RR −

613*5.05.0 −=−

49*)5.0(5.0 =−−−



−−

−

5100

13910

6401

M

M

M

R3/-2

−−

102-00

13910

6401

M

M

M

© Sameh Nassar

−−

−

5100

13910

6401

M

M

M

235*)9(13 −=−−−

32 )9( RR −−

−5-100

32010

14001

M

M

M

31 *4 RR −145*46 =−−−

X1 = 14

X2 = -32

X3 = -5



Check:

1=−+ (-5)(-32)14*2

© Sameh Nassar

53

5

=++

−=++

(-5)(-32)14*

4(-5)*2(-32)*214*


LU Decomposition

• LU decomposition is a class of elimination methods

• Its major appeal is that the time-consuming elimination step can be formulated so that it involves only operations on the matrix of coefficients A.

© Sameh Nassar

operations on the matrix of coefficients A.

• Therefore, it is will suited for situations where many right-hand-size vectors {B} (i.e. b) must be evaluated for a single value of A.

• LU decomposition provides an efficient means for computing the matrix inverse, i.e. A-1.


LU Decomposition

Define:

Then:

© Sameh Nassar

Similar to the 1st phase of Gauss elimination, consider:

Then:


LU Decomposition

© Sameh Nassar


LU Decomposition

General Forms (for computing the elements of L and U):

Computing the 1st column of L (for known u11) and 1st row of U (for known l11)

© Sameh Nassar

Computing the elements of the ith row of U or the jth column of L

Computing the main diagonal elements of L (for known uii) and U (for known l ii)


LU Decomposition

General Solution (for d and x):

dlbi 1

−∑−

In General:

b

© Sameh Nassar

Forward Substitution

Back Substitution1,....,2,11 −−=

−=

∑+= nnifor

u

xud

xii

n

ijjiji

inn

nn u

dx =

niforl

dlb

dii

jjiji

i ,....,3,21 =−

=∑

=

11

11 l

bd =


LU Decomposition

• Thus, the LU decomposition method is decomposing (factoring) the coefficient matrix A into a product of 2 matrices L (lower-triangular matrix) and U (upper-triangular matrix).

© Sameh Nassar

• Several methods can be used to obtain the corresponding L and U.

Methods to be discussed:

(1) Doolittle Decomposition (all l ii “i.e. L main diagonal elements” = 1).

(2) Crout’s method all uii “i.e. U main diagonal elements” = 1).


LU Decomposition

• Is related to Gauss Elimination.

(1) Doolittle Decomposition (Factorization):

131211 aaa

Case of 3 Equations (forex.), i.e. n =3:

© Sameh Nassar

333231

232221

131211

aaa

aaa

aaa

33

2322

a"00

a'a'0

22

3232

11

3131

11

2121 a'

a'l

a

al

a

al ===

1ll01l001

3231

21

Using Gauss Elimination:

A

U

L

A = LU

Coefficients of Gauss elimination process


LU Decomposition

=

11

b

b

d

d

01l

00111

* dlbd

bd

−=

=

Forward

(1) Doolittle Decomposition (Factorization): “cont.”

© Sameh Nassar

=

3

2

3

2

b

b

d

d

1ll

01l

3231

21

1233

122

**

*

dldlbd

dlbd

3132

21

−−=

−= Forward Substitution

=

3

2

1

3

2

1

d

d

d

x

x

x

a"00

a'a'0

aaa

33

2322

131211

111213

2223

33

axaxadx

a'xa'dx

a"dx

/)**(

/)*(

/

2311

322

33

−−=−=

=Back

Substitution


LU Decomposition


All l ii are equal to 1.

© Sameh Nassar

Note that pivoting is required if any uii is zero


LU Decomposition

Forward nifordlbdi

,....,3,21

=−= ∑−


bd =

© Sameh Nassar


Back Substitution1,....,2,11 −−=

−=

∑+= nnifor

u

xud

xii

n

ijjiji

inn

nn u

dx =

nifordlbdj

jijii ,....,3,21

=−= ∑=

11 bd =

Note that pivoting is required if any uii is zero


Example #4

Solve the following system of equations by LU

decomposition (Doolittle approach) without pivoting:

114 =−+ xx8x

© Sameh Nassar

762

114

=+−

=++−

=−+

321

321

321

xxx

4x5x2x

xx8x



762

114

=+−=++−

=−+

321

321

321

xxx

4x5x2x

xx8x

=7

4

11

B

−=

612

152-

1-48

A

© Sameh Nassar

1332211 === lll

25.08

2

11

2121 −=−==

u

al81111 == au

41212 == au

11313 −== au25.0

8

2

11

3131 ===

u

al



64*)25.0(5* 12212222 =−−=−= ulau

75.01*)25.0(1* 13212323 =−−−=−= ulau

33333.06

4*25.01* 12313232 −=−−=−=

u

ulal

© Sameh Nassar

5.600

0.7560

1-48

UL

−−

133333.025.0

25.0 01

001

622u

5.675.0*)33333.0(1*25.06** 233213313333 =−−−−=−−= ululau



=

−− 4

11

25.0 2

1

d

d

01

001

5.611*25.75.6*(7

75.611*)25.04

11

2

1

=−−==−−=

=

0-0.33333)d

(d

d

© Sameh Nassar

− 7133333.025.0 3d 5.611*25.75.6*(73 =−−= 0-0.33333)d

=

5.6

75.6

11

5.600 3

2

1

x

x

x

0.7560

1-48

1.0/)0.1*0.1*)1(11(

0.1/)0.1*75.75.6(

0.15./5.6

1

2

3

=−−−==−=

==

84x

60x

6x


LU Decomposition

(2) Crout’s Method:

All uii are

equal to 1.

© Sameh Nassar

Note that pivoting is required if any lii is zero


LU Decomposition

Forward dlbi

jiji

1

−∑−

(2) Crout’s Method: “cont.”

1bd =

© Sameh Nassar


Back Substitution

1,....,2,11

−−=−= ∑+=

nniforxudxn

ijjijiinn dx =

niforl

dlb

dii

jjiji

i ,....,3,21 =−

=∑

=11

11 l

bd =

Note that pivoting is required if any lii is zero


Example #5

Solve the following system of equations by LU

decomposition (Crout’s approach) without pivoting:

114 =−+ xx8x

© Sameh Nassar

762

114

=+−

=++−

=−+

321

321

321

xxx

4x5x2x

xx8x



762

114

=+−=++−

=−+

321

321

321

xxx

4x5x2x

xx8x

=7

4

11

B

−=

612

152-

1-48

A

© Sameh Nassar

1332211 === uuu

5.08

4

11

1212 ===

l

au81111 == al

22121 −== al

23131 == al125.0

8

1

11

1313 −=−==

l

au



65.0*)2(5* 12212222 =−−=−= ulal

125.06

125.0*)2(1*

22

13212323 =−−−=−=

l

ulau

25.0*21* −=−−=−= ulal

© Sameh Nassar

1000.125100.125-0.51

UL

−−

5.6222 06

008

5.6125.0*)2(125.0*26** 233213313333 =−−−−=−−= ululal

25.0*21* 12313232 −=−−=−= ulal



=

−−

7411

5.6222 2

1

ddd

06008

15.6/]125.1*)2(375.1*27[

125.16/]375.1*)24[

375.18/11

2

1

=−−−==−−=

==

d

(d

d

© Sameh Nassar

− 75.622 3d 15.6/]125.1*)2(375.1*27[3 =−−−=d

=

1125.1375.1

100 3

2

1

xxx

0.125100.125-0.51

1.01*50.01*)125.0(375.1

0.1)0.1*125.125.1(

0.1

1

2

3

=−−−==−=

=

x

0x

x

Same results as Doolittle (Ex. #4)


The Matrix Inverse

Why the Matrix Inverse is needed for solving systems of linear equations?

bA.x = .bAx -1=

Note that the condition for the existence of A-1 is that: |A| # 0.

© Sameh Nassar

=

100010001

zzzzzzzzz

aaaaaaaaa

333231

232221

131211

333231

232221

131211


-1AA I

By definition: where I is the identity matrix


Matrix Inverse using LU Decomposition

Using LU Decomposition

(where M is an intermediate Matrix)

The matrix inverse can be obtained using LU Decomposition

© Sameh Nassar

=−

333231

232221

131211

zzzzzzzzz

1A


=

333231

232221

131211

mmm

mmm

mmm

M

31

21

11

m

m

mThe vectors: , and will be obtained as the vector.

3

2

1

d

d

d

32

22

12

m

m

m

33

23

13

m

m

m


=

=

333231

232221

131211

333231

232221

131211

3231

21

III

III

III

100

010

001

mmm

mmm

mmm

1ll

01l

001

Doolittle LU Decomposition:


© Sameh Nassar

3332313332313231 III100mmm1ll

LM = I = I

Forward Substitution Compute M

11223232

122222

1212

**

*

mlmlIm

mlIm

Im

3132

21

−−=

−=

=

13233333

132323

1313

**

*

mlmlIm

mlIm

Im

3132

21

−−=

−=

=

11213131

112121

1111

**

*

mlmlIm

mlIm

Im

3132

21

−−=

−=

=


=

232221

131211

232221

131211

2322

131211

mmm

mmm

mmm

zzz

zzz

zzz

a"00

a'a'0

aaa


Doolittle LU Decomposition:

© Sameh Nassar

33323133323133 mmmzzza"00

UA-1 = M

Back Substitution Compute A-1

111213

2223

33

azazamz

a'za'mz

a"mz

/)**(

/)*(

/

21311111

312121

3131

−−=

−=

=

111213

2223

33

azazamz

a'za'mz

a"mz

/)**(

/)*(

/

22321212

322222

3232

−−=

−=

=

111213

2223

33

azazamz

a'za'mz

a"mz

/)**(

/)*(

/

23331313

332323

3333

−−=

−=

=


Example #6

114 =−+ 321 xx8x

Solve the following system of equations using the

matrix inverse approach (LU by Doolittle approach):

© Sameh Nassar

762 =+−

=++−

321

321

xxx

4x5x2x

- Use the L and U matrices obtained from Example #4.

- When obtaining A-1 , check it by verifying [A][A-1] = [I].



−=

612

152-

1-48

A

From Example #4

=7

4

11

b

© Sameh Nassar

=5.600

0.7560

1-48

U

−−=

133333.025.0

25.0 01

001

L



=

−−

100

010

001

mmm

mmm

mmm

01

001

333231

232221

131211

133333.025.0

25.0

LM = I

© Sameh Nassar

33333.00*25.1*)33333.0(0

10*)25.0(1

0

32

22

12

=−−−==−=

=

0m

-m

m

10*25.0*)33333.0(1

00*)25.00

0

33

23

13

=−−−==−−=

=

0m

(m

m

-0.166671*25.25.0*)33333.00

25.01*)25.0(0

1

31

21

11

=−−−==−−=

=

0(m

m

m

−

=

133333.016667.0

0125.0

001

M

LM = I



−=

133333.016667.0

25.0

5.600

01

001

zzz

zzz

zzz

0.7560

1-48

333231

232221

131211

UA-1 = M

© Sameh Nassar

UA-1 = M

93590.090.04487)/8*4-0.02564*1)(1(

0.0448726-0.02564)/*0.750.25(

-0.02564150.16667/6.

11

21

31

=−−−==−=

=−=

z

z

z

.=−−−==−=

==

073718-080.160256)/*40.051282*1)(0(

0.16025660.051282)/*0.75(1

0.05128250.33333/6.

12

22

32

z

z

z

0.028846)/801923-0*40.153846*1)((0

019231-060.153846)/*0.75(0

0.1538461/6.5

13

23

33

= .−−−= .=−=

==

z

z

z



−−

−=−

153846.0051282.0025641.0

019231.0160256.0044872.0

028846.0073718.0099359.01A

© Sameh Nassar

Check:

=

−−

−

−=−

100

010

001

152-

1-48

153846.0051282.0025641.0

019231.0160256.0044872.0

028846.0073718.0099359.0

612

1AA



.bAx -1=

=7

4

11

b

−−

−=−

153846.0051282.0025641.0

019231.0160256.0044872.0

028846.0073718.0099359.01A

− 11028846.0073718.0099359.0x

=

3

2

1

x

x

x

x

© Sameh Nassar

−−

−=

=7

4

11

.

153846.0051282.0025641.0

019231.0160256.0044872.0

028846.0073718.0099359.0

3

2

1

x

x

x

x

=

++−−++−

=1

1

1

7*153846.04*051282.011*025641.0

7*019231.04*160256.011*044872.0

7*028846.04*073718.011*099359.0

Same results as Ex. #4


Matrix Inverse using Gauss-Jordan Elimination

=

010

001

zzz

zzz

aaa

aaa

232221

131211

232221

131211


By definition: where I is the identity matrix

© Sameh Nassar

100zzzaaa 333231333231

-1AA I

=

0

0

1

31

21

11

z

z

z

aaa

aaa

aaa

333231

232221

131211

=

0

1

0

32

22

12

z

z

z

aaa

aaa

aaa

333231

232221

131211

=

1

0

0

33

23

13

z

z

z

aaa

aaa

aaa

333231

232221

131211

A x1 = b1 A x2 = b2 A x3 = b3

Hence, Gauss-Jordan elimination (for example) can be used to solve 3 systems of linear equations of the forms (i.e. applying it 3 times in this case):


Recall: Solving Systems of Linear Equations (SLEs)

Methods:


(2) Gauss-Jordan Elimination Direct (Elimination) Methods

© Sameh Nassar

(2) Gauss-Jordan Elimination

(3) LU Decomposition

(4) Jacobi

(4) Gauss-Seidel

Direct (Elimination) Methods

Iterative methods


Iterative Methods for Solving SLEs

• Called iterative methods, and hence, are similar in spirit to the iterative techniques of root finding (Ch. 3), specifically the fixed-point iteration technique.

• Employ initial guesses and then iterates to obtain refined

© Sameh Nassar

• Employ initial guesses and then iterates to obtain refined estimates of the solution.

• These methods are particularly suitable for large number of equations where elimination methods can be subject to round-off errors.

• However, there are certain instances where the method will not converge to the correct answer.


Gauss-Seidel Method

%100x

xx||

j,i

1j,ij,ijai

−−=ε

where:

i = 1, 2,…., n

1st

Iteration

© Sameh Nassar

i = 1, 2,…., n

j = 1, 2,…. is the

iteration number

2nd

Iteration

Note that the unknowns are updated sequentially in every iteration using the most updated value for each unknown.


Jacobi Method

%100x

xx||

j,i

1j,ij,ijai

−−=ε

where:

i = 1, 2,…., n

1st

Iteration

© Sameh Nassar

i = 1, 2,…., n

j = 1, 2,…. is the

iteration number

2nd

Iteration

Note that the unknowns are updated only once (i.e. all together) in every iteration using the values for all unknowns from the previous iteration.



∑=

>n

1jijii |a||a|

Convergence Criterion

= 3n333231

2n232221

1n131211

a......aaa

a......aaa

a......aaa

A

© Sameh Nassar

≠=

ij1j

nnn3n2n1 a......aaa

MMMMM

• Condition: the absolute value of the main diagonal element must be greater than the sum of the absolute values of the off-diagonal elements for each row.

• Such systems are called Diagonally Dominant.

• This condition is not necessary for convergence but it guarantees convergence.


Example #7

Use the Gauss-Seidel method to solve the following

system of equations to a tolerance of εs = 5%:

50x12x3x- =++

© Sameh Nassar

Start with initial values of zero for all unknowns.

40x9xx6

3xxx6

50x12x3x-

321

321

321

=++

=−−

=++



40x9xx6

3xxx6

50x12x3x-

321

321

321

=++

=−−

=++

50x12x3x-

40x9xx6

3xxx6

321

321

321

=++

=++

=−−

Diagonally Dominant

© Sameh Nassar

40x9xx6 321 =++

6

xxx 32

1

3 ++=

50x12x3x- 321 =++

0xxx000 321 ===

9

xx6x 31

2

40 −−=

12

xx3x 21

3

50 −+=



3.9490744.111110.5*350

x

4.111119

00.5*640x

0.56

003x

1

1

2

1

=−+=

=−−=

=++=

%100100%03.949074

|ε|

100%100%4.11111

04.11111|ε|

100%100%0.5

00.5|ε|

1

1a

1a

2

1

=−=

=−=

=−=1st Iteration

© Sameh Nassar

3.94907412

4.111110.5*350x

13 =−+= %100100%3.949074

03.949074|ε| 1

a3=−=

2nd Iteration

4.39611212

2.7767491.843364*350x

2.7767499

3.9490741.843364*640x

1.8433646

3.9490744.111113x

2

2

2

3

2

1

=−+=

=−−=

=++=

10.17%100%4.396112

3.949074-4.396112|ε|

48.05%100%2.776749

4.11111-2.776749|ε|

72.88%100%1.843364

0.51.843364|ε|

2a

2a

2a

3

2

1

==

==

=−=



iteration unknown value εεεεa maximum εεεεa x1 0.5 100.00%

1 x2 4.111111 100.00% x3 3.949074 100.00% 100.00%

x 1.843364 72.88% 72.88%

© Sameh Nassar

x1 1.843364 72.88% 72.88% 2 x2 2.776749 48.05% x3 4.396112 10.17%

x1 1.695477 8.72% 8.72% 3 x2 2.82567 1.73% x3 4.355063 0.94%

x1 1.696789 0.08% 4 x2 2.829356 0.13% 0.13% x3 4.355084 0.00%


Example #8

Use the Jacobi method to solve the following

system of equations:

50x12x3x- 321 =++

© Sameh Nassar

Start with initial values of zero for all unknowns and

apply 2 iterations only.

40x9xx6

3xxx6

321

321

321

=++

=−−



40x9xx6

3xxx6

50x12x3x-

321

321

321

=++

=−−

=++

50x12x3x-

40x9xx6

3xxx6

321

321

321

=++

=++

=−−

Diagonally Dominant

© Sameh Nassar

40x9xx6 321 =++

6

xxx 32

1

3 ++=

50x12x3x- 321 =++

0xxx000 321 ===

9

xx6x 31

2

40 −−=

12

xx3x 21

3

50 −+=



4.16666700*350

x

4.4444449

00*640x

0.56

003x

1

1

2

1

=−+=

=−−=

=++=

%100100%04.166667

|ε|

100%100%4.444444

04.444444|ε|

100%100%0.5

00.5|ε|

1

1a

1a

2

1

=−=

=−=

=−=1st Iteration

© Sameh Nassar

4.16666712

00*350x

13 =−+= %100100%4.166667

04.166667|ε| 1

a3=−=

2nd Iteration

3.92129612

4.4444440.5*350x

3.6481489

4.1666670.5*640x

1.9351856

4.1666674.4444443x

2

2

2

3

2

1

=−+=

=−−=

=++=

6.26%100%3.921296

4.166667-3.921296|ε|

%21.83100%3.648148

4.444444-3.648148|ε|

%74.16100%1.935185

0.51.935185|ε|

2a

2a

2a

3

2

1

==

==

=−=


True Errors and Residuals

• For a system of linear equations Ax = b with true solution vector xtrue solution and numerical “approximate” solution vector xnumerical solution

, the following error vectors are defined:

True Error Vector e: e = xtrue solution - xnumerical solution

© Sameh Nassar

Residual Vector r (or simply Residuals):

r = Axtrue solution - Axnumerical solution = b - Axnumerical solution

• Similar to what discussed before for single variables (Lecture #2), rcan be estimated while e is usually unknown.

• However, r does not really indicate how small the errors in the solution are and hence a small residual vector r does not guarantee a small true error vector e.


True Error and Residual Vectors

• Assume we have:

© Sameh Nassar

• Assume the following 2 numerical solution vectors were obtained:

• Let’s now obtain the true error vectors and the corresponding residual vectors of the 2 cases (numerical solutions) and then compare the results.


True Error and Residual Vectors

e is sensitive to the errors in the

numerical solution

© Sameh Nassar

r is not sensitive to the errors in the

numerical solution

• Note that in case (1), the residuals (r1) have the same order of magnitude as the true errors (e1), while in case (2) they are not.

• This is a typical example of an ill-conditioned coefficient matrix A.


• Based on the discussions in previous slides, a more accurate estimate of the error in a numerical solution (of SLEs) should be obtained.

• In order to relate the magnitude of the residuals to the magnitude of the true errors, and also to determine the bounds for these

Norms

© Sameh Nassar

of the true errors, and also to determine the bounds for these errors, we are going to use quantities that measure the magnitude of matrices and vectors.

• These quantities are called norms.

• A norm is a non-negative real number assigned to a matrix or a vector.

• The norm of a vector νννν is denoted by ||ν||||ν||||ν||||ν||.

• The norm of a matrix A is denoted by ||||||||[A]|||||||| (or simply ||||||||A|||||||| ).


Matrix Norms

Categories (Types):

© Sameh Nassar

Where νννν is an eigenvector of the matrix A corresponding to an eigenvalue λ(discussed later!)

Euclidean

2-norm ||||||||[A] |||||||| =

(m x n matrix)

Additional Property:


Error Bounds of SLEs Numerical Solutions

• For a system of linear equations Ax = b with true solution vector xT

and numerical “approximate” solution vector xN , the following terms (using norms) are defined:

Relative Error Re: Re = , e = xT - xN (true error)

© Sameh Nassar

Relative Residual Rr: Rr = , r = b - AxN (residual)

• Thus, a major objective here is to use Rr (that can be estimated from the numerical solution) to obtain “or estimate” an upper and lower bounds of Re (recall that Re itself is unknown).


Error Bounds of SLEs Numerical Solutions

© Sameh Nassar

Lower Bound Upper Bound

• The term is called the Condition Number of A.

Lower Bound Upper Bound


Condition Number of a Matrix

Properties:

(I is the identity matrix)

• If Cond(A) is large (i.e. >> 1), a small Rr will not guarantee a small

© Sameh Nassar

• If Cond(A) is large (i.e. >> 1), a small Rr will not guarantee a small Re (i.e. small changes in b or the elements of A may result in large

changes in x).

• In this case, the SLE (and hence A) is ill-conditioned.

• If Cond(A) is small (i.e. ~ 1), a small Rr will indicate a small Re (i.e.

small changes in b or the elements of A will result in small changes in x).

• In this case, the SLE (and hence A) is well-conditioned.


Eigenvalue Problems

• Known also as characteristic-value problems.

• An eigenvalue is a characteristic value of a square matrix (n x n).

Background #1:

If A is an n x n matrix, usually there is no obvious geometric

© Sameh Nassar

• If A is an n x n matrix, usually there is no obvious geometric relationship between a vector x and its image Ax (i.e. when A is multiplied by x).

• However, there are some nonzero vectors “x” that A is mapping into scalar multiples of themselves.

• These types of vectors arise in several engineering applications that involve vibrations, electrical systems, quantum mechanics, mechanical stress and elasticity.


Eigenvalue Problems

© Sameh Nassar

No Geometric Relation between x and Ax

Ax is a scalar multiple of x

“x is an eigenvector”


Eigenvalue Problems

• In the latter case, Ax = λλλλx, where λλλλ is a scalar.

• The scalar λλλλ is called an eigenvalue (or characteristic value) of Aand x is said to be an eigenvector corresponding to λλλλ.

Background #2:

(a) The system of equations: A.x = b

© Sameh Nassar

(a) The system of equations: A.x = bis called non-homogeneous

(b) The system of equations: A.x = 0is called homogeneous

Why?

• For case (a), a unique solution of x can be obtained if the equations are linearly independent (i.e. |A| ≠ 0).

• For case (b), nontrivial solutions (any solutions rather than x = 0) can be obtained, however, they are generally not unique (i.e. several solutions of x are available).


Eigenvalues λλλλ & Eigenvectors x

• The non-unique solution for x is obtained since several available relationships can be established between the x-vector elements that can be satisfied by various combinations of the x-values.

• To obtain the relations of the x-values, the eigenvalues λλλλ of A are obtained first, where the general form of the equations will be:

© Sameh Nassar

A.x = λλλλ.x A.x - λλλλ.I.x = 0 [A - λλλλ.I].x = 0


Eigenvalues λλλλ & Eigenvectors x

• Hence, the unknown parameter λλλλ (eigenvalue) should be obtained and then the relations for x-values can be evaluated.

Condition to obtain λλλλ for nontrivial solutions of x:

|A - λλλλ.I| = 0

© Sameh Nassar

• The values of λλλλ can be obtained by expanding the determinant |A - λλλλ.I| and equalize to zero, which will yield a polynomial in the unknown λλλλ.

• The values of λλλλ are the roots of the polynomial (any method of solving SLEs can be used).

• Then, the eigenvector x can be evaluated for each value of λλλλ.

• For any constant c, cx is also an eigenvector of A (therefore, the eigenvectors x are normalized to be unit vectors, i.e. ||||||||x||||||||2= 1.


Example #9

Find the eigenvalues of the given matrix:

−−

=23

A

© Sameh Nassar

Then, evaluate the corresponding eigenvectors.

=01

A



−−=

01

23A

Eigenvalues:

|A - λλλλ.I| = 0 001

23=

−−−−

λλ

0)1*2(*)3( =−−−−− λλ 2 =++ λλ

© Sameh Nassar

0)1*2(*)3( =−−−−− λλ 0232 =++ λλ

1*2

2*1*433 2 −±−=λ2

13±−=λ

11 −=λ 22 −=λ



Eigenvectors:

==

−

−−−00

001

23

2

1

xx

λλ

[A - λλλλ.I].x = 0

λ = λ1 = -1:

© Sameh Nassar

0)1(01

2)1(3

2

1 =

−−−−−−

x

x0

1122

2

1 =

−−xx

12 xx −=

Use ||||||||x||||||||2= 1 122

21 =+ xx 1)( 2

121 =−+ xx

2

11 =x

2

12 −=x

−=

2

12

1

1λx



Eigenvectors:

02)2(3 1 =

−−−− x

021 1 =

−− x 1

xx −=

λ = λ2 = -2:

© Sameh Nassar

0)2(01 2

1 =

−− x

021

21

2

1 =

−−x

x12 2

1xx −=

Use ||||||||x||||||||2= 1 122

21 =+ xx 1)

2

1( 2

121 =−+ xx

5

21 =x

5

12 −=x

−=

5

15

2

2λx


• Sections:

4.1

4.2

4.3

Textbook Readings

© Sameh Nassar

4.3

4.4

4.5

4.6

4.7

4.10

4.11

4.12 (excluding 4.12.1 – 14.12.4)

engg 407_p12_l20_lecture_03

Documents