numerical methods notes

1

Solution of Linear System of Equations

Gaussian Elimination

Objectives:

Introduction


Pitfalls in Gauss Elimination Method

Pivoting

Scaling

Introduction

We’ll see how to solve a set of n linear equations with n

unknowns.

If n ≤ 3, the small set of linear equations can be solved

analytically.

Simultaneous linear equations arise in a variety of

engineering problems context i.e. trusses, reactors, electric

circuits,…

Broadly there are two types of methods to solve the linear

equations: Direct methods and Iterative methods.

2


Solving a set of n linear equations:

11212111 ... bxaxaxa nn

22222121 ... bxaxaxa nn

.

.

nnnnnn bxaxaxa ...2211

Solution consists of two phases:

1. Forward elimination of unknowns

2. Back substitution

1. Forward elimination of the unknowns

Eliminate the first unknown, 1x , from the second

through the nth equations. Multiply equation (1) by

1121 / aa

1

11

211

11

21212

11

21121 ... b

a

axa

a

axa

a

axa nn

Subtract equation (2) from second equation to get

1

11

2121

11

212212

11

2122 )(...)( b

a

abxa

a

aaxa

a

aa nnn

or

3

22222 ... bxaxa nn

where the prime indicates the new values for the elements.

Similarly equation (1) can be multiplied by 1131 / aa and

the result subtracted from the third equation.

Repeating the procedure for the remaining equations

give the following modified forms:

11313212111 ... bxaxaxaxa nn

22323222 ... bxaxaxa nn

33333232 ... bxaxaxa nn

. .

. .

nnnnnn bxaxaxa ...3322

The second unknown 2x is eliminated by multiplying

through the next equation by 2232 / aa and subtracting

the result from the following equation.

The procedure is continued using the remaining pivot

equations. At the end of the elimination stage, the

original system of equations is transformed to an

upper triangular system:

11313212111 ... bxaxaxaxa nn

22323222 ... bxaxaxa nn

33333 ... bxaxa nn

4

. .

. .

)1()1( n

nn

n

nn bxa

2. Back Substitution

The last unknown nx is now given by:

)1(

)1(

n

nn

n

nn

a

bx

This value of nx is then back-substituted into equation

)1( n to find 1nx . This procedure can be repeated to

evaluate the values of the remaining x’s.

5

Example 1

Use Gauss elimination to solve:

3 1x – 0.1 2x – 0.2 3x = 7.85 (a)

0.1 1x + 7 2x – 0.3 3x = -19.3 (b)

0.3 1x – 0.2 2x + 10 3x = 71.4 (c)

Use 6 significant figures in your computation.

6

Solution of Example 1

1. Forward elimination

Multiply 1st equation (a) by 0.1/3 and subtract from 2nd

equation (b) to eliminate 1x from the latter equation:

7.00333 2x – 0.293333 3x = -19.5617

Then multiply 1st equation (a) by 0.3/3 and subtract from

3rd equation (c) to eliminate 1x from the latter equation:

-0.190000 2x + 10.0200 3x = 70.6150

The system of equations is now:

3 1x – 0.1 2x – 0.2 3x = 7.85 (a’)

7.00333 2x – 0.293333 3x = -19.5617 (b’)

-0.190000 2x + 10.0200 3x = 70.6150 (c’)

Now let’s eliminate 2x from latest 3rd equation (c’) by

multiplying equation (b’) by -0.190000/7.00333 and

subtract the result from equation (c’):

10.0200 3x = 70.0843

The system of equations is now reduced to an upper

triangular form:

7

3 1x – 0.1 2x – 0.2 3x = 7.85 (a’’)

7.00333 2x – 0.293333 3x = -19.5617 (b’’)

10.0200 3x = 70.0843 (c’’)

2. Back Substitution

Solve last equation (c’’) to find 3x :

3x = 70.0843/10.0200 = 7.00003

Substitute 3x in equation (b’’) and solve for 2x :

7.00333 2x – 0.293333 (7.00003) = -19.5617

2x = -2.50000

Finally substitute values of 2x and 3x in equation (a’’) to

find 1x :

3 1x – 0.1(-2.5000) – 0.2(7.00003) = 7.85

1x = 3.00000

3. Check your results

Substitute the values of 1x , 2x and 3x in the original system

of equations.

3 (3.00000) – 0.1 (-2.50000) – 0.2 (7.00003) = 7.84999

0.1 (3.00000) + 7 (-2.50000) – 0.3 (7.00003) = -19.30000

8

0.3 (3.00000) – 0.2 (-2.50000) + 10 (7.00003) = 71.40030

Some Pitfalls of Gauss Elimination Method

1. Division by zero

Caused by coefficient value equals zero or very close to

zero

This can be solved by using pivoting technique.

2. Round-off errors

This may be due to some of the following:

Large number of equations to be solved due to the fact

that every result is dependent on previous results.

Errors in early steps will tend to propagate.

Round-off errors may be solved by:

Using more significant figures.

Using fractions instead of decimals.

Always substitute your answers back into the original

system of equations to check for occurrence of

substantial errors.

9

Pivoting

This involves the following steps:

1. Determine the largest coefficient available in the

column below the pivot element.

1. Switch the rows so that the largest element is the pivot

element. This is known as partial pivoting.

1. If columns as well as rows are searched for the largest

element and then switched, the process is called

complete pivoting.

Example 2

Solve the following system of equations

0.0003 1x + 3.0000 2x = 2.0001 (1)

1.0000 1x + 1.0000 2x = 1.0000 (2)

1. Using naïve Gauss elimination

2. Using partial pivoting

10

Solution of example 2

1. Firstly, let’s solve these equations the same way we did in

example 1:

Forward elimination:

− Multiply equation (1) by 1/0.0003 to get

1x + 10,000 2x = 6667

− Subtract the resulting equation from equation (2) to get

-9999 2x = -6666 2x = 2/3

Back substitution:

0.0003 1x + 3.0000 2x = 2.0001

1x = 2.0001 – 3(2/3)

0.0003

11

Now let’s check the effects of number of significant digits

on the result of 1x due to 2x :

Significant

figures 2x 1x ε 1x (%)

3 0.667 -3.33 1099

4 0.6667 0.0000 100

5 0.66667 0.30000 10

6 0.666667 0.330000 1

7 0.6666667 0.3330000 0.1

2. Secondly, let’s solve these equations by applying partial

pivoting:

1.0000 1x + 1.0000 2x = 1.0000 (1)

0.0003 1x + 3.0000 2x = 2.0001 (2)

Forward elimination:

− Multiply equation (1) by 0.0003 to get

0.0003 1x + 0.0003 2x = 0.0003

− Subtract the resulting equation from equation (2) to get

2.9997 2x = 1.9998 2x = 2/3

Back substitution:

12

1x + 2x = 1 1x = 1 – (2/3)

1

This case is much less sensitive to the number of significant

figures on the result of 1x due to 2x :

Significant

figures 2x 1x ε 1x (%)

3 0.667 0.333 0.1

4 0.6667 0.3333 0.01

5 0.66667 0.33333 0.001

6 0.666667 0.333333 0.0001

7 0.6666667 0.3333333 0.00001

13

Scaling

Scaling is necessary when different units are used in the

same system of equations.

Scaling is used to:

1. Standardize the size of the coefficients in the system

of equations.

2. Minimize the round-off errors caused by having much

larger coefficients than others.

Example 3

Use three significant figures to solve

2 1x + 100000 2x = 10000 (1)

1x + 2x = 2. (2)

1. Naïve Gauss elimination

2. Gauss elimination with scaling and pivoting

3. Gauss elimination without scaling and with pivoting

Solution of example 3

1. Gauss elimination:

− Forward elimination

2 1x + 100000 2x = 10000 (1)

14

- 50000 2x = -5000 (2)

− Back substitution

2x = 0.10 and 1x = 0.00

2. Gauss elimination with scaling and pivoting:

− Scaling

0.00002 1x + 2x = 0.1 (1)

1x + 2x = 2 (2)

− Pivoting

1x + 2x = 2 (1)

0.00002 1x + 2x = 0.1 (2)


1x + 2x = 2

2x = 0.1


1x = 1.90 and 2x = 0.10

15

3. Gauss elimination without scaling and with pivoting of the

original equations:

− Pivoting

1x + 2x = 2.

2 1x + 100000 2x = 10000 (1)


1x + 2x = 2

100000 2x = 10000


1x = 1.90 and 2x = 0.10

16

Factorization Methods

Objectives:

Introduction

LU Decomposition

Computational complexity

The Matrix Inverse

Extending the Gaussian Elimination Process

Introduction

Provides an efficient way to compute matrix inverse by

separating the time consuming elimination of the Matrix

[A] from manipulations of the right-hand side {B}.

Gauss elimination, in which the forward elimination

comprises the bulk of the computational effort, can be

implemented as an LU decomposition.

LU Decomposition

The matrix [A] for the linear system [A]{X}={B} is

factorized into the product of two matrices [L] and [U] (L-

lower triangular matrix and U- upper triangular matrix)

[L][U]=[A]

[L][U]{X}={B}

Similar to first phase of Gauss elimination, consider

17

[U]{X}={D}

[L]{D}={B}

The solution can be obtained by

1. First solve [L]{D}={B} to generate an intermediate

vector {D} by forward substitution

2. Then, solve [U]{X}={D} to get {X} by back

substitution.

18

In matrix form, this is written as

3

2

1

3

2

1

333231

232221

131211

b

b

b

x

x

x

aaa

aaa

aaa

How to obtain the triangular factorization?

333231

232221

131211

100

010

001

aaa

aaa

aaa

A

Use Gauss elimination and store the multipliers mij as the

subdiagonal entries in [L]

The multipliers are

11

2121

a

am ,

11

3131

a

am ,

22

3232

a

am

The triangular factorization of matrix [A]

A = [L] [U]

19

33

2322

131211

3231

21

00

0

1

01

001

a

aa

aaa

mm

mA

Example :

Use LU decomposition to solve:

3x1 – 0.1x2 – 0.2x3 = 7.85

0.1x1 + 7x2 – 0.3x3 = -19.3

0.3x1 – 0.2x2 + 10x3 = 71.4

use 6 significant figures in your computation.

Example 1 - Solution

In matrix form

The multipliers are

0333333.03

1.21 m , 100000.0

3

3.31 m

0271300.000333.7

193.032

m

4.71

3.19

85.7

102.03.0

3.071.0

2.01.03

3

2

1

x

x

x

20

The LU decomposition is

The solution can be obtained by

1. First solve [L]{D}={B} for {D} by forward

substitution

Then, solve [U]{X}={D} to get {X} by back

substitution.

0120.1000

293333.000333.70

2.01.03

10271300.0100000.0

010333333.0

001

A

4.71

3.19

85.7

10271300.0100000.0

010333333.0

001

3

2

1

d

d

d

0843.70)5617.19(02713.0)85.7(1.04.71

5617.19)85.7(0333333.03.19

85.7

3

2

1

d

d

d

0843.70

5617.19

85.7

0120.1000

293333.000333.70

2.01.03

3

2

1

x

x

x

21

33/))00003.7(2.0)5.2(1.085.7(

5.200333.7/))00003.7(293333.05617.19(

00003.70120.10/0843.70

1

2

3

x

x

x

22

Computational Complexity

The triangular factorization portion of [A]=[L][U] requires

(N3-N)/3 multiplications and divisions

(2N3-3N2+N)/6 subtractions

Finding the solution to [L][U]{X}={B} requires

N2 multiplications and divisions

N2-N subtractions

The bulk of the calculation lies in the triangularization

portion.

LU decomposition is usually chosen over Gauss

elimination when the linear system is to be solved many

times, with the same [A] but with different {B}.

Saves computing time by separating time-consuming

elimination step from the manipulations of the right hand

side.

Provides efficient means to compute the matrix inverse

which provides a means to test whether systems are ill-

conditioned

The Matrix Inverse

Find matrix 1A , the inverse of [A], for which

[A] 1A = 1

A [A]=[I]

The inverse can be computed in a column-by-column

fashion by generating solutions with unit vectors {B}

constants.

23

The solution of [L][U]{X}={B} with

0

0

1

will

be the first column of 1A


0

1

0

will

be the second column of 1A


1

0

0

will

be the third column of 1A .

Example 2:

Use LU decomposition to determine the matrix inverse for

the following system and use it to find the solution:

3x1 – 0.1x2 – 0.2x3 = 7.85

0.1x1 + 7x2 – 0.3x3 = -19.3

0.3x1 – 0.2x2 + 10x3 = 71.4

use 6 significant figures in your computation.

24

Example 2- Solution

In matrix form

The triangular factorization of [A]

The first column of 1A

The second column of 1A

102.03.0

3.071.0

2.01.03

A

10271300.0100000.0

010333333.0

001

L

0120.1000

293333.000333.70

2.01.03

U

1009.0

03333.0

1

0

0

1

10271300.0100000.0

010333333.0

001

3

2

1

D

d

d

d

01008.0

00518.0

33249.0

1009.0

03333.0

1

0120.1000

293333.000333.70

2.01.03

3

2

1

X

x

x

x

25

The third column of 1A

The matrix inverse 1A is:

02713.0

1

0

0

1

0

10271300.0100000.0

010333333.0

001

3

2

1

D

d

d

d

00271.0

142903.0

004944.0

02713.0

1

0

0120.1000

293333.000333.70

2.01.03

3

2

1

X

x

x

x

1

0

0

1

0

0

10271300.0100000.0

010333333.0

001

3

2

1

D

d

d

d

09988.0

004183.0

006798.0

1

0

0

0120.1000

293333.000333.70

2.01.03

3

2

1

X

x

x

x

09988.000271.001008.0

004183.0142903.000518.0

006798.0004944.033249.01A

26

Check your result by verifying that [A] 1A =[I]

The final solution is

7

50002.2

3

4.71

3.19

85.7

09988.000271.001008.0

004183.0142903.000518.0

006798.0004944.033249.01

BAX

27

Extending the Gaussian Elimination Process

If pivoting is required to solve [A]{X}={B}, then there

exists a permutation matrix [P] so that:

[P][A ]=[L][U]

The solution {X} is found in four steps:

1. Construct the matrices [L], [U] and [P].

2. Compute the column vector [P]{B}.

3. Solve [L]{D}=[P]{B} for {D} using forward

substitution.

4. Solve [U]{X}={D} for {X} using back substitution.

Example 3

Use LU decomposition with permutation to solve the

following system of equations

0.0003 x1 + 3.0000 x2 = 2.0001

1.0000 x1 + 1.0000 x2 = 1.0000


In matrix form [A ]{X}={B}

1

0001.2

11

30003.0

2

1

x

x

28

We saw previously that pivoting is required to solve this

system of equations, hence [P][A ]=[L][U]

The solution {X} is found in four steps:

1. Construct the matrices [L], [U] and [P].

1. Compute the column vector [P]{B}.

2. Solve [L]{D}=[P]{B} for {D} using forward

substitution.

3. Solve [U]{X}={D} for {X} using back substitution.

01

10P

10003.0

01L

9997.20

11U

0001.2

1

1

0001.2

01

10

9998.1

1

0001.2

1

10003.0

01

2

1D

d

d

66667.0

33333.0

9998.1

1

9997.20

11

2

1X

x

x

29

Iterative Methods

Objectives:

Introduction

Jacobi Method

Gauss-Seidel Method

Introduction

To solve the linear system Ax = b we may use either:

Direct Methods

- Gaussian elimination

- PLU decomposition

Iterative Methods

- Jacobi Method

- Gauss-Seidel Method

Iterative Methods

Suppose we solve Ax = b for a given matrix A by finding

the PLU decomposition

If we change the vector b, we may continue to use the PLU

If we change A, we now have to re-compute the PLU

decomposition: expensive

Instead, suppose we have solved the system Ax = b

for a given matrix A

30

Suppose we change A slightly, e.g., modify a single resistor

in a circuit

If we call that new matrix Amod, is it possible to use the

solution to Ax = b to solve Amodx = b?

They provide an alternative to the elimination method.

Let Ax = b be the set of equations to be solved.

The system Ax = b is reshaped by solving the first equation

for 1x , the second equation for 2x , and the third for 3x ,

…and nth equation for nx .

For ease of computation, let’s assume we have a 3x3

system of equations to solve.

If the diagonal elements are all non-zero then:

3333232131

2323222121

1313212111

bxaxaxa

bxaxaxa

bxaxaxa

33

23213133

22

32312122

11

31321211

a

xaxabx

a

xaxabx

a

xaxabx

31

Jacobi Iteration Method

1. Assume all the x’s are zero

2. Substitute the zeros into the three equations to get:

11

11

a

bx ,

22

22

a

bx ,

33

33

a

bx

3. Repeat the procedure until the error criterion is satisfied:

33

23213131

3

22

32312121

2

11

31321211

1

a

xaxabx

a

xaxabx

a

xaxabx

iii

iii

iii

si

j

i

j

i

j

jax

xx

%100

1

,

32

Gauss-Seidel Method

It is the most commonly used iterative method.

Gauss-Seidel Procedure

1. Assume all the x’s are zero

2. Substitute the zeros into the first equation i.e. equation (1)

to give:

11

11

a

bx

3. Substitute the new value of 1x and 3x = 0 into equation (2)

to compute 2x

4. Substitute the value of 1x and the new value of 2x in

equation (3) to estimate 3x

5. Return to equation (1) and repeat the entire procedure until

the error criterion is satisfied:

33

1

232

1

13131

3

22

323

1

12121

2

11

31321211

1

a

xaxabx

a

xaxabx

a

xaxabx

iii

iii

iii

si

j

i

j

i

j

jax

xx

%100

1

,

33

Example 1:

Use Gauss-Seidel method to solve the following set of linear

equations:

3 1x – 0.1 2x – 0.2 3x = 7.85 (1)

0.1 1x + 7 2x – 0.3 3x = -19.3 (2)

0.3 1x – 0.2 2x + 10 3x = 71.4 (3)


First we have:

10

2.03.04.71

7

3.01.03.19

3

2.01.085.7

213

312

321

xxx

xxx

xxx

34

1st iteration

Assume that 2x = 0 and 3x = 0, we obtain

Substitute 1x = 2.616667 and 3x = 0 into equation (2)

Substitute 1x = 2.616667 and 2x = -2.794524 into

equation (3)

This completes the first iteration

2nd iteration

616667.23

85.71 x

794524.27

0)616667.2(1.03.192

x

005610.710

)794524.2(2.0)616667.2(3.04.713

x

990557.23

)005610.7(2.0)79454.2(1.085.71

x

499625.27

)005610.7(3.0)990557.2(1.03.192

x

35

Error estimate

For 1x

For 2x

For 3x

000291.710

)499625.2(2.0)990557.2(3.04.713

x

%5.12%100 990557.2

6166667.2990557.21,

a

%8.11%100 499625.2

)794524.2(499625.22,

a

%07.0%100 000291.7

005610.7000291.73,

a

n

ijj

ijii aa1

36

Convergence

Gauss-Seidel is similar in spirit to the simple fixed-point

iteration.

Gauss-Seidel will converge if for every equation of the

system, we have:

Such system is said to be diagonally dominant.

This criterion is sufficient but not necessary for

convergence.

Relaxation

Designed to Enhance convergence.

After each new value of x is computed, that value is

modified using:

Where 20 is a weighting factor.

The choice of is problem-specific and is often

determined empirically.

old

i

new

i

new

i xxx 1

37

Gauss-Seidel/Jacobi Iteration Methods

Gauss-Seidel/Jacobi Iteration Methods

Gauss-Seidel iteration converges more rapidly than the

Jacobi iteration does; since, it uses the latest updates.

But there are some cases that Jacobi iteration does

converge but Gauss-Seidel does not.

38

Eigenvalues and Eigenvectors

Objectives

Introduction

Mathematical background

Physical background

Polynomial Method

Power Method

Introduction

Eigenvalue problems are a special class of problems that

are common in engineering contexts involving vibrations

and elasticity.

Many systems of ODEs can be reduced to eigenvalue

problems.

Mathematical Background

So far we learned to solve [A]{x}={b}

Such systems are called nonhomogeneous because of the

presence of {b}.

If det[A] ≠ 0 unique solution of {x}

39

Homogeneous systems has the general form [A]{x}=0

Nontrivial solutions of such systems are possible but

generally they are not unique.

Eigenvalue problems are of the general form: 0...)( 1212111 nnxaxaxa

0...)( 2222121 nnxaxaxa

. . . .

. . . .

. . . .

0)(...2211 nnnnn xaxaxa

is the unknown parameter called the eigenvalue or

characteristic value.

A solution { 1x , 2x , …, nx } for such a system is referred to

as an eigenvector.

The set of equations may also be expressed as: 0 xIA

The determinant of the matrix [[A]- [I]] must equal to

zero for nontrivial solutions to be possible.

40

Expanding the determinant yields a polynomial in .

The roots of this polynomial are the solutions to the

eigenvalues.

Physical Background

The following mass-spring system is a simple illustration of

how eigenvalues occur in engineering context.

Force balance for each mass is developed using Newton’s

second law:

)( 1212

1

2

1 xxkkxdt

xdm

41

2122

2

2

2 )( kxxxkdt

xdm

From vibration theory, the solutions to these equations )sin( tXx ii

where

Xi is the amplitude of the vibration of im

ω is the frequency of the vibration given bypT

2

Tp is the period.

This system of equations can be converted to an eigenvalue

problem of vibrations.

0)2

( 2

1

1

2

1

Xm

kX

m

k

0)2

( 2

2

2

1

2

Xm

kX

m

k

42

The Polynomial Method

When dealing with complicated systems or systems with

heterogeneous properties, analytical solutions are often

difficult or impossible to obtain.

Numerical solutions to such equations may be the only

practical alternatives.

These equations can be solved by substituting a central

finite-divided difference approximation for the derivatives.

Writing this equation for a series of nodes yields a

homogeneous system of equations.

The Polynomial Method Procedure

Convert the system to an eigenvalue problem

[[A]- λ[I]] {x}= 0

Expand determinant det[[A]- λ[I]] = 0. This will yield a

polynomial in λ.

Solve for λ

For each value of λ, establish the relationship between the

unknowns x’s called an eigenvector (note no unique

solution).

43

Example 1

Use the polynomial method to evaluate the eigenvalues and

eigenvectors of the spring-mass example for the case where m1

= m2 = 40 kg and k = 200 N/m.



05)10( 21

2 XX

0)10(5 2

2

1 XX

Expand determinant det[[A]- λ[I]] = 0.

07520)( 222

Solve for 2

2 = 15 and 2 = 5 2s

44

The frequencies for the vibrations of the masses are ω =

3.873 and ω = 2.236 1s

The periods for the vibrations

Tp = 1.62 s and Tp = 2.81 s

For each value of 2 , plug into matrix equation to solve for

eigenvectors X’s.

- For the first mode ( 2 = 15)

05)1510( 21 XX

0)1510(5 21 XX

1X = - 2X

- Similarly, for the second mode ( 2 = 5) 1X = 2X

What does this mean physically?

Valuable information about:

Period

Amplitude

1st mode 1X = - 2X

2nd mode 1X = 2X

The Power Method

45

An iterative approach that can be employed to determine

the largest eigenvalue.

With slight modification, it can also be used to determine

the smallest eigenvalue.

To determine the largest eigenvalue the system must be

expressed in the form: }{}{ XXA

46

The Power Method Procedure

Rearrange equations so that we have: }{}{ XXA

Plug in an initial guess for LHS X . Assume all the X’s on

the LHS of the equations are equal to 1.

Solve for RHS.

Pull scalar out of RHS so maximum value in vector is equal

to 1.

Plug eigenvector back into LHS and repeat until eigenvalue

converges with εa < εs

Example 2

Employ the power method to determine the highest eigenvalue

and its associated eigenvector of a three mass-four spring system

for the case where m1 = m2 = m3 = 1 kg and k1 = k2 = k3 = k4 = k

= 20 N/m.

47



0)2

( 2

1

1

2

1

Xm

kX

m

k

02

3

2

2

2

2

1

2

X

m

kX

m

kX

m

k

02

3

2

3

2

3

X

m

kX

m

k

Substitute the values of m’s and k’s and express the system

in the matrix form

}{}{ XXA

3

2

1

3

2

1

40200

204020

02040

X

X

X

X

X

X

48

Eventually converges

Eigenvalue = 68.28427

Eigenvector =

707107.0

1

707107.0

The Power Method - Extra

Example 2 found the largest eigenvalue and its

corresponding eigenvector.

49

To find the smallest eigenvalue, apply the power method to

the matrix inverse of A. The power method will converge

to the largest value of 1/λ (which means the smallest λ).

After finding the largest eigenvalue, it is possible to

determine the next highest by replacing the original matrix

by one that includes only the remaining eigenvalues. This

process is called deflation.

Assignment #5

27.11, 28.25, 28.27

50

Curve Fitting and Interpolation

Polynomial Interpolation

Objectives

Introduction

Newton’s Divided Difference Method

i. Linear interpolation

ii. Quadratic interpolation

iii. General Form of Newton’s Interpolation

Lagrange Interpolation

Introduction

Techniques to fit curves to discrete values of data to obtain

intermediate estimates.

- Regression (imprecise data)

- Interpolation (precise data)

Curve fitting is widely used in engineering

- Trend analysis: extrapolation and interpolation

- Hypothesis testing: compare a mathematical

model with measured data.

51

Linear Interpolation

Using similar triangles,

01

01

0

01 )()()()(

xx

xfxf

xx

xfxf

And rearranging, we get

)()()(

)()( 0

01

0101 xx

xx

xfxfxfxf

f1(x) is a first order interpolating polynomial.

52

f1(x) represents the slope of the line connecting the points.

01

01 )()(

xx

xfxf

The smaller the interval between data points, the better the

approximation.

Example 1:

Estimate the natural logarithm of 2 using linear interpolation:

1. Interpolate between ln1 and ln6

2. Use interval ln1 to ln4

Example1 - Solution

Linear interpolation

)()()(

)()( 0

01

0101 xx

xx

xfxfxfxf

1. Using ln1 and ln6

3583519.0)12(16

1ln6ln1ln)2(1

f εt=48.3%

2. Using ln1 and ln4

4620981.0)12(14

1ln4ln1ln)2(1

f εt=33.3%

53

Quadratic Interpolation

This is a mean of improving an estimate by introducing a

curvature into the line connecting the points.

Using three data points, a second-order polynomial or

quadratic polynomial or parabola is used to carry out the

estimate:

f2(x) = b0 + b1(x-x0) + b2(x-x0)(x-x1)

= b0 + b1x - b1x0 + b2x2 - b2x0x - b2xx1 + b2x0x1

54

= a0 + a1x + a2x2

Where

a0 = b0 - b1x0 + b2x0x1

a1 = b1 - b2x0 - b2x1

a2 = b2

f (x) = b0 + b1(x-x0) + b2(x-x0)(x-x1)

The values of coefficients b0 , b1 and b2 are computed as

follow:

Evaluate f at x = x0 )( 00 xfb

Evaluate f at x = x1 01

01

1

)()(

xx

xfxfb

Evaluate f at x = x2

02

01

01

12

12

2

)()()()(

xx

xx

xfxf

xx

xfxf

b

The first two terms in equation (2) are equivalent to linear

interpolation from x0 to x1.

b1 represents the slope of the line connecting points x0 and

x1.

55

b2(x-x0)(x-x1) introduces the second-order curvature into the

formula.

Example 2:

Fit a second-order polynomial to the three points used to

evaluate the natural logarithm of 2

i.e.

x0 = 1 f (x0) = 0

x1 = 4 f (x1) = 1.386294

x2 = 6 f (x2) = 1.791759


First, let’s compute the coefficients b0 , b1 and b2: 01ln)( 00 xfb

4620981.03

0386294.1

14

1ln4ln)()(

01

011

xx

xfxfb

02

01

01

12

12

2

)()()()(

xx

xx

xfxf

xx

xfxf

b

56

0518731.016

14

1ln4ln

46

4ln6ln

The quadratic polynomial is then:

f2(x) = 0 + 0.4620981(x-1) – 0.0518731(x-1)(x-4)

Let’s now evaluate f2(x) at x=2

f2(2) = 0.5658444

and the relative error εt = 18.4%

57

General Form of Newton’s Interpolation

The analysis used in linear and quadratic interpolation can

be generalized to fit an (n-1)th order polynomial to n data

points.

The data points are used to evaluate the coefficients :

The bracketed function evaluations are finite divided

differences

))...()((...)()( 21010101 nnn xxxxxxbxxbbxf

],,...,,[

.

.

],,[

],[

)(

01211

0122

011

00

xxxxfb

xxxfb

xxfb

xfb

nnn

01

0321210121

],...,,[],...,,[],,...,,[

],[],[],,[

)()(],[

xx

xxxfxxxfxxxxf

xx

xxfxxfxxxf

xx

xfxfxxf

n

nnnnnn

ki

kjji

kji

ji

ji

ji

58

The general form of Newton’s interpolating polynomial:

Example 3

Fit a third-order Newton’s interpolating polynomial to the four

points used to evaluate the natural logarithm of 2

i.e.

x0 = 1 f (x0) = 0

x1 = 4 f (x1) = 1.386294

x2 = 6 f (x2) = 1.791759

x3 = 5 f (x3) = 1.609438

0121210

0121001001

,,...,,))...()((...

,,))((,)()(

xxxxfxxxxxx

xxxfxxxxxxfxxxfxf

nnn

n

59


The third-order polynomial is

The values of the coefficients of the polynomial are:

Therefore, the third order polynomial is

Then

))()(())(()()( 210310201014 xxxxxxbxxxxbxxbbxf

0078655.0],,,[

051873.0],,[

462098.0],[

0)(

01232

0122

011

00

xxxxfb

xxxfb

xxfb

xfb

))()((0078655.0))((051873.0)(462098.00)( 2101003 xxxxxxxxxxxxxf

%3.9 6287686.0)2(3 tf

60

Lagrange Interpolating Polynomials

Lagrange interpolating polynomial is a reformulation of the

Newton polynomial without the computation of divided

differences.

where

Π represents “the product of”.

For n = 1 i.e. linear (1st order) version:

For n = 2 i.e. quadratic (2nd order) version:

)()()(0

i

n

iin xfxLxf

n

ijj ji

j

ixx

xxxL

0

)(

)()()( 1

01

00

10

11 xf

xx

xxxf

xx

xxxf

)())((

))((

)())((

))((

)())((

))(()(

2

1202

10

1

2101

20

0

2010

212

xfxxxx

xxxx

xfxxxx

xxxx

xfxxxx

xxxxxf

61

Example 4:

Use a Lagrange interpolating polynomial of the first and second

order to evaluate ln2 given the following data:

x0 = 1 f (x0) = 0

x1 = 4 f (x1) = 1.386294

x2 = 6 f (x2) = 1.791759


Using first order Lagrange polynomial:

Using second order Lagrange polynomial:

The results are similar to those of Newton’s interpolation.

4620981.0386294.114

120

41

42)(1

xf

5658444.0791759.1)46)(16(

)42)(12(

386294.1)64)(14(

)62)(12(

0)61)(41(

)62)(42()(2

xf

62

Interpolation by Spline Functions

Objectives

Introduction

Linear Splines

Quadratic Splines

Cubic Splines

Introduction

Higher order polynomials can lead to erroneous results

because of round off error and oscillations to be

avoided.

Lower-order polynomial fits should be used.

Alternative approach is to apply lower-order polynomials

to subsets of data points : spline functions.

The drafting technique of using a spline to draw smooth

curves through a series of points. Notice how, at the end

points, the spline straightens out This is “natural” spline.

63

Notation used to derive splines. Notice that there are n–1

intervals and n data points.

Why Splines ?

64

Original Functio

n

1

6 t

h Order Polynomial

8 th Order

Polynomial

4 th Order

Polynomial

65

Higher order polynomial interpolation is a bad idea

f(x) = 1/(1+25x2)

Spline Interpolation Definition

Notation used to derive splines. Notice that there are n–1

intervals and n data points.

Each interval i has its own spline function Si(x)

66

Given n distinct knots xi such that:

with n knot values fi find a spline function

with each Si(x) a polynomial of degree at most n.

nn xxxx 121 ...

],[ )(

],[ )(

],[ )(

)(

11

322

211

nnn xxxxS

xxxxS

xxxxS

xS

67

Linear Splines

Simplest form of spline interpolation

Given the data

Consecutive data points connected through straight lines.

Each Si is a linear interpolation function constructed as:

),(),,(),.....,,(),,( 112211 nnnn fxfxfxfx

)()( iiii xxbaxS

68

Using the formulation of Newton linear interpolation, the

linear splines can be written as:

Example 1

Given the following data, evaluate the function at x=5

using linear splines.


i xi fi

1 3.0 2.5

2 4.5 1.0

3 7.0 2.5

4 9.0 0.5

],[ )(

)()()()(

],[ )( )()(

)()(

],[ )( )()(

)()(

)(

10

1

111

322

23

2322

211

12

1211

nn

nn

nnnn xxxxx

xx

xfxfxfxS

xxxxxxx

xfxfxfxS

xxxxxxx

xfxfxfxS

xS

69

x=5 is between x=4.5 and x=7. Hence, f(5) is evaluated

using linear splines on the second interval

The function value at x=5

)5.4(4.5-7.0

1.0-2.51.0

)( )()(

)()( 2

23

2322

x

xxxx

xfxfxfxS

3.14.5)(55407

015201)5()5( 2

.-.

.-..Sf

70

Quadratic Splines

Quadratic splines are rarely used for interpolation for

practical purposes

Ideally quadratic splines are only used to understand cubic

splines

Each Si is a second-order polynomial constructed as:

The splines are given by:

2)()()( iiiiii xxcxxbaxS

71

Find ai, bi, ci for i=1,2,…,n-1 3(n-1) unknown constants

to evaluate 3(n-1) equations are required.

Derivation of Quadratic Splines

1. The function must pass through all the points (Continuity

condition), i.e. for x=xi, we have:

(n-1) equations

2. Functions value of adjacent polynomials must be equal at

the knots i.e. for knot i+1 we have:

By setting hi = xi+1 – xi, the above equation simplifies to:

],[ )()()(

],[ )()()(

],[ )()()(

)(

1

2

111111

32

2

222222

21

2

111111

nnnnnnnn xxxxxcxxbaxS

xxxxxcxxbaxS

xxxxxcxxbaxS

xS

2)()()( iiiiiiiiii xxcxxbafxS ii fa

2

1111111

2

11 )()()()( iiiiiiiiiiiiii xxcxxbfxxcxxbf

1

2

iiiiii fhchbf

72

this equation can be written for knots i=1,2,…,n-1

(n-1) equations

3. The first derivatives at the interior nodes must be equal

(adjacent splines will be joined smoothly).

Quadric spline function can be differentiated to

first derivatives at interior knot i+1 gives:

(n-2) equations

4. Make one arbitrary choice i.e. assume the second derivative

is zero at the first point.

(1) equations

)(2)('

iiii xxcbxS

12 iiii bhcb

01 c

73

Example 2


using Quadratic splines.


Since there are 4 data points, 3 quadratic splines pass

through them

i xi fi

1 3.0 2.5

2 4.5 1.0

3 7.0 2.5

4 9.0 0.5

]9,7[ )7()7()(

]7,5.4[ )5.4()5.4()(

]5.4,3[ )3()3()(

2

3333

2

2222

2

1111

xxcxbaxS

xxcxbaxS

xxcxbaxS

74

The necessary function and interval width values are:

i xi fi hi

1 3.0 2.5 4.5 – 3.0 = 1.5

2 4.5 1.0 7.0 – 4.5 = 2.5

3 7 2.5 9.0 – 7.0 = 2.0

4 9 0.5

Setting up the equations

1. The function must pass through all the points:


the knots:

ii fa

5.2

0.1

5.2

33

22

11

fa

fa

fa

1

2

iiiiii fhchbf

4

2

33333

3

2

22222

2

2

11111

fhchbf

fhchbf

fhchbf

5.040.25.2

5.225.65.20.1

0.15.15.2

33

22

1

cb

cb

b

75

3. The first derivatives at the interior nodes must be equal

4. Assume the second derivative is zero at the first point.

All the equations can be assembled in a matrix form as

These equations can be solved with the results:

Along with and

12 iiii bhcb 3222

21

2 bhcb

bb

322

21

5 bcb

bb

01 c

0

0

2

5.1

5.1

01510

00011

42000

0025.65.20

00005.1

3

3

2

2

1

c

b

c

b

b

6.1

2.2

64.0

0.1

0.1

3

3

2

2

1

c

b

c

b

b

5.2

0.1

5.2

3

2

1

a

a

a

01 c

76

The quadratic splines are then given by

Because x=5 lies in the second interval, we use S2 to make

the prediction

]9,7[ )7(1.6-)7(2.25.2)(

]7,5.4[ )5.4(64.0)5.4(0.1)(

]5.4,3[ )3(5.2)(

2

3

2

2

1

xxxxS

xxxxS

xxxS

66.0)5.45(0.64)5.45(0.1)5( 2

2 S

77

Cubic Splines

Third-order curves used to connect each pair of data points

are called cubic splines.

We have n data points n-1 intervals 4n-4 unknowns

constants 4n-4 conditions are needed

The first 3 conditions are identical to those used for the

quadratic splines:

1. Spline functions must pass through all the data points

2. Function values must be equal at the interior knots

3. First derivatives at the interior knots must be equal

4. Second derivatives at the interior knots must be equal

5. Second derivatives at the end knots are zero

Or

5. First derivatives at the first and last knots are specified

Derivation of Cubic Splines

1. The function must pass through all the points (Continuity

condition), i.e. for x=xi, we have:


the knots i.e. for knot i+1 we have:

32 )()()()( iiiiiiii xxdxxcxxbaxs

ii fa

1

32

iiiiiiii fhdhchbf

78

3. The first derivatives at the interior nodes must be equal i.e.

at interior knot i+1 gives:

4. The second derivatives at the interior nodes must also be

equal.

Cubic spline function can be differentiated to

second derivatives at interior knot i+1 gives:

5. Second derivatives at the end knots are zero

Let’s summarize the obtained equations:

(1)

(2)

1

232 iiiiii bhdhcb

)(62)(''

iiii xxdcxS

13 iiii chdc

0

01

nc

c

ii fa

1

32

iiiiiiii fhdhchbf

1

232 iiiiii bhdhcb

79

(3)

(4)

(5)

Solve Eq. (4) for di:

(6)

Substitute into Eq. (2) and (3)

(7)

(8)

Solve Eq.(7) for bi:

(9)

Reduce the index of Eq. (8) and (9) by 1

13 iiii chdc

0

01

nc

c

i

iii

h

ccd

3

1

11

2

)2(3

iiii

iii fcch

hbf

)( 11 iiiii cchbb

)2(3

11

iii

i

iii cc

h

h

ffb

)2(3

11

1

11 ii

i

i

iii cc

h

h

ffb

80

(10)

(11)

Substitute Eq. (9) and (10) into Eq. (11)

(12)

Equation (12) can be written in matrix form as

)( 111 iiiii cchbb

]),[],[(3)(2 111111 iiiiiiiiiii xxfxxfchchhch

0

])[],[(3

])[],[(3

0

1

)(2

)(2

1

211

1223

1

2

1

1122

2211

nnnn

n

nnnnn xxfxxf

xxfxxf

c

c

c

c

hhhh

hhhh

81

Example 3


using Cubic splines.


Since there are 4 data points, 3 quadratic splines pass

through them

i xi fi

1 3.0 2.5

2 4.5 1.0

3 7.0 2.5

4 9.0 0.5

]9,7[ )7()7()7()(

]7,5.4[ )5.4()5.4()5.4()(

]5.4,3[ )3()3()3()(

3

3

2

3333

3

2

2

2222

3

1

2

1111

xxdxcxbaxS

xxdxcxbaxS

xxdxcxbaxS

82

The necessary function and interval width values are:

i xi fi hi

1 3.0 2.5 4.5 – 3.0 = 1.5

2 4.5 1.0 7.0 – 4.5 = 2.5

3 7 2.5 9.0 – 7.0 = 2.0

4 9 0.5

Determine the c coefficients from the set of simultaneous

equations

Use equations (9) and (6) to compute bi and di

0

])[],[(3

])[],[(3

0

1

)(2

)(2

1

2334

1223

4

3

2

1

3322

2211

xxfxxf

xxfxxf

c

c

c

c

hhhh

hhhh

0

8.4

8.4

0

1

295.2

5.285.1

1

4

3

2

1

c

c

c

c

0

7665.0

8395.0

0

4

3

2

1

c

c

c

c

022.0

16.0

42.1

3

2

1

b

b

b

83

The cubic splines for each interval

The function value at x=5 is calculated using S2

)2(3

11

iii

i

iii cc

h

h

ffb

i

iii

h

ccd

3

1

128.0

214.0

187.0

3

2

1

d

d

d

]9,7[ )7(128.0)7(0.767-)7(022.05.2)(

]7,5.4[ )5.4(214.0)5.4(84.0)5.4(16.00.1)(

]5.4,3[ )3(187.0)3(42.15.2)(

32

3

3

2

2

2

3

1

xxxxxS

xxdxxxS

xxxxS

1.103

5452140545840545160015 3

2

2

2

).(d.).(.).(..)(S

84

Example - Summary

Spline fits of the set of four points given in the example

(a) Linear spline,

(b) quadratic spline, and

(c) cubic spline, with a cubic

85

Curve Fitting and Interpolation

Least Squares Regression

Objectives

Introduction

Linear regression

Polynomial regression

Multiple linear regression

General linear least squares

Nonlinear regression

Curve Fitting

Experimentation

Data available at discrete points or times

Estimates are required at points between the discrete

values

Curves are fit to data in order to estimate the

intermediate values

Two methods - depending on error in data

Interpolation

- Precise data

- Force through each data point

86

Regression

- Noisy data

- Represent trend of the data

0

1

2

3

4

5

6

7

8

9

0 2 4 6 8 10 12 14 16

x

f(x)

0

20

40

60

80

100

120

0 1 2 3 4 5

Time (s)

Tem

pera

ture

(deg

F)

87

Least Squares Regression

Experimental Data

Noisy (contains errors or inaccuracies)

x values are accurate, y values are not

Find relationship between x and y = f(x)

Fit general trend without matching individual points

Derive a curve that minimizes the discrepancy

between the data points and the curve Least-

squares regression

Linear Regression: Definition

Noisy Data From Experiment

x y

2.10 2.90

6.22 3.83

7.17 5.98

10.5 5.71

13.7 7.74

88

Straight line characterizes trend without passing through

particular point

Linear Regression: criteria for a “best” fit

How do we measure goodness of fit of the line to the data?

0

1

2

3

4

5

6

7

8

9

0 2 4 6 8 10 12 14 16

x

f(x)

y 1 y 2

y 4

y 5

y 3

e 3

e 2

Regression Model

y = a 0 + a 1x

Residual

e = y - (a 0 + a 1x )

Data points

0

1

2

3

4

5

6

7

8

9

0 2 4 6 8 10 12 14 16

x

f(x)

89

Use the curve that minimizes the residual between the data

points and the line

Model: xaay 10 , ii xaay 10

iii xaaye 10

2

1 110

2 ][

n

i

n

iiiir xaayeS

Find the values of a0 and a1 that minimize Sr

Minimize Sr by taking derivatives WRT

a0 and a1,

First a0

Finally

n

ii

n

ii yaxna

11

10

Second a1

0

)1(xa a y2

a2

a

xa a ya

= a

S

n

1=i

i10i

n

1=i 0

n

1=i

2

0

n

1=i

2

i10i

00

r

0

)x(xa a y2

a2

a

xa a ya

= a

S

n

1=i

ii10i

n

1=i 1

n

1=i

2

1

n

1=i

2

i10i

11

r

90

Finally

n

iii

n

ii

n

ii yxaxax

11

1

2

01

Set of two simultaneous linear equations with two

unknowns ( a0 and a1):

n

ii

n

ii yaxna

11

10

n

iii

n

ii

n

ii yxaxax

11

1

2

01

The normal equations can be solved simultaneously for:

2

11

2

1111

n

ii

n

ii

n

ii

n

ii

n

iii

xxn

yxyxn

a

xaya 10

91

Example 1:

Fit a straight line to the values in the following table

x y

2.10 2.90

6.22 3.83

7.17 5.98

10.5 5.71

13.7 7.74

i xi yi xi2 xiyi

1 2.10 2.90 4.41 6.09

2 6.22 3.83 38.69 23.82

3 7.17 5.98 51.41 42.88

4 10.5 5.71 110.25 59.96

5 13.7 7.74 187.69 106.04

Σ 39.69 26.16 392.32 238.74

x4023190.0 038392.2 y

92

Linear Regression: Quantification of error

Suppose we have data points (xi, yi) and modeled (or

predicted) points (xi, ŷi) from the model ŷ = f(x).

Data {yi} have two types of variations;

(i) variation explained by the model and

(ii) variation not explained by the model.

Residual sum of squares: variation not explained by the

model

n

iiir yyS

1

2)ˆ(

Regression sum of squares: variation explained by the

model

n

iit yyS

1

2)ˆ(

The coefficient of determination r2

t

rt

S

SSr

2

For a perfect fit

Sr=0 and r=r2=1, signifying that the line explains 100% of

the variability of the data.

For r=r2=0, Sr=St, the fit represents no improvement.

In addition to r2, r

Define 2

n

SS r

xy

= standard error of the estimate

93

- Represents the distribution of the residuals around the

regression line

- Large Sy|x large residuals

- Small Sy|x small residuals

Example 2

Compute the total standard deviation, the standard error of

the estimate, and the correlation coefficient for the data in

Example 1.


i xi yi 2)( yyi 2

10 )( ii xaay

1 2.1 2.9 5.4382 0.0003

2 6.22 3.83 1.9656 0.5045

3 7.17 5.98 0.5595 1.1183

4 10.5 5.71 0.2285 0.3049

5 13.7 7.74 6.2901 0.0363

Σ 39.69 26.16 14.4819 1.9643

The standard deviation is

The standard error of the estimate is

9028.115

4819.14

1

n

SS t

y

94

The correlation coefficient r is

8092.025

9643.1

2|

n

SS r

xy

8644.04819.14

9643.14819.142

t

rt

S

SSr

9297.08644.0 r

95

Linearization of Nonlinear Relationships

If relationship is an exponential function bxaey

To make it linear, take logarithm of both sides

bxay lnln

Now it’s a linear relation between ln(y) and x

If relationship is a power functionbaxy

To make linear, take logarithm of both sides

xbay lnlnln

Now it’s a linear relation between ln(y) and ln(x)

97

Polynomial Regression

Minimize the residual between the data points and the

curve -- least-squares regression

Linear ii xaay 10

Quadratic 2

210 iii xaxaay

Cubic 3

3

2

210 iiii xaxaxaay

General m

imiiii xaxaxaxaay ...3

3

2

210

Must find values of a0 , a1, a2, … am

Residual m

imiiii xaxaxaaye ....2

210

Sum of squared residuals

2

1

2

2101

2 ....

n

i

m

imiii

n

iir xaxaxaayeS

Minimize by taking derivatives

Normal Equations

n

1=ii

m

i

n

1=ii

2

i

n

1=iii

n

1=ii

m

2

1

0

n

1=i

m2

i

n

1=i

2m

i

n

1=i

1m

i

n

1=i

m

i

n

1=i

2m

i

n

1=i

4

i

n

1=i

3

i

n

1=i

2

i

n

1=i

1m

i

n

1=i

3

i

n

1=i

2

i

n

1=ii

n

1=i

m

i

n

1=i

2

i

n

1=ii

yx

yx

yx

y

a

a

a

a

xxxx

xxxx

xxxx

xxxn

98

Example 3

Fit a third-order polynomial to the data given in the Table

below

x 0 1.0 1.5 2.3 2.5 4.0 5.1 6.0 6.5 7.0 8.1 9.0

y 0.2 0.8 2.5 2.5 3.5 4.3 3.0 5.0 3.5 2.4 1.3 2.0

x 9.3 11.0 11.3 12.1 13.1 14.0 15.5 16.0 17.5 17.8 19.0 20.0

y -0.3 -1.3 -3.0 -4.0 -4.9 -4.0 -5.2 -3.0 -3.5 -1.6 -1.4 -0.1


3597.9943

242.6037

88.316

30.1

a

a

a

a

0.2235181200.127801482.75283579.46342

0.127801482.75283579.463422.3060

2.75283579.463422.30606.229

79.463422.30606.22924

3

2

1

0

01206020.0

35319014.0

3051112.2

35934718.0

a

a

a

a

3

2

1

0

99

Regression Equation

y = - 0.359 + 2.305x - 0.353x2 + 0.012x3

-6

-4

-2

0

2

4

6

0 5 10 15 20 25

x

f(x)

100

Multiple Linear Regression

y = a0 + a1x1 + a2x2 + e

Again very similar.

Minimize e

Polynomial and multiple regression fall within definition of

General Linear Least Squares.

residualsE

tscoefficienunknown A

variabledependent theof valuedobservedY

t variableindependen theof valuesmeasured at the

functions basis theof valuescalculated theofmatrix

functions basis 1 are 10

221100

Z

EAZY

m, z, , zz

ezazazazay

m

mm

2

1 0

n

i

m

jjijir zayS

101

Minimized by taking its partial derivative w.r.t. each of the

coefficients and setting the resulting equation equal to zero

Not all equations can be broken down into General Linear

Least Squares model i.e. eeayxa

i )1( 1

0

Solve with nonlinear least squares using iterative methods

like Gauss-Newton method

Equation could possibly be transformed into linear form

Caveat: when fitting transformed data you minimize

the residuals of the data you are working with

May not give you the exact same fit as nonlinear

regression on untransformed data

103

Numerical Differentiation and Integration

Numerical Differentiation

Objectives

Introduction

Taylor Series

Forward difference method

Backward difference method

Central difference method

Introduction

Calculus is the mathematics of change. Because engineers

must continuously deal with systems and processes that

change, calculus is an essential tool of engineering.

Standing in the heart of calculus are the mathematical

concepts of differentiation and integration:

b

a

iix

ii

dxxfI

x

xfxxf

dx

dy

x

xfxxf

x

y

)(

)()(lim

)()(

0

104

Integration and differentiation are closely linked processes.

They are, in fact, inversely related.

Types of functions to be differentiated or integrated:

1. Simple polynomial, exponential, trigonometric

analytically

2. Complex function numerically

3. Tabulated function of experimental data

numerically

Applications

Differentiation has so many engineering applications (heat

transfer, fluid dynamics, chemical reaction kinetics, etc…)

Integration is equally used in engineering (compute work in

ME, nonuniform force in SE, cross-sectional area of a river,

etc…)

Differentiation

105

The finite difference becomes a derivative as Δx approaches

zero.

x

xfxxf

x

y ii

)()(

x

xfxxf

dx

dy ii

x

)()(lim

0

106

Integration

Differentiation vs. Integration

108

Taylor Series Expansion

Non-elementary functions such as trigonometric,

exponential, and others are expressed in an approximate

fashion using Taylor series when their values, derivatives,

and integrals are computed.

Any smooth function can be approximated as a polynomial.

Taylor series provides a means to predict the value of a

function at one point in terms of the function value and its

derivatives at another point.

Numerical Application of Taylor Series

If f (x) and its first n+1 derivatives are continuous on an

interval containing xi+1 and xi , then:

Where the remainder Rn is defined as:

n

n

iii

n

iii

iii

iiiii

Rxxn

xfxx

xf

xxxf

xxxfxfxf

1

3

1

3

2

1

''

1

'

1

! ...

!3

!2

1

1

1

!1

n

ii

n

n xxn

fR

109

The series is built term by term

zero order approximation

Continuing the addition of more terms to get better

approximation we have: 1st order approximation

2nd order approximation

Finally

where

ξ is a value of x that lies somewhere between xi and xi+1.

ii xfxf 1

iiiii xxxfxfxf 1

'

1

21

''

1

'

1!2

iii

iiiii xxxf

xxxfxfxf

n

n

iii

n

iii

iii

iiiii

Rxxn

xfxx

xf

xxxf

xxxfxfxf

1

3

1

3

2

1

''

1

'

1

! ...

!3

!2

1

1

1

!1

n

ii

n

n xxn

fR

110

Taylor Series ξ in the Remainder Term

Limitations

ξ is not exactly known but lies somewhere between xi

and xi+1

To evaluate Rn, the (n+1) derivative of f (x) has to be

determined. To do this f (x) must be known

if f (x) was known there would be no need to perform the

Taylor series expansion!!!

Modification

Rn = O(hn+1) the truncation error is of the order of hn+1 . (h = xi+1

– xi)

If the error is O(h), halving the step size will halve the

error.

If the error is O(h2), halving the step size will quarter

the error.

In general, the truncation error is decreased by

addition of more terms in the Taylor series.

Forward Difference Formulas- 1st derivative

2nd order Taylor series expansion of f(x) can be written as:

(1)

Then, the first derivative can be expressed as:

)(2

)('')(')()( 32

1 hOhxf

hxfxfxf iiii

)(2

)('')()()(' 21 hOh

xf

h

xfxfxf iii

i

111

(2)

Given that )(xf can be approximated by:

(3)

Substituting equation (3) into equation (2):

Collecting terms and simplifying, we have:

Note that the inclusion of the second-derivative term has

improved the accuracy to O(h2).

Forward Difference Formulas- 2nd derivative

Start with Lagrange interpolation polynomial for f(x) based

on the four points xi, xi+1, xi+2 and xi+3.

Differentiate the products in the numerators twice

Substitute x = xi and consider the fact that xj - xi = (j- i)h

)()()(2)(

)(''2

12 hOh

xfxfxfxf iii

i

)(2

)()(2)()()()(' 2

2

121 hOhh

xfxfxf

h

xfxfxf iiiii

i

)(2

)(3)(4)()(' 212 hO

h

xfxfxfxf iii

i

112

The expression of the second derivative is then:

Backward Difference Formulas- 1st derivative

Using backward difference in the Taylor series expansion,

And given that )(xf can be approximated by:

The second-order estimate of )(xf can be obtained:

Backward Difference Formulas- 2nd derivative

Start with Lagrange interpolation polynomial for f(x) based

on the four points xi, xi-1, xi-2 and xi-3 .

Differentiate the products in the numerators twice

Substitute x = xi and consider the fact that xj - xi = (j- i)h

)()(2)(5)(4)(

)('' 2

2

123 hOh

xfxfxfxfxf iiii

i

)(2

)('')(')()( 32

1 hOhxf

hxfxfxf iiii

)()()(2)(

)(''2

21 hOh

xfxfxfxf iii

i

)(2

)()(4)(3)(' 221 hO

h

xfxfxfxf iii

i

113

The expression of the second derivative is then:

Centered Difference Formulas- 1st derivative [O(h2)]

Start with the 2nd degree Taylor expansions about x for

f(x+h) and f(x-h):

(4)

(5)

Subtract (5) from (4)

Hence

Centered Difference Formulas- 1st derivative [O(h4)]

Start with the difference between the 4th degree Taylor

expansions about x for f(x+h) and f(x-h):

)()()(4)(5)(2

)('' 2

2

321 hOh

xfxfxfxfxf iiii

i

)(2

)('')(')()( 32

1 hOhxf

hxfxfxf iiii

)(2

)('')(')()( 32

1 hOhxf

hxfxfxf iiii

)()('2)()( 3

11 hOhxfxfxf iii

)(2

)()()(' 211 hO

h

xfxfxf ii

i

)(!3

)('''2)('2)()( 53

11 hOhxf

hxfxfxf iiii

114

(6)

Use the step size 2h, instead of h, in (6)

(7)

Multiply equation (6) by 8, subtract (7) from it, and solve

for f’(x)

Centered Difference Formulas- 2nd derivative [O(h2)]

Start with the 3rd degree Taylor expansions about x for

f(x+h) and f(x-h):

(8)

(9)

Add equations (8) and (9), and solve for )(xf

)(!3

)('''16)('4)()( 53

22 hOhxf

hxfxfxf iiii

)(12

)()(8)(8)()(' 42112 hO

h

xfxfxfxfxf iiii

i

)(!3

)('''

2

)('')(')()( 432

1 hOhxf

hxf

hxfxfxf iiiii

)(!3

)('''

2

)('')(')()( 432

1 hOhxf

hxf

hxfxfxf iiiii

115

Centered Difference Formulas- 2nd derivative [O(h4)]

Start with the addition between the 5th degree Taylor

expansions about x for f(x+h) and f(x-h):

(10)

Use the step size 2h, instead of h, in (10)

(11)

Multiply equation (10) by 16, subtract (11) from it, and

solve for f’’(x)

Error Analysis

Generally, if numerical differentiation is performed, only

about half the accuracy of which the computer is capable is

obtained unless we are fortunate to find an optimal step

size.

)()()(2)(

)('' 2

2

11 hOh

xfxfxfxf iii

i

)(!4

)(2

!2

)(''2)(2)()( 64

)4(2

11 hOhxf

hxf

xfxfxf iiiii

)(!4

)(32

!2

)(''8)(2)()( 64

)4(2

22 hOhxf

hxf

xfxfxf iiiii

)(12

)()(16)(30)(16)()('' 4

2

2112 hOh

xfxfxfxfxfxf iiiii

i

116

The total error has part due to round-off error plus a part

due to truncation error.

Total Numerical Error

Total numerical error = truncation error + round-off error.

Trade-off between truncation and round-off errors

Example 1

Estimate the first and second derivatives of:

at x = 0.5 and h = 0.25 using

432 1.015.05.025.02.1)( xxxxxf

117

a) forward finite-divided difference

b) Centered finite-divided difference

c) backward finite-divided difference?


a) Forward difference 1st derivative computation

The data needed is:

First derivative:

εt = 5.82%

True value=-0.9125

2.0)( 1

636328.0)( 75.0

925.0)( 5.0

22

11

ii

ii

ii

xfx

xfx

xfx

859375.0)25.0(2

)5.0(3)75.0(4)1()5.0('

2

)(3)(4)()(' 12

ffff

h

xfxfxfxf iii

i

118

Second derivative computation

The data needed is:

Second derivative:

b) Centered finite-divided difference

The data needed is:

94336.1)( 25.1

2.0)( 1

636328.0)( 75.0

925.0)( 5.0

33

22

11

ii

ii

ii

ii

xfx

xfx

xfx

xfx

6.39)25.0(

)925.0(2)636328.0(5)2.0(4)94336.1()5.0(''

)(2)(5)(4)()(''

2

2

123

f

h

xfxfxfxfxf iiii

i

2.0)( 1

636328.0)( 75.0

103516.1)( 25.0

2.1)( 0

22

11

11

22

ii

ii

ii

ii

xfx

xfx

xfx

xfx

119

First derivative:

εt = 0%.

True value=-0.9125

c) Backward finite-divided difference

The data needed is:

9125.0)('

)25.0(12

)2.1()103516.1(8)636328.0(8)2.0()('

12

)()(8)(8)()(' 2112

i

i

iiiii

xf

xf

h

xfxfxfxfxf

925.0)( 5.0

103516.1)( 25.0

2.1)( 0

11

22

ii

ii

ii

xfx

xfx

xfx

120

First derivative :

εt = 3.77%

True value=-0.9125

878125.0)25.0(2

)2.1()103516.1(4)925.0(3)('

2

)()(4)(3)(' 21

i

iiii

xf

h

xfxfxfxf

121

Numerical Differentiation and Integration

Numerical Integration (I)

Objectives

Introduction

Trapezoidal Rule

Multiple-Application Trapezoidal Rule

Simpson’s 1/3 Rule

Multiple-Application Simpson’s 1/3 Rule


Integration with unequal segments

Gauss Quadrature

Integration – Graphical representation

122

b

a

dxxf )(

Integration – Engineering applications

123

Newton-Cotes Integration Formulas

The Newton-Cotes formulas are the most common

numerical integration schemes.

This involves the replacement of a complicated function or

tabulated data with an approximating function that is easy

to integrate.

Where fn(x) is a polynomial of order n

Closed and open forms

Closed: data points at the beginning and end of the

limits of integration are known

Open: integration limits that extend beyond the range

of data

b

a n

b

adxxfdxxfI )()(

n

n

n

nn xaxaxaxaaxf

1

1

2

210 ...)(

124

Trapezoidal Rule

The Trapezoidal rule is the first of the Newton-Cotes

closed integration formulas, corresponding to the case

where the polynomial is first order i.e. n = 1

f1(x) is expressed using linear interpolation formula

The area under the straight line is an estimate of the

integral of f1(x):

The result of this integration is

b

a

b

adxxfdxxfI )()( 1

)()()(

)()(1 axab

afbfafxf

dxaxab

afbfafI

b

a

)(

)()()(

125

Trapezoidal rule

Area of a trapezoid = (Width) X (Average Height)

Error of the Trapezoidal Rule

Truncation error is expressed as:

An approximation of the second derivative is given by

2

)()()(

bfafabI

)('')(12

1 3 fabEt

ab

dxxfff

b

a

)(''

'')(''

126

Approximate error estimate is then:

Example 1

Use Trapezoidal Rule to integrate numerically the following

function:

from a = 0 to b = 0.8


Evaluate f(a) and f(b)

Evaluate the integral

Approximate error estimate

'')(12

1 3 fabEa

5432 400900675200252.0)( xxxxxxf

2.0)0( f

232.0)8.0( f

1728.02

232.02.0)08.0(

I

6008.0

)8000108004050400()(''

8.0

0

32

dxxxx

xf

56.2)08.0)(60(12

1))((''

12

1 33 abxfEa

127

True value of the integral is 1.640533

Et = 1.640533 – 0.1728

= 1.467733 εt = 89.5%

Ea = 2.56

Discrepancy between Et and Ea due to the fact that )(xf is

not an accurate approximation of )(f for an interval of

this size.

Multiple-Application or Composite Trapezoidal Rule

Divide integration interval from a to b into n segments of

equal width improve the accuracy of the trapezoidal rule

The total integral can be represented as

n

n

x

x

x

x

x

xdxxfdxxfdxxfI

1

2

1

1

0

)(...)()(

128

Substituting the trapezoidal rule for each integral, we get

By grouping terms, we have

Multiple-Application Trapezoidal Rule

Error of Multiple-Application Trapezoidal Rule

2

)()(...

2

)()(

2

)()( 12110 nn xfxfh

xfxfh

xfxfhI

)()(2)(2

1

10 n

n

ii xfxfxf

hI

129

Truncation error is obtained by summing the individual

error for each segment

By estimating the average value of )(xf for the entire

interval

Approximate error estimate for multiple-application of

trapezoidal rule is then:

As you double the number of segments error is quartered

Example 2

Apply two segments in the trapezoidal rule to estimate the

integral of:

from a = 0 to b = 0.8

n

it f

n

abE

13

3

)(''12

)(

n

f

f

n

i

1

)(''

''

'')(''1

fnfn

i

''12

)(2

3

fn

abEa

5432 400900675200252.0)( xxxxxxf

130


We have n = 2


True and Approximate error estimate

Et = 1.640533 – 1.0688 = 0.571733


It corresponds to the case where n = 2 i.e. a polynomial of

second order:

Let a = x0 and b = x2 and f2(x) a second order Lagrange

polynomial, then

4.02

08.0

h

2.0)0( f 456.2)4.0( f 232.0)8.0( f

0688.14

232.0)456.2(22.08.0

I

64.0)60()2(12

)08.0(2

3

aE

b

a

b

adxxfdxxfI )()( 2

dxxfxxxx

xxxx

xfxxxx

xxxxxf

xxxx

xxxxI

x

x

)())((

))((

)())((

))(()(

))((

))((

2

1202

10

1

2101

200

2010

212

0

131

After integration, we have:

Estimated truncation error

Example 3

Evaluate the integral of:

from a = 0 to b = 0.8 using Simpson’s 1/3 rule.

Compute the truncation errors Et and Ea


Evaluate the integral with h = (b-a)/2 = 0.4

)()(4)(3

210 xfxfxfh

I 2

abh

)4(5

2880

)(f

abEa

ab

dxxfff

b

a

)(

)(

)4(

)4()4(

5432 400900675200252.0)( xxxxxxf

3674671232045624203

40)()(4)( 210 ..).(.

.xfxfxf

3

hI

132

True truncation error

Et = 1.640533 – 1.367467 = 0.2730667 εt = 16.6%

Approximate truncation error

Multiple-Application of Simpson’s 1/3 Rule

Just as the trapezoidal rule, Simpson’s 1/3 rule can be

improved by dividing the integration interval into n

segments of equal width

The total integral can then be represented by

Example 4:

Use n = 4 to evaluate the integral of:

240008.0

)()( 4800021600)(

8.0

0

)4(

)4()4(

dxxf

xfxxf

2730667.0)2400(2880

)8.0()(

2880

)( 5)4(

5

xfab

Ea

)()(2)(4)(3

2

,...6,4,2

1

,...5,3,10 n

n

ji

n

ii xfxfxfxf

hI )4(

4

5

180

)(f

n

abEa

5432 400900675200252.0)( xxxxxxf

133

from a = 0 to b = 0.8 using Simpson’s 1/3 rule.

Compute the truncation errors Et and Ea

Example 4 – Solution

The integral using composite simpson’s 1/3 rule

h = (0.8-0)/4 = 0.2

f(0)=0.2 f(0.2)=1.288 f(0.4)=2.456 f(0.6)=3.464

f(0.8)=0.232

Et = 1.640533 – 1.623466 = 0.017067 εt = 1.0%,

Approximate truncation error

623466.1

232.0)464.3(4456.23

2.0456.2)288.1(42.0

3

2.0

I

017067.0)2400()4(180

)8.0(

180

)(4

5)4(

4

5

fn

abEa

134


3rd order polynomial is fit to 4 points and integrated to

yield

Simpson’s 3/8 rule has an error of

There is also a multiple segment version.

Combination of Simpson’s 1/3 and 3/8 rules

Can also use combinations of Simpson’s 1/3 and 3/8 rules

for unevenly spaced data

)()(3)(3)(8

33210 xfxfxfxf

hI

3

abh

)(6480

)( )4(5

fab

Et

135

Example 5

1. Use Simpson’s 3/8 rule to integrate:

from a = 0 to b = 0.8

2. Use Simpson’s 1/3 and 3/8 rules to integrate the same

function for five segments.

Example 5 – Solution

1. A single application of Simpson’s 3/8 rule requires 4

equally spaced points:

5432 400900675200252.0)( xxxxxxf

136

h = (0.8-0)/3 = 0.2667

f(0) = 0.2 f(0.2667) = 1.432724

f(0.5333) = 3.487177 f(0.8)=0.232

2. For five segments:

h = (0.8-0)/5 = 0.16

f(0) = 0.2, f(0.16) = 1.296919, f(0.32) = 1.743393

f(0.48) = 3.186015, f(0.64)=3.181929, f(0.8) = 0.232

The integral for the first 2 segments (Simpson’s 1/3 rule):

For the last 3 segments (Simpson’s 3/8 rule)

The total integral

51970.1232.0)487177.3432724.1(32.08

)2667.0(3I

3803237.0743393.1)296919.1(42.03

)16.0(I

264754.1232.0)181929.3186015.3(3743393.18

)16.0(3I

645077.1264754.13803237.0 I

137

Remarks – Trapezoidal & Simpson’s Rule

Rather than applying trapezoidal rule with finer segments,

go to higher order polynomials

– Quadratic is used: Simpson’s 1/3

• Must be used with even number of segments

– Cubic is used: Simpson’s 3/8

• Odd number of segments

• Slightly more accurate than 1/3 but requires more

points

Simpson’s 1/3 is preferred since need fewer data points for

same order accuracy

Therefore, for odd number of segments, use 1/3 rule for all

but the last 3 segments, then use 3/8 rule.

Integration with unequal segments

Experimental data is often not equally spaced

All the previous equations were for equally spaced data

points

If you have some equally spaced points in the data set, Fit

with Simpson’s rules

Fit the rest of the segments with individual Trapezoidal

rules

138

Gauss Quadrature

Gauss quadrature implements a strategy of positioning two

points on a curve to define a straight line that would

balance the positive and negative errors.

The area evaluated under this straight line provides a faster

and an improved estimate of the integral compared to

Trapezoid and Simpson.

Find position of these points using Gauss-Legendre

formulas

139

Method of Undetermined Coefficients

The trapezoidal rule yields exact results when the function

being integrated is a constant or a straight line, such as y=1

and y=x:

)()( 10 bfcafcI

140

Solve simultaneously

Trapezoidal rule )(

2)(

2

2

022

22

1

10

10

10

2/)(

2/)(

10

2/)(

2/)(

10

bfab

afab

I

abcc

abc

abc

abcc

dxxab

cab

c

dxcc

ab

ab

ab

ab

141

Two-Point Gauss-Legendre Formula

The object of Gauss quadrature is to determine the

equations of the form

I ≈ c0 f(x0) + c1 f(x1)

However, in contrast to trapezoidal rule that uses fixed end

points a and b, the function arguments x0 and x1 are not

fixed end points but unknowns.

Thus, four unknowns to be evaluated require four

conditions.

First two conditions are obtained by assuming that the

above equation fits the integral of a constant and a linear

function exactly.

The other two conditions are obtained by extending this

reasoning to a parabolic and a cubic functions.

142

Solved simultaneously

Yields an integral

estimate that is third

order accurate

1

1

3

1100

1

1

2

1100

1

1

1100

1

1

1100

0)()(

3

2)()(

0)()(

21)()(

dxxxfcxfc

dxxxfcxfc

dxxxfcxfc

dxxfcxfc

3

1

3

1

3

1,

3

1,1 1010

ffI

xxcc

143

Notice that the integration limits are from -1 to 1. This was done

for simplicity and make the formulation as general as possible.

A simple change of variable is used to translate other limits

of integration into integration limits are from -1 to 1

If x=a and x=b correspond to xd=-1 and xd=1 respectively

==

This yields to

Example 6

Use two-point Gauss quadrature to integrate:

from a = 0 to b = 0.8

10 axax d

10

10

aab

aaa

2

2

1

0

aba

aba

2

)()( abxabx d ddx

abdx

2

)(

5432 400900675200252.0)( xxxxxxf

144


First, perform a variable change


The right hand side is suitable for using Gauss quadrature

εt = 11.1%

4.04.0 dxx

ddxdx 4.0

dxxx

xxx

dxxxxxxI

dd

ddd

4.0))4.04.0(400)4.04.0(900

)4.04.0(675)4.04.0(200)4.04.0(252.0(

)400900675200252.0(

54

31

1

2

5438.0

0

2

822578.1305837.1516741.03

1

3

1

ffI

145

Higher-Point Gauss Quadrature Formulas

Higher-point formulas can be developed in the general

form

I ≈ c0 f(x0) + c1 f(x1)+…+ cn-1 f(xn-1)

Requires function to be known, since unlikely to get data

points that match these locations in a data table

Provided that the higher order derivatives do not increase

substantially with increasing number of points (n), Gauss

quadrature is superior to Newton-Cotes formulas.

)()!22()32(

)!1(2 )22(

3

432

n

n

t fnn

nE

146

Error for the Gauss-Legendre formulas

147

Numerical Solutions of Ordinary Differential Equations

Initial Value Problem: Euler’s Method

Objectives

Introduction

Euler’s Method

Convergence analysis

Error analysis

Error estimate

Introduction

Differential equations: equations composed of an unknown

function and its derivatives.

Where,

v- dependent variable

t- independent variable

Differential equations play a fundamental role in

engineering because many physical phenomena are best

formulated mathematically in terms of their rate of change.

)()(

tvm

cg

dt

tdv

148

Fundamental laws of physics, mechanics, electricity and

thermodynamics are written in terms of the rate of change

of variables (t = time and x = position):

Newton’s second law of motion

Fourier’s heat law

Lapalce equation (steady state)

Heat conduction

etc.

One independent variable Ordinary Differential

Equation (ODE)

Two or more independent variables Partial Differential

Equation (PDE)

Order of a differential equation is determined by the

highest derivative.

02

2

2

2

y

T

x

T

m

F

dt

dv

dx

dTkq '

2

2

'x

Tk

t

T

149

Higher order equations can be reduced to a system of first

order equations, by redefining a variable.

An ODE is accompanied by auxiliary conditions. These

conditions are used to evaluate the integral that result

during the solution of the equation. An nth order equation

requires n conditions.

If all conditions are specified at the same value of the

independent variable, then we have an initial-value

problem.

If the conditions are specified at different values of the

independent variable, usually at extreme points or

boundaries of a system, then we have a boundary-value

problem.

Euler’s Method

Solve ordinary differential equations of the form

With initial condition

The estimate of the solution is

),( yxfdx

dy

ii yxy )(

150

• is the estimate of slope at xi

• is the differential equation evaluated at xi and yi

• The estimate is given by

hyy ii 1

),( ii yxf

),( ii yxf

hyxfyy iiii ),( 1

151

This is Euler’s method (Euler-Cauchy, Point-slope)

Error Analysis for Euler’s Method

The numerical solution of ODE involves 2 types of error:

1. Round-off errors: this caused by limited numbers of

significant digits that can be retained by a computer.

2. Truncation errors:

a) Local truncation error: due to the application of the

method over a single step.

b) Propagated truncation error: due to approximation

produced during previous steps.

152

Total or global truncation error = local truncation error +

propagated truncation error.

Error Estimate for Euler’s Method

The general form of the differential equation being

integrated:

(1)

Taylor’s series expansion of y about a starting value (xi,yi):

(2)

(1) into (2) gives

(3)

Subtract Euler’s method equation from (3) gives the

truncation error

Example 1:

1. Use Euler’s method to integrate numerically the ODE:

),(' yxfydx

dy

)(!

...!2

1)(

2''

'

1

nnn

iiiii hOh

n

yh

yhyyy

)(!

),(...

!2

),(),( 1

)1(2

'

1

nnii

n

iiiiii hOh

n

yxfh

yxfhyxfyy

)(!

),(...

!2

),(1

)1(

2

'

nnii

n

ii

t hOhn

yxfh

yxfE

5.820122 23 xxxdx

dy

153

from x = 0 to x = 4 with a step size of 0.5 and an

initial condition at x = 0 is y = 1?

2. Estimate and tabulate the errors


1. Using Euler’s method

h=0.5, y(0)=1

Evaluate the first prediction y(0.5)

Evaluate the second prediction y(1)

Evaluate the third prediction

hyxfyy iiii ),( 1

0.5)( )1 ,0()0( )5.0( fyy

5.85.8)0(20)0(12)0(2 )1 ,0( 23 f

5.258.5)(0.5)(1)( )5.0( y

0.5)( )25.5 ,5.0()5.0( )1( fyy

5.8)5.0(20)5.0(12)5.0(2 )1 ,0( 23f

875.5)(0.5)(5.25)( )1( y

0.5)( )875.5 ,1()1( )5.1( fyy

5.125-1.5)(0.5)(5.875 )5.1( y

154

x yEuler

0.0 1.000

0.5 5.250

1.0 5.875

1.5 5.125

2.0 4.500

2.5 4.750

3.0 5.875

3.5 7.125

4.0 7.000

155

2. Estimate the truncation error

For the first step

4)3(

3''

2'

!4

),(

!3

),(

!2

),(h

yxfh

yxfh

yxfE iiiiii

t

12)1,0( 12 ),(

24)1,0( 2412 ),(

20)1,0( 20246 ),(

)3()3(

''''

'2'

fyxf

fxyxf

fxxyxf

ii

ii

ii

03125.2)5.0(24

12)5.0(

6

24)5.0(

2

20 432

tE

156

x yEuler Et

0.00000 1.00000 0.00000

0.50000 5.25000 -2.03125

1.00000 5.87500 -2.87500

1.50000 5.12500 -2.90625

2.00000 4.50000 -2.50000

2.50000 4.75000 -2.03125

3.00000 5.87500 -1.87500

3.50000 7.12500 -2.40625

4.00000 7.00000 -4.00000

Remarks about Euler’s Method

The Taylor series provides only an estimate of the local

truncation error. It does not give a measure of the global

truncation error.

The global truncation error is O(h), that is, it is proportional

to the step size.

The error can be reduced by decreasing the step size.

157

If the underlying function, y(x), is linear, the method

provide error-free predictions. This is because for a straight

line the second derivative would be zero.

158

Numerical Solutions of Ordinary Differential

Equations

Initial Value Problem: Runge-Kutta Methods

Objectives

Introduction

Runge-Kutta Methods

Second-order Runge-Kutta

Heun Method

Midpoint Method

Ralston’s Method

Fourth-order Runge-Kutta

System of ODE

Introduction

A fundamental source of error in Euler’s method is that the

derivative at the beginning of the interval is assumed to

apply across the entire interval.

Improving the estimate of the slope will improve the

solution.

Runge-Kutta Methods

159

Runge-Kutta (RK) methods achieve the accuracy of a

Taylor series without requiring the calculation of higher

derivatives.

For a differential equation of the form:

The estimate of the solution is given by

where is the increment function which is the

slope over the interval

The increment function can be written in general form as

where a1, a2, a3,…, an are constants and k1, k2, k3,…, kn are

),( yxfdx

dy

hhyxyy iiii ),,( 1

),,( hyx ii

nnkakakaka ...332211

)...,(

.

.

.

),(

),(

),(

11,122,111,11

22212123

11112

1

hkqhkqhkqyhpxfk

hkqhkqyhpxfk

hkqyhpxfk

yxfk

nnnnninin

ii

ii

ii

160

Where p1, p2,…,pn-1 and q1, q2,…,qn-1 are all constants

n is order number of the RK method

k’s are recurrence functions. Because each k is a functional

evaluation, this recurrence makes RK methods efficient for

computer calculations.

Various types of RK methods can be devised by employing

different number of terms in the increment function as

specified by n.

Once n is chosen, values of a’s, p’s, and q’s are evaluated

by setting general equation equal to terms in a Taylor series

expansion.

First-order RK method (n = 1) is Euler’s method.

Second-order Runge-Kutta

The second-order version of Runge-Kutta methods is

(i)

where

Evaluate values for a1, a2, p1 and q11 by setting equation (i)

equal to a Taylor series expansion to the second-order term

(see Box 25.1 p.703 for details).

Three equations are derived to solve four unknown

constants

hkakayy ii )( 22111

),(

),(

11112

1

hkqyhpxfk

yxfk

ii

ii

2/1

2/1

1

112

12

21

qa

pa

aa

161

Since there are three equations with four unknowns, we

must assume a value of one of the unknowns to determine

the other three.

Suppose that we specify a value for a2, then

There are an infinite number of second-order RK methods

Because we can choose an infinite number of values for a2,

there are an infinite number of second-order RK methods.

Every version would yield exactly the same results if the

solution to ODE were quadratic, linear, or a constant.

However, they yield different results if the solution is more

complicated (typically the case).

Three of the most commonly used RK second-order

methods are:

Heun’s Method with a Single Corrector (a2=1/2)

The Midpoint Method (a2=1)

Raltson’s Method (a2=2/3)

Heun Method with a Single Corrector

a2 is set to a2=1/2

2

111

21

2

1

1

aqp

aa

2111

21

2/1

2/11

aqp

aa

162

Substituting into equation (i)

Where

k1 is the slope at the beginning of the interval and k2 is the

slope at the end of the interval.

The Midpoint Method

a2 is set to a2=1


Where

hkkyy ii )2

1

2

1( 211

),(

),(

12

1

hkyhxfk

yxfk

ii

ii

2

1

0

111

1

qp

a

hkyy ii 21

)2

1,

2

1(

),(

12

1

hkyhxfk

yxfk

ii

ii

163

Ralston’s Method

a2 is set to a2=2/3


Where

4/3

3/1

111

1

qp

a

hkkyy ii )3

2

3

1( 211

)4

3,

4

3(

),(

12

1

hkyhxfk

yxfk

ii

ii

164

Fourth-order Runge-Kutta

This is the most popular of the RK methods

Where

hkkkkyy ii )22(6

1 43211

),(

)2

1,

2

1(

)2

1,

2

1(

),(

34

23

12

1

hkyhxfk

hkyhxfk

hkyhxfk

yxfk

ii

ii

ii

ii

165

Example 1

Integrate using fourth-order Runge-Kutta method:

Using h = 0.5 with y(0) = 2 from x = 0 to 0.5.


Compute the slope at the beginning of the interval:

Use this slope to compute the value of y and slope at

midpoint

Use this slope to calculate another value of y and slope at

the midpoint

yedx

dy x 5.04 8.0

3)2(5.04)2,0(),( )0(8.0

1 efyxfk ii

75.2)25.0(32 )25.0( y

510611.3)75.2(5.04)75.2,25.0( )25.0(8.0

2 efk

877653.2)25.0(510611.32 )25.0( y

446785.3)877653.2(5.04)877653.2,25.0( )25.0(8.0

3 efk

166

Use the this slope to compute the slope at the end of the

interval:

Evaluate the final prediction at the end of the interval

System of Equations

Many practical problems in engineering require the

solution of a system of simultaneous ODE rather than a

single equation

723392.3)5.0(446785.32 )5.0( y

105603.4)723392.3(5.04)723392.3,5.0( )5.0(8.0

4 efk

hkkkkyy ii )22(6

1 43211

751669.3

5.0 )105603.4446785.32510611.323(6

12 )5.0(

y

),...,,,(

.

.

.

),...,,,(

),...,,,(

21

2122

2111

nnn

n

n

yyyxfdx

dy

yyyxfdx

dy

yyyxfdx

dy

167

Solution requires that n initial conditions be known at the

starting value x.

Procedure for Solution Using RK Methods

1. Compute the slopes for all variables at the beginning of the

interval (set of k1’s).

2. Use these slopes to make predictions of the dependent

variable at the midpoint of the interval (y1, y2,…). Use the

midpoint values to compute a first set of slopes at the

midpoint of the interval (set of k2’s)

3. The new set of slopes k2 are used to make another set of

midpoint predictions (y1, y2,…). Use the latest set of

midpoint predictions (y1, y2,…) to compute the second set

of midpoint slopes (set of k3’s)

4. Use the values of k3’s to make predictions at the end of the

interval (y1, y2,…). Use the predictions at the end of the

interval to compute the endpoint slopes (set of k4’s)

5. Combine the k’s into a set of increment functions and

brought back to the beginning to make the final prediction

of (y1, y2,…).

168

Example 2

Solve the following set of differential equations using the 4th

order RK method, assuming that at x = 0, y1 = 4 and y2 = 6.

Integrate to x = 2 with a step size of 0.5


1. Compute the slopes for all variables at the beginning of the

interval (set of k1’s).

2. Use set of k1’s to make predictions of (y1, y2,…) at the

midpoint of the interval. Use these (y1, y2,…) to compute a

first set of slopes at the midpoint of the interval (set of k2’s)

122

1

1.03.04

5.0

yydx

dy

ydx

dy

),,(1.03.04

),,(5.0

212122

2111

yyxfyydx

dy

yyxfydx

dy

ddxdx 4.0

8.1)4(1.0)6(3.04)6,4,0(

2)4(5.0)6,4,0(

22,1

11,1

fk

fk

45.6)25.0)(8.1(62

)0( )25.0(

5.3)25.0)(2(42

)0( )25.0(

2,122

1,111

hkyy

hkyy

169

3. Use the set of k2’s to make another set of midpoint

predictions (y1, y2,…). Use the latest set of (y1, y2,…) to

compute the second set of midpoint slopes (set of k3’s)

4. Use the values of k3’s to make predictions at the end of the

interval (y1, y2,…). Use the predictions at the end of the

interval to compute the endpoint slopes (set of k4’s)

715.1)5.3(1.0)45.6(3.04)45.6,5.3,25.0(

75.1)5.3(5.0)45.6,5.3,25.0(

22,2

11,2

fk

fk

42875.6)2

5.0)(715.1(6

2)0( )25.0(

5625.3)2

5.0)(75.1(4

2)0( )25.0(

2,222

1,211

hkyy

hkyy

715125.1)5625.3(1.0)42875.6(3.04)42875.6,5625.3,25.0(

78125.1)5625.3(5.0)42875.6,5625.3,25.0(

22,3

11,3

fk

fk

857563.6)5.0)(715125.1(6)0( )5.0(

109375.3)5.0)(78125.1(4)0( )5.0(

2,322

1,311

hkyy

hkyy

170

5. Combine the k’s into a set of increment functions and

brought back to the beginning to make the final prediction

of (y1, y2,…).

631794.1)109375.3(1.0)857563.6(3.04)857563.6,109375.3,5.0(

554688.1)109375.3(5.0)857563.6,109375.3,5.0(

22,4

11,4

fk

fk

857670.65.0 )]631794.1()715125.1(2)715.1(2)8.1[(6

16 )5.0(

115234.35.0 )]554688.1()78125.1(2)75.1(2)2[(6

14 )5.0(

2

1

y

y

171

Repeating the procedure for the remaining steps gives the

following results

x y1 y2

0.0 4 6

0.5 3.115234 6.857670

1.0 2.426171 7.632106

1.5 1.889523 8.326886

2.0 1.471577 8.946865

172

Numerical Solutions of Ordinary Differential Equations

Boundary Value Problems

Objectives

Introduction

Shooting Method

Finite Difference Method

Introduction

An ODE is accompanied by auxiliary conditions. These

conditions are used to evaluate the integral that result

during the solution of the equation. An nth order equation

requires n conditions.

If all conditions are specified at the same value of the

independent variable, then we have an initial-value

problem.

If the conditions are specified at different values of the

independent variable, usually at extreme points or

boundaries of a system, then we have a boundary-value

problem.

Initial-value versus boundary-value problems

173

Initial-value problem where all the conditions are specified

at the same value of the independent variable.

Boundary-value problem where the conditions are specified

at different values of the independent variable.

Determination of eigenvalues: Special class of boundary-

value problems that are common in engineering involving

vibrations, elasticity, and other oscillating systems.

174

Two general approaches for solving BVP:

Shooting method

Finite-difference method

Both approaches will be illustrated by an example of heat

balance.

Heat balance problem

Heat balance of a long, thin rod

Rod not insulated along its length and in a steady state

175

Equation describing the problem

Boundary value conditions

Analytical solution

The Shooting Method

Converts the boundary value problem to initial-value

problem.

A trial-and-error approach is then implemented to solve the

initial value approach.

2

2

2

01.0

10

20

0)(

mh

mL

T

TThdx

Td

a

a

200)(

40)0(

2

1

TLT

TT

204523.534523.73 1.01.0 xx eeT

176

For example, the 2nd order equation can be expressed as

two first order ODEs:

An initial value is guessed, say z(0) = 10.

The solution is then obtained by integrating the two 1st

order ODEs simultaneously.

Using a 4th order RK method with a step size of 2:

T(10)=168.3797.

This differs from T(10)=200. Therefore a new guess is

made, z(0)=20 and the computation is performed again:

T(10)=285.8980

Because the original ODE is linear, the two sets of points,

(z, T)1 and (z, T)2, are linearly related, a linear interpolation

formula is used to compute the value of z(0) as

)( aTThdx

dz

zdx

dT

6907.12)3797.168200(3797.1688980.285

102010)0(

z

177

z(0) = 12.6907 is then used to determine the correct

solution.

First shot

z(0) = 10 T(10) = 168.3797

Second shot

z(0) = 20 T(10) = 285.8980

Final exact hit

z(0) = 12.6907 T(10) = 200

178

The Shooting Method

Nonlinear Two-Point Problems.

For a nonlinear problem a better approach involves

recasting it as a roots problem.

Driving this new function, g(z0), to zero provides the

solution.

Finite Differences Methods

The most common alternatives to the shooting method.

200)()(

)(200

)(

00

0

010

zfzg

zf

zfT

179

Finite differences are substituted for the derivatives in the

original equation.

Finite differences equation applies for each of the interior

nodes. The first and last interior nodes, Ti-1 and Ti+1,

respectively, are specified by the boundary conditions.

Thus, a linear equation transformed into a set of

simultaneous algebraic equations.

It will be tridiagonal which can be solved efficiently.

If we use a segment length Δx = 2 m (4 interior nodes)

Thus, a set of simultaneous algebraic equations.

2

11

2

2 2

x

TTT

dx

Td iii

0)(2

2

11

ai

iii TThx

TTT

aiii TxhTTxhT 2

1

2

1 )2(

8.004.2 11 iii TTT

180

which can be solved for

8.200

8.0

8.0

8.40

04.2100

104.210

0104.21

00104.2

4

3

2

1

T

T

T

T

4795.1595382.1247785.939698.65T

T

numerical methods notes

Documents