math 3795 lecture 13. numerical solution of nonlinear ...zeng/teaching/math3620/... · math 3795...

MATH 3795Lecture 13. Numerical Solution of Nonlinear

Equations in RN .

Dmitriy Leykekhman

Fall 2008

GoalsI Learn about different methods for the solution of F (x) = 0, their

advantages and disadvantages.

I Convergence rates.

I MATLAB’s fsolve

D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares – 1

Nonlinear Equations.

I Goal: Given a function f : RN → RN we want to find x∗ ∈ RN

such that F (x∗) = 0.

I DefinitionA point x∗ with F (x∗) = 0 is called a root of F or zero of F .

I We want to extend the methods from the last lecture, like Newton’smethod and Secant Method to find roots to vector valued problems.


Convergence of Sequences.Let {xk} be a sequence of vectors in RN and let ‖ · ‖ denote a vectornorm.

1. The sequence is called q-linearly convergent if there existsc ∈ (0, 1) and k̂ ∈ N such that

‖xk+1 − x∗‖ ≤ c‖xk − x∗‖ for all k ≥ k̂.

2. The sequence is called q-superlinearly convergent if there exists asequence {ck} with ck > 0 and limk→∞ ck = 0 such that

‖xk+1 − x∗‖ ≤ ck‖xk − x∗‖

or, equivalently, if

limk→∞

‖xk+1 − x∗‖‖xk − x∗‖

= 0.

3. The sequence is called q-quadratically convergent to x∗ iflimk→∞ xk = x∗ and if there exists c > 0 and k̂ ∈ N such that

‖xk+1 − x∗‖ ≤ c‖xk − x∗‖2 for all k ≥ k̂.


Taylor Expansion.

I Given a differentiable function F , we define its Jacobian

F ′(x) = JF (x) =

∂F1∂x1

(x) . . . ∂F1∂xn

(x)...

...∂Fn

∂x1(x) . . . ∂Fn

∂xn(x)

I If F is continuously differentiable around x0, then

F (x) ≈ F (x0) + F ′(x0)(x− x0)


Taylor Expansion.

ExampleLet F : R2 → R2 be given by

F (x) =(

x21 + x2

2 − 2ex1−1 + x3

2 − 2

)Then its Jacobian is

F ′(x) =(

2x1 2x2

ex1−1 3x22

)

If x0 =(

00

), then for x =

(x1

x2

)close to

(00

)

F (x) ≈(

−2e−1 − 2

)+(

0 0e−1 0

)(x1

x2

)=(

−2e−1(1 + x1)− 2

)


Newton’s Method.

I Suppose we are given an approximation x0 of a root x∗ of F .

I Taylor approximation of F around x0 gives

F (x∗) = F (x0 + (x∗ − x0)) ≈ F (x0) + F ′(x0)(x∗ − x0).

I We useM(x) = F (x0) + F ′(x0)(x− x0)

as a model for F .

I The root of M(x) i.e. the solution of linear system

F ′(x0)(x− x0) = −F (x0)

is used as an approximation of the root of F .

I Write the previous identity as

F ′(x0)s0 = −F (x0) step (correction)

x1 = x0 + s0.


Newton’s Method.

Input: Initial values x(0), tolerance tol,maximum number of iterations maxit

Output: approximation of the root

1 For k = 0:maxit do2 Compute F’(x(k))s(k) = -F(x(k)). (LU-decomposition)3 Compute x(k+1) = x(k) + s(k).4 Check for truncation5 End


Newton’s Method.

ExampleConsider

F (x) =(

x21 + x2

2 − 2ex1−1 + x3

2 − 2

)with starting point x0 = (1.5, 2)T and stopping criteria‖F (xk)‖2 ≤ 1e− 10, the Newton’s method gives the computed solutionx = (1.0000, 1.0000)T and the historyk ‖xk‖2 ‖F (xk)‖2 ‖sk‖20 2.500000e+ 000 8.750168e+ 000 8.805454e− 0011 1.665941e+ 000 2.073196e+ 000 3.234875e− 0012 1.450739e+ 000 4.127937e− 001 1.606253e− 0013 1.423306e+ 000 6.177196e− 002 2.206725e− 0024 1.414386e+ 000 1.401191e− 003 6.087256e− 0045 1.414214e+ 000 9.730653e− 007 3.964708e− 0076 1.414214e+ 000 4.415589e− 013 0.000000e+ 000


Convergence of Newton’s Method.

TheoremLet D ∈ RN be an open set and let F : D → RN be differentiable on Dwith Lipschitz continuous derivative, i.e. let L > 0 be such thtat

‖F ′(y)− F ′(x)‖ ≤ L‖y − x‖ ∀x, y ∈ D.

If x∗ ∈ D is a root and if F ′(x∗) is nonsingular, then there exists anε > 0 such that Newton’s method with starting point x0 with‖x0 − x∗‖ < 0 generates iterates xk which converge to x∗,

limk→∞

xk = x∗,

and which obey‖xk+1 − x∗‖ ≤ c‖xk − x∗‖2

for all k and some positive constant c.

Newton’s method is locally q-quadratically convergent (under theassumptions stated in the theorem).


Stopping Criteria.

We have discussed iterative methods which generate sequences xk with(under suitable conditions) limk→∞ xk = x∗. When do we stop theiteration?

I For a given tolerance tola > 0 we want to find xk such that‖xk − x∗‖ < tola, stop if absolute error is small; or‖xk − x∗‖ < tolr‖x∗‖, stop if relative error is small.

I For some tk ∈ [0, 1]

‖F (xk)‖ = ‖F (xk)− F (x∗)‖

≥ 12‖(F ′(x∗))−1‖−1‖xk − x∗‖, if xk is suff. close to x∗

Hence, if ‖F (xk)‖ < tolf , and if xk is sufficiently close to x∗, then

‖xk − x∗‖ < 2tolf‖(F ′(x∗))−1‖.

I Limit maximum number of iterations.


Variations of Newton’s Method.Recall Newton’s Method:Input: Initial values x0,tolerance tol, maximum number of iterations maxit


1. For k = 0, . . . ,maxit do

2. Compute sk = −f(xk)/f ′(xk).

3. xk+1 = xk + sk.

4. Check for truncation

5. End

I Requires the evaluation of derivatives and solution of a linear system.

I Linear system solves are done using LU decomposition (cost 2/3n3

flops for each matrix factorization).

I If Jacobian/derivative evaluations are expensive or difficult tocompute, or if the LU factorization of the derivative is expensive ,the following variations are useful

I Finite difference Newton method and Secant method.


Finite Difference Newton Method.Recall

∂Fi(x)∂xj

= limh→0

Fi(x1, . . . , xj−1, xj + h, xj+1, . . . , xn)− Fi(x)h

.

Thus, for all i = 1, . . . , n∂F1(x)

∂xj

...∂Fn(x)

∂xj

= limh→0

F (x1, . . . , xj−1, xj + h, xj+1, . . . , xn)− F (x)h

.

The idea is to replace∂F1(x)

∂xj

...∂Fn(x)

∂xj

≈ 1hj

(F (x1, . . . , xj−1, xj + h, xj+1, . . . , xn)− F (x)

).

Good choice for the step size hj = ε‖x‖.


Finite Difference Newton Method.Recall

∂Fi(x)∂xj

= limh→0

Fi(x1, . . . , xj−1, xj + h, xj+1, . . . , xn)− Fi(x)h

.

Thus, for all i = 1, . . . , n∂F1(x)

∂xj

...∂Fn(x)

∂xj

= limh→0

F (x1, . . . , xj−1, xj + h, xj+1, . . . , xn)− F (x)h

.

The idea is to replace∂F1(x)

∂xj

...∂Fn(x)

∂xj

≈ 1hj

(F (x1, . . . , xj−1, xj + h, xj+1, . . . , xn)− F (x)

).

Good choice for the step size hj = ε‖x‖.D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares – 12

Finite Difference Newton Method.

Input: Initial values x0,tolerance tol, maximum number of iterations maxit


1. For k = 0, . . . ,maxit do

2. Compute finite difference approximation of F ′(xk)i.e., compute matrix B with columns B of F ′(xk)

Bej =1hj

(F (x1, . . . , xj−1, xj + h, xj+1, . . . , xn)− F (x)

)3. Factor B

4. Solve Bsk = −F (xk)5. Compute xk+1 = xk + sk

6. Check for truncation

7. End


Secant Method.

I Recall that in the secant method we replace the derivative f ′(xk+1)by f(xk)−f(xk+1)

xk−xk+1.

I If we set bk+1 = f(xk)−f(xk+1)xk−xk+1

then bk+1 satisfies

bk+1(xk+1 − xk) = f(xk+1)− f(xk)

which is called the secant equation

I The next iterate is given by

xk+2 = xk+1 + sk+1,

where sk+1 is the solution of bk+1sk+1 = −f(xk+1).


Secant Method.

I Try to extend this to the problem of finding a root of F : Rn → Rn.

I Given two iterates xk, xk+1 ∈ Rn we try to find a nonsingular matrixBk+1 ∈ Rn×n which satisfies the so-called secant equation

Bk+1(xk+1 − xk) = F (xk+1)− F (xk).

I Then we compute the new iterate as follows

Solve Bk+1sk+1 = −F (xk+1),xk+2 = xk+1 + sk+1.


Secant Method.

There is a problem with this approach.

I If n = 1 the the secant equation

bk+1(xk+1 − xk) = f(xk+1)− f(xk)

has a unique solution.

I If n > 1, the we need to determine n2 entries of Bk+1 from nequations

Bk+1(xk+1 − xk) = F (xk+1)− F (xk).

There is no unique solution.

I For example, for n = 2, xk+1 − xk = (1, 1)T andF (xk+1)− F (xk) = (1, 2), then the matrices(

1 00 2

),

(0 12 0

)satisfy secant equation.


Secant Method.I Therefore we chose Bk+1 ∈ Rn×n as the solution of

minB(xk+1−xk)=F (xk+1)−F (xk)

‖B −Bk‖F .

I Motivation: Bk+1 should satisfy the secant equation

B(xk+1 − xk) = F (xk+1)− F (xk)

and Bk+1 should be as close to the old matrix Bk as possible topreserve as much information contained in Bk as possible.

I Common notation

sk = xk+1 − xk and yk = F (xk+1)− F (xk).

With these definition the previous minimization problem becomes

mins:t: Bsk=yk

‖B −Bk‖F .

Unique solution

Bk+1 = Bk +(yk −Bksk)sT

k

sTk sk

Broyden update.


Broyden’s Method.

Input: Initial values x0,tolerance tol, maximum number of iterations maxit


1 For k = 0, . . . ,maxit do2 Solve Bksk = −F (xk) for sk

3 Set xk+1 = xk + sk.4 Evaluate F (xk+1)5 Check for truncation6 Set yk = F (xk+1)− F (xk)7 Set Bk+1 = Bk + (yk−Bksk)sT

k

sTk sk

.

8 End


Broyden’s Method.

I Note that due to the definition of yk and sk we have that

Bk+1 = Bk +(yk −Bksk)sT

k

sTk sk

= Bk +F (xk+1)sT

k

sTk sk

I To start Broyden’s method we need an initial guess x0 for the rootx∗ and an initial matrix B0 ∈ Rn×n. In practice one often chooses

B0 = F ′(x0) or B0 = γI;

where γ is a suitable scalar. Other choices, for example finitedifference approximations to F ′(x0) or choices based on the specificstructure of F ′ are also used.


Broyden’s Method.

ExampleConsider

F (x) =(

x21 + x2

2 − 2ex1−1 + x3

2 − 2

)with starting point

x0 =(

1.52

)and B0 = F ′(x0),

and stopping criteria ‖F (xk)‖2 ≤ 1e− 10.


Broyden’s Method.

the Broyden’s method produces

k ‖xk‖2 ‖F (xk)‖2 ‖sk‖20 2.500000e+ 000 8.750168e+ 000 8.805454e− 0011 1.665941e+ 000 2.073196e+ 000 1.922038e− 0012 1.476513e+ 000 8.734179e− 001 1.321894e− 0013 1.410326e+ 000 3.812507e− 001 1.555213e− 0014 1.417633e+ 000 1.586346e− 001 9.620188e− 0025 1.423860e+ 000 4.298504e− 002 1.043037e− 0026 1.415846e+ 000 4.681398e− 003 2.583147e− 0037 1.414375e+ 000 6.074087e− 004 6.288185e− 0048 1.414212e+ 000 4.051447e− 006 1.805771e− 0069 1.414214e+ 000 2.724111e− 008 1.154246e− 008

10 1.414214e+ 000 1.182169e− 011 0.000000e+ 000


MATLAB’s fsolve.

Similar to a build-in function fzero, Optimization Toolbox has a functionfsolve that tries to solve a system of nonlinear equations

F (x) = 0.

Warning: You need this Toolbox in order to use this function.

Syntax:

x = fsolve(fun,x0)x = fsolve(fun,x0,options)x = fsolve(problem)[x,fval] = fsolve(fun,x0)[x,fval,exitflag] = fsolve(...)[x,fval,exitflag,output] = fsolve(...)[x,fval,exitflag,output,jacobian] = fsolve(...)


MATLAB’s fsolve.

x = fsolve(fun,x0)

starts at x0 and tries to solve the equations described in fun.

[x,fval] = fsolve(fun,x0)

returns the value of the objective function fun at the solution x.


MATLAB’s fsolve.

Thus in our example with

F (x) =(

x21 + x2

2 − 2ex1−1 + x3

2 − 2

)and x0 =

(1.52

)[x,fval] = fsolve(’ex1’,[1.5 2])

produces

x = 0.999999999999919 1.000000000000164

fval = 1.0e-012 *

0.1652011860642230.410338429901458


MATLAB’s fsolve.

ExampleLet’s say you want to solve

2x1 − x2 = e−x1

−x1 + 2x2 = e−x2.

In other words we want to solve nonlinear system

2x1 − x2 − e−x1 = 0

−x1 + 2x2 − e−x2= 0.


MATLAB’s fsolve.First we create a function in a m-file.

function varargout = ex3(x)%%% [F] = ex3(x) returns the function value in F% [F,Jac] = ex3(x) returns the function value in F and% the Jacobian in Jac%

% return the function valuevarargout{1} = [ 2*x(1)-x(2)-exp(-x(1));

-x(1) + 2*x(2) - exp(-x(2))];

if( nargout > 1 )% return the Jacobian as the second argumentvarargout{2} = [ 2+exp(-x(1)) -1;

-1 2+exp(-x(2))];end


MATLAB’s fsolve.

Now calling

[x,fval] = fsolve(’ex3’,[-5 -5])

produces

x =

0.5671430313973570.567143031397357

fval =

1.0e-006 *

-0.405909605705190-0.405909605705190


MATLAB’s fsolve.We got 6 digits of accuracy. If we need more we can use

options = optimset(’TolFun’,1e-10)

then calling

[x,fval] = fsolve(’ex3’,[-5 -5],options)

produces

x =

0.5671432904097720.567143290409772

fval =

1.0e-013 *

-0.179856129989275-0.179856129989275


MATLAB’s fsolve.

You could learn more about fsolve by typing

help fsolve

or looking at the function on mathworks website fsolveSimilarly you learn there more about the options by typing

help optimset

or just

optimset


http://www.mathworks.com/access/helpdesk/help/toolbox/optim/index.html?/access/helpdesk/help/toolbox/optim/ug/fsolve.html&http://www.google.com/search?hl=en&q=matlab+fsolve+mathwork

math 3795 lecture 13. numerical solution of nonlinear ...zeng/teaching/math3620/... · math 3795...

Documents