iterative methods for nonlinear systems of equations: an introduction · 2011-05-31 · iterative...

Iterative methods for nonlinear systems of equations:

an introduction

Laboratori de Càlcul Numèric (LaCàN)Dep. de Matemàtica Aplicada III

Universitat Politècnica de Catalunyawww-lacan.upc.es

2

Contents

§ Problem statement§ Motivation§ Functional iteration§ Method of direct iteration§ Picard’s method§ Newton’s method§ Quasi-Newton methods

3

Problem statement

§ Starting point

§ General form of nonlinear system

(1 nonlinear equation with 1 unknown)

(linear system of order n)

f transforms vectors into vectors

α is a zero of f if

4

Problem statement

In more detail:

In general, all components of f are nonlinear w.r.t. all components of x

5

Problem statement

§ Particular form of nonlinear system

with and

Similar structure to linear system

Can be (trivially) transformed into the general form:

6

Motivation

Mathematical models in engineeringLinear models§ Response of physical system proportional to external actions§ Simple models§ A first approximation to the real behaviour

Examples: linear systems of equations; linear PDEs

¬ Nonlinear models§ No proportionality between actions and response§ More complex models§ More realistic description of real beaviourExamples: nonlinear systems of equations; nonlinear PDEs

7

Functional iteration

§ Analogy with root finding in 1-D:1-D problem n-D problem

§ Consistency: function φ must verify

(zeros of f) (fixed points of φ)

Nonlinear equation(s)

Initial approximation

Iterative scheme

8

Functional iteration

§ Convergence: contractive mapping theorem

Let φ: D D, D a closed subset of R . If there exists λ ∈ [0,1) such that

then:(a) there exists a unique fixed point α of φ in D.

(b) for any initial approximation x in D, the sequence {x } generated by x = φ(x )remains in D and converges linearly to α with constant λ.

n

0k kk+1

9

Method of direct iteration(or successive approximations)

Advantages§ Very simple technique (evaluate f once per iteration)Drawbacks§ Contractivity of φ not guaranteed§ Convergence is typically linear (if it converges!)

Problem in general form

Iteration function

Iteration scheme

10

Method of direct iteration(or successive approximations)

11

Picard’s method(or secant matrix method)

Problem in particular form

If matrix is inversible,

define the iteration function

so the iteration scheme is

Attention: do not invert matrix!

12

Picard’s method(or secant matrix method)

§ Solve one linear system per iteration

§ Matrix A(x) and vector b(x) one iteration behind

If matrix is inversible,

Practical algorithm

13

Picard’s method(or secant matrix method)

Matrix A(x) is a secant matrix

14

Picard’s method(or secant matrix method)

Advantages§ If A(x) has a special structure (e.g. banded SPD),

it can be exploited when solving the linear systems

Drawbacks§ Matrix A(x) may be singular for some x

§ Convergence is typically linear (if it converges!)

§ Computational cost: matrix A(x) and vector b(x) change at every iteration

15

Newton’s method(or Newton-Raphson’s method)

Problem in general form

Correction of non-converged approximation:

First-order Taylor’s series expansion:

16

Newton’s method(or Newton-Raphson’s method)

Jacobian matrix

Involves the computation of derivatives

17

Newton’s method(or Newton-Raphson’s method)

Practical algorithm

Iteration function is

Jacobian matrix not inverted in practice!

18

Newton’s method(or Newton-Raphson’s method)

19

Newton’s method(or Newton-Raphson’s method)

For problem in particular form, the iterative scheme is

similar to iterative schemes for linear systems

The Jacobian matrix does not retain structure of A(x)

20

Newton’s method(or Newton-Raphson’s method)

Advantages§ Convergence is quadratic (for J(α) not singular)

Drawbacks§ Matrix J(x) may be singular for some x

§ Computational cost: at every iteration, (1) compute matrix J(x) and vector f(x) and (2) solve linear system

§ If A(x) has a special structure (e.g. banded SPD), it is lost when computing J(x)

21

Quasi-Newton methods§ Secant method in 1-D

Similar to Newton’s method

with tangent approximated

by secant defined by the last two iterations k-1 and k

22

Quasi-Newton methods

23

Quasi-Newton methods§ Extension to nonlinear systems

Jacobian matrix is approximated by a secant matrix

defined by the last two iterations k-1 and k

n unknowns and only n equations additional conditions on matrix S required

2