introduction to multigrid methods - matematikcentrum · introduction to multigrid methods ......

Introduction to Multigrid Methods

Chapter 7: Elliptic equations and Sparse linear systems

Gustaf Soderlind

Numerical Analysis, Lund University

Textbooks: A Multigrid Tutorial, by William L Briggs. SIAM 1988

A First Course in the Numerical Analysis of Differential Equations, by Arieh Iserles. Cambridge 1996

Matrix-based multigrid: Theory and Applications, by Yair Shapira. Springer 2008

Multi-Grid Methods and Applications, by Wolfgang Hackbusch, 1985

c© Gustaf Soderlind, Numerical Analysis, Mathematical Sciences, Lund University, 2009-2010

Introduction to Multigrid Methods – p. 1/61

1. Overview

Consider simplest linear 2p-BVP (1D Poisson)

y′′ = f(x)

y(0) = α; y(1) = β

Introduce equidistant grid with ∆x = 1/(N + 1)

Discretization

yi+1 − 2yi + yi−1

∆x2= f(xi)

y0 = α; yN+1 = β


Linear system of equations

Tridiagonal N ×N matrix formulation

1

∆x2

−2 1

1 −2 1. . .

1 −2

y1

y2

...

yN

=

f(x1)− α/∆x2

f(x2)...

f(xN)− β/∆x2

In matrix–vector form

T∆xy = f


Elliptic model problem: 2D Poisson equation

∂2u

∂x2+

∂2u

∂y2= f(x, y)

on Ω = [0, 1]× [0, 1] with Dirichlet conditions u = 0 on ∂Ω

Uniform grid xi, yjN,Ni,j=1, mesh width ∆x = ∆y = 1/(N + 1)

Discretization Finite differences with ui,j ≈ u(xi, yj)

ui−1,j − 2ui,j + ui+1,j

∆x2+

ui,j−1 − 2ui,j + ui,j+1

∆y2= f(xi, yj)


Equidistant mesh ∆x = ∆y

ui−1,j + ui,j−1 − 4ui,j + ui,j+1 + ui+1,j

∆x2= f(xi, yj)

Participating approximations and mesh points

xi−1 xi xi+1

yj

yj+1

yj−1


Computational “stencil” for ∆x = ∆y

ui−1,j + ui,j−1 − 4ui,j + ui,j+1 + ui+1,j

∆x2= f(xi, yj)

“Five-point operator”1

1 −4 1

1


The FDM linear system of equations

Lexicographic ordering of unknowns⇒ partitioned system

1

∆x2

T I 0 . . .

I T I

I T I

. . . I

. . . 0 I T

u·,1

u·,2

u·,3

...

u·,N

=

f(x·, y1)

f(x·, y2)

f(x·, y3)...

f(x·, yN )

with Toeplitz matrix T = tridiag(1 −4 1)

The system is N 2 ×N 2, hence large and very sparse


3D Poisson equation. The “curse” of dimension

Partitioned system

1

∆x2

T I 0 . . . I

I T I. . .

I T I

I. . . I

. . . 0 I T

u1,1,1

...

...

...

uN,N,N

=

f1,1,1

...

...

...

fN,N,N

with Toeplitz matrix T = tridiag(1 −6 1)

The system is N 3 ×N 3, hence extremely large and sparse


Galerkin method (Finite Element Method)

1. Basis functions ϕi

2. Approximate u =∑

cjϕj

3. Determine cj from∑

cj

∫

∇ϕi · ∇ϕj =∫

fϕi

The cj are determined by the linear system

Kc = F

The stiffness matrix K has similar structure to FDM matrix

Stiffness matrix elements kij =∫

∇ϕi · ∇ϕj = a(ϕi, ϕj)

Right-hand side Fi =∫

ϕif = 〈ϕi, f〉 =∑

fj〈ϕi, ϕj〉


The FEM mesh. Domain triangulation

Piecewise linear basis ϕj with triangulation mesh

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Same number of nodes too


Iterative methods

All discretizations lead to very large linear systems

Ty = f

typically having millions of equations

Matrix factorization methods are out of the question! Useiterative methods instead!

Explicit iterative methods ym+1 = Bym + c

Implicit iterative methods Dym+1 = Bym + c


What are multigrid methods?

Multigrid methods are iterative methods that use the factthat the origin of the linear system is some discretization,and that the grid properties affect the convergence rate

There is a relation to Fourier analysis as it turns out thatmesh width (inverse spatial frequency) is a key factorgoverning convergence

The methods are called multigrid, because the iteration willalternate between several different grids in order to speedup convergence


2. Iterative methods for linear systems

Given a linear system Au = f construct sequence um → u

Then

Aum = f + rm

Au = f

Definitions

1. The error is defined by em = um − u

2. The residual is defined by rm = Aum − f

3. Relation via the error–residual equation Aem = rm


The basic iterative methods

There are four different basic iterative methods

1. The Jacobi method

2. The Gauss–Seidel method

3. The Successive Overrelaxation (SOR) method

4. The Symmetric SOR method

Advanced methods include the Conjugate Gradient (CG)method; the Generalized Minimum Residiual (GMRES)method; and various forms of Multigrid (MG) methods


The Jacobi method

Splitting Write Au = f as (D − L− U)u = f with

D diagonal; L lower triangular; U upper triangular

Du = (L + U)u + f

u = D−1(L + U)u + D−1f

Jacobi method Use fixed point iteration

um+1 = D−1(L + U)um + D−1f


The Jacobi method. Implementation

Given um, calculate residual rm, and update according to

rm ← Aum − f

um+1 ← um −D−1rm

Note The scheme implies that each single, scalarequation is solved independently of the other equations

It can be directly used on massively parallel computers


Relation to fixed point iteration

From actual implementation

um+1 = um −D−1((D − L− U)um − f) = D−1(L + U)um + D−1f

For analysis purposes this is written

um+1 = PJum + D−1f

where the Jacobi iteration matrix is

PJ = D−1(L + U)


1D Poisson + Jacobi method

If A = T∆x then

PJ = D−1(L + U) = tridiag(1/2 0 1/2)

andum+1 = PJum + D−1f

For the exact solution

u = PJu + D−1f

Error recursion em+1 = PJem


1D Poisson + Jacobi. Convergence

Error recursion em+1 = PJem

Convergence em → 0

1. Necessary condition ρ[PJ ] < 1

2. Sufficient condition ‖PJ‖ < 1

Definition Spectral radius ρ[A] = maxk |λk[A]|

For 1D Poisson, we need to calculate the eigenvalues ofthe Jacobi iteration matrix PJ = tridiag(1/2 0 1/2)


Eigenvalues of Toeplitz matrices

PJ = tridiag(1/2 0 1/2) = S/2

Su =

0 1 0 . . .

1 0 1

1 0 1. . . 1

. . . 0 1 0

u1

u2

...

uN

= λu

Find the eigenvalues of S!


Eigenvalues of symmetric Toeplitz matrix S

Consider the nth equation of Su = λu

un+1 + un−1 = λun

Linear difference equation with boundary values

u0 = 0; uN+1 = 0

Characteristic equation

z2 − λz + 1 = 0


Eigenvalues of S. . .

Roots of z2 − λz + 1 = 0 are z and 1/z (product 1)

General solution un = αzn + βz−n

Boundary condition u0 = 0 = α + β ⇒

Solution un = α(zn − z−n)

Boundary conditionuN+1 = 0 = α(zN+1 − z−(N+1)) ⇒ z2(N+1) = 1

Roots zk = exp( kπi

N + 1

)

k = 1 : N


Eigenvalues. . .

Sum of the roots of z2 − λz + 1 = 0 are

λk = zk + 1/zk ⇒

λk[S] = exp( kπi

N + 1

)

+ exp(

−kπi

N + 1

)

= 2 coskπ

N + 1

Hence the eigenvalues of PJ = tridiag(1/2 0 1/2) are

λk[PJ ] = coskπ

N + 1∈ (−1, 1)


Eigenvalue locationsEigenvalues get very close to ±1 for large N (N = 31)

0 0.5 1 1.5 2 2.5 3−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1Eigenvalues cos(k*pi*dx) of Jacobi matrix

k*pi*dx, for k=1:N, with N=31

Eig

enva

lues


Peacock plot. Eigenvalues and unit circle, N = 31

Eigenvalues λk = cos kπ∆x are projections on real axis

In terms of Chebyshev zeros, T ′

N+1(λk) = 0 and UN(λk) = 0


Slow, slower, slowest

With λk[PJ ] = cos kπ/(N + 1) we have

ρ[PJ ] = λ1[PJ ] = −λN [PJ ]≈ 1−π2∆x2

2+ O(∆x4)

So Jacobi’s method will converge as ρ[PJ ] < 1, butconvergence will be painfully slow!

Example N = 99 ⇒ ρ[PJ ] ≈ 0.9995


Convergence history

Plot of L2 error as a function of iteration number (N = 99)

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 1000010

−3

10−2

10−1

100

Convergence history

Iteration number

L2 E

rror n

orm


Unit error reduction requires O(N2) iterations

Plot of L2 error after N 2 iterations (N = 15, 31, 63, 127)

0 2000 4000 6000 8000 10000 12000 14000 16000 1800010

−3

10−2

10−1

100

Convergence history: N=15, 31, 63, 127

Iteration number

L2 E

rror n

orm


3. Power iteration

The fixed point iteration ym+1 = Aym is also called poweriteration

It implies ym = Amy0 Note powers of A!

Assume distinct eigenvalues Axk = λkxk with |λ1| > |λk|,

for k = 2, . . . , N . Let y0 = Σαkxk. Then

ym

λm1

= α1x1 +

N∑

k=2

αkxk

(

λk

λ1

)m

→ α1x1

So the vector ym gets aligned with eigenvector x1


The power method

Once we have a good approximation to x1 we obtain thecorresponding eigenvalue from the Rayleigh quotient

λ1 ≈〈ym, Aym〉

〈ym, ym〉

The power method Take y0 and iterate until convergence:

ym := ym/‖ym‖2

ym+1 := Aym

σm := 〈ym, ym+1〉 → λ1


Why study the power method?

The error recursion em+1 = PJem is a power iteration

Let PJv = λv, with |λ| = ρ[PJ ]. Then any convergent fixedpoint type iteration will have the following properties:

1. The error will initially decay relatively fast

2. Convergence then slows to be governed by ρ[PJ ]

3. The error em will become aligned with the eigenvector v


What is the convergence to x1 like?

As

ym

λm1

= α1x1 +

N∑

k=2

αkxk

(

λk

λ1

)m

→ α1x1

we see that the error decay is exponential and governed bythe ratio of the largest to the next largest eigenvalue

The decay may be very slow if eigenvalue gap is small

For PJ the gap is very small, with ratio

λ2

λ1=

cos 2π∆x

cos π∆x≈

2− 4π2∆x2

2− π2∆x2≈ 1−

3π2∆x2

2


What is the convergence to max |λ| like?

Non-normal matrices linear convergence

Normal matrices quadratic convergence

Example

A =

12 2 2.05

2 9 1

1.95 1 7

B =

12 2 2

2 9 1

2 1 7

Approximate eigenvalues λ ∈ 13.74 8.00 6.26, but thenormal matrix has orthogonal eigenvectors


Computation. Non-normal vs. normal matrix

Convergence history: Error in λmax vs. iteration number

0 10 20 3010

−15

10−10

10−5

100

0 10 20 3010

−15

10−10

10−5

100


Good news, bad news

Normal differential operator⇒ normal matrix

ut = uxx

ut = ∇ · (p∇u)

ut + ux = 0

ρt +∇ · (ρv) = 0

Non-normal differential operators

ut = ux +1

Peuxx


4. Eigenvectors of PJ

From eigenvalue problem of PJ = tridiag(1/2 0 1/2)

Recall un = znk − z−n

k and zk = exp( kπiN+1

), then

ukn = exp

( knπi

N + 1

)

− exp(

−knπi

N + 1

)

∼ sin( knπ

N + 1

)

Same eigenvectors as for T∆x


Eigenvectors of PJ at N = 31

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−1

−0.5

0

0.5

1Lowest mode, sin(n*pi*dx), on fine grid

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−1

−0.5

0

0.5

1Highest mode, sin(n*N*pi*dx), on fine grid


Eigenvectors of PJ at N = 31

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−1

−0.5

0

0.5


0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−1

−0.5

0

0.5



From fine grid to coarse – now you see it

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−1

−0.5

0

0.5


0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−1

−0.5

0

0.5



From fine grid to coarse – now you don’t

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−1

−0.5

0

0.5

1Lowest mode, sin(n*pi*dx), on coarse grid

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−1

−0.5

0

0.5

1Highest mode, sin(n*N*pi*dx), on coarse grid


And yet. . . Nyquist Sampling Theorem

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−1

−0.5

0

0.5

1Lowest mode, sin(n*pi*dx), on coarse grid

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−1

−0.5

0

0.5

1Highest mode, sin(n*N*pi*dx), on coarse grid


High frequency modes and grid density

What is a high frequency is a grid property

Nyquist Sampling Theorem On a grid with N interiorpoints the highest frequency that can be represented isun = sin Nπn/(N + 1)

Note

sinNπn

N + 1= sin

(

πn−πn

N + 1

)

= (−1)n+1 sin( πn

N + 1

)

Highest frequency is (−1)n modulation of lowest frequency!


Downsampling and aliasing

Aliasing If we take only every second sample, the highestfrequency un = sin(Nπn/(N + 1)) “maps to” the function

vl = sin(2l + 1)π

N + 1∼ sin πx

or to the function

wl = − sin2lπ

N + 1∼ − sin πx

The former highest frequency maps to the lowest!

In general, if the Nyquist frequency of the coarse grid is ω

then a frequency ω + δω < 2ω “folds back” down to ω − δω


Jacobi iteration demonstration

Error vector sequence (50 iterations) at N = 19

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−0.01

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07Error reduction in 50 iterations

Err

or

x


5. The Lapacian and the 5-point FD operator

∂2u

∂x2+

∂2u

∂y2= λu

Computational domain Ω = [0, 1]× [0, 1] (unit square)Dirichlet conditions u(x, y) = 0 on boundary

Variable separation u(x, y) := v(x)w(y) implies

vxxw + vwyy = λvw

orvxx

v+

wyy

w= λ


Eigenvalues and eigenfunctions of the Laplacian

Let v′′ = κkv and w′′ = µmw, then

λk,m = κk + µm

uk,m(x, y) = vk(x) · wm(y)

Modulation Theorem The Laplacian on the unit squarehas eigenvalues and eigenvectors, given by

λk,m = (k2 + m2)π2 k,m ∈ Z+

uk,m(x, y) = sin kπx sin mπy

Analogous modulation result in 3DIntroduction to Multigrid Methods – p. 46/61

The 5-point finite difference Laplacian

Equidistant mesh width ∆x = ∆y = 1/(N + 1)

Discretization Finite differences with ui,j ≈ u(xi, yj)

ui−1,j − 2ui,j + ui+1,j

∆x2+

ui,j−1 − 2ui,j + ui,j+1

∆x2= λ∆x

k,m · ui,j

Discrete variable separation uk,mi,j := vk

i · wmj implies

vki−1 − 2vk

i + vki+1

vki ∆x2

+wm

j−1 − 2wmj + wm

j+1

wmj ∆x2

= κ∆xk + µ∆x

m


Discrete eigenvalues and eigenfunctions

Discrete Modulation Theorem The five-point finitedifference Laplacian on the unit square has N 2 discreteeigenvalues and eigenvectors, given by

λk,m = −4(N + 1)2 ·(

sin2 kπ

2(N + 1)+ sin2 mπ

2(N + 1)

)

uk,mi,j = sin

kπi

N + 1· sin

mπj

N + 1; i, j, k,m = 1 : N

Again analogous results hold in 3DThus 1D theory gives most of the necessary insight


6. Advanced iteration methods

As Jacobi iteration is so slow, what can be done to speedup convergence?

Damped Jacobi (under-relaxation) Use a damping factorω and iterate according to

rm ← Aum − f

um+1 ← um − ωD−1rm

The iteration stops only when rm = 0, so it still solves theproblem Au = f


Damped Jacobi method

Damped Jacobi is equivalent to

um+1 =(

(1− ω)I + ωD−1(L + U))

um + ωD−1f

so the iteration has damped Jacobi iteration matrix

Pω = (1− ω)I + ωPJ

Theorem The eigenvalues of Pω are

λk[Pω] = 1− ω + ω coskπ

N + 1


Eigenvalue locations of damped Jacobi matrix

Eigenvalues for N = 31 and ω = 1, 0.5, 0.3, 0.2

0 0.5 1 1.5 2 2.5 3−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

k*pi*dx, for k=1:N, with N=31

Eig

enva

lues

Eigenvalues of damped Jacobi matrix


Eigenvalues of damped Jacobi matrix

By choosing ω we can (at least) eliminate large negativeeigenvalues; what happens to the largest eigenvalue?

Largest eigenvalue becomes

λ1[Pω] = 1− ω + ω cosπ

N + 1= 1− 2ω sin2 π∆x

2

So

λ1[Pω] ≈ 1−ωπ2∆x2

2

(Ouch, this is even closer to 1 for ω < 1. . . )


The Gauss–Seidel method

Splitting Write Au = f as (D − L− U)u = f with

D diagonal; L lower triangular; U upper triangular

(D − L)u = Uu + f

u = (D − L)−1Uu + (D − L)−1f

Gauss–Seidel method Use fixed point iteration

um+1 = (D − L)−1Uum + (D − L)−1f


The Gauss–Seidel method. Implementation

Given um, calculate residual rm, and update according to

rm ← Aum − f

um+1 ← um − (D − L)−1rm

Note The scheme implies that each equation is solved insequence of scalar equations. It cannot be parallelized in astraightforward way

The Gauss–Seidel iteration matrix is

PGS = (D − L)−1U


1D Poisson + Gauss–Seidel method

Major features compared to Jacobi iterations are

Regularizing iteration – no high-frequency error

Faster convergence, although still slow

Eigenvalues of PGS satisfy generalized eigenvalue problemUu = λ(D − L)u, according to Briggs:

λk[PGS ] = cos2 kπ

N + 1≈ 1− k2π2∆x2

but this result is incorrect!


Gauss–Seidel eigenvalues and eigenvectors

[N/2] eigenvalues of PGS are

λk[PGS ] = cos2 kπ

N + 1≈ 1− k2π2∆x2

and the remaining eigenvalues are zero!

Note Eigenvectors of PGS and T∆x do not coincide!

Therefore we need other ways of studying convergence


Digital filters and Bode diagrams

Study error recursion

em+1 = PGSem

one frequency at a time! Study “output” y = PGSu whenu = sin kπ∆x, an eigenvector of T∆x, and compute the L2

(root-mean-square) “attenuation”

‖y‖∆x

‖u‖∆x

=‖PGSu‖∆x

‖u‖∆x

=‖PGSu‖2‖u‖2

Frequency response Plot vs frequency ω = kπ∆x ∈ (0, π)

or as a function of wave number k = 1 : N (Bode diagram)Introduction to Multigrid Methods – p. 57/61

Bode diagram. Gauss–Seidel and Jacobi

Linear attenuation vs frequency: Gauss–Seidel and Jacobi

0 0.5 1 1.5 2 2.5 30

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Frequency (wave number)

Atte

nuat

ion

(line

ar)Frequency response: Guass−Seidel (b) and Jacobi (r)


GS is a low-pass filter and J a band-stop filter

Logarithmic attenuation (dB): Gauss–Seidel and Jacobi

0 0.5 1 1.5 2 2.5 3−20

−18

−16

−14

−12

−10

−8

−6

−4

−2

0

Frequency (wave number)

Atte

nuat

ion

(dB

)Frequency response: Guass−Seidel (b) and Jacobi (r)


Gauss–Seidel as LP filter

Gauss–Seidel has good damping of all frequencies exceptlow frequencies which are only weakly suppressed

High frequency damping is better than 0.5, meaning thathigh frequency modes decay faster than 0.5m

Thus Gauss–Seidel is smoothing (LP filter) – the errorbecomes a smoother function as the iterations pass

As Jacobi does not damp high frequencies it is not asmoother; it blocks a narrow band of mid-frequencies


Gauss–Seidel iteration demonstration

Error vector sequence (50 iterations) at N = 19

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−0.04

−0.02

0

0.02

0.04

0.06

0.08


introduction to multigrid methods - matematikcentrum · introduction to multigrid methods ......

Documents