1

Iterative Solvers for Linear Systems of Equations

Presented by:Kaveh Rahnema

rahnema@in.tum.de

Supervisor:Dr. Stefan Zimmerzimmer@in.tum.de

2

Outline

1. Short explanation of the system

2. Jacobi method

3. Eigenvalue analysis of the Jacobi method

4. The Multigrid Method

3

Short explanation of the system

n

abhfxf

h

uuujj

jjj

,)(22

11

fuu 2

To solve this equation numerically we need to descretize it and bring it into finite difference notation

fAu

f

f

f

f

u

u

u

u

h

nn

1

3

2

1

1

3

2

1

2

21

121

121

12

1

4

Jacobi method

The system can be written as

fDuULDu

fuULDufuULD

ULDA

11 )(

)()(

Where D is the Diagonal , L is the lower triangle and U is the upper triangle part of the matrix A

If we define the Jacobi iteration matrix as

)(1 ULDPJ

Then the Jacobi method appears in matrix form as

fDuPu J

1)0()1(

5

Modified Jacobi

The Jacobi method can be modified as

)()1( 1)0()0()1( fDuPwuwu J

Where w is a damping factor to help Jacobi converge

faster to the solution

Choosing w = 1 we get the original Jacobi method

Now that we have the formulation of the Jacobi how should we use it to solve the system?

We start with an initial guess (sometimes might be even a bad guess) and solve the system iteratively using the results from the last iteration

to perform the current step until our stopping criteria is satisfied

For stopping criteria we can use different schemes, one might use the difference between 2 iterations , residual etc.

6

Eigenvalue analysis of the Jacobi method

Jww wPIwPfwDuPu )1(,1)0()1(

The Jacobi formulation can also be written as

It follows that the eigenvalues of Pw and A are related by

21

121

121

12

2

wIPw

)(2

1)( Aw

Pw

And we have

7

Eigenvalue analysis of the Jacobi method

)2(sin21)(

11,)2(sin4)(

2

2

N

kwP

NkN

kA

wk

k

NjN

jkw jk 0,)sin(,

Now the problem becomes finding the eigenvalues of A

And A and P have the same eigenvectors

We can expand the error of the initial guess using the eigenvalues of A

Rcwce k

N

kkk

,

1

1

)0(

8

The error after n iterations is given by

1

1

1

1

)0()( )(N

k

N

kkw

n

kkk

n

wk

n

w

n wPcwPcePe

As we see after n iterations the error is only reduced by a factor but the fourier modes don‘t mix

and the condition for Modified Jacobi to converge is

101)( wPwk

In the next slide you see the plot of damping factor of Pw for fourier modes for 4 different w

Eigenvalue analysis of the Jacobi method

9

Frequency dependent damping

10

Example solved by Jacobi method

0uHere as an example we solve the equation

In the [0,1] domain with boundary condition

0)1()0( uu

To solve this system we started with a random initial guess, which might not be used often but for some reasons is a good guess for this

presentation

As you will see in the next slide the Jacobi method works good to damp the high frequencies but damps the low frequencies slow,

This makes solving systems of equations with Jacobi hard, and one may look for other better solvers or techniques to solve the equations faster

11

Modified Jacobi Damping Clip

100

90

80

70

60

50

40

30

20

10

00 20 40 60 80 100 120

Grid Points

Val

ue a

t G

rid P

oint

s

SLO

W

12

Example solved by Jacobi method

As said before the high frequencies damp fast but the low frequencies remain and damp slow

This gets even worse going to finer grid points

What shall we do?!

Shall we wait for a long time till we get the result when we have a large number of grid

points?!!

13

The Multigrid Method

As we saw in the last slides the Jacobi method is slow for the low frequencies

But let‘s take look at the initial guess at the coarser grid

Let‘s take every second point for the next coarser grid

The next coarse grid has half as much grid points as the fine grid has, so the fourier modes represented in the fine grid can be rewritten as follows

for the coarse grid

h

jk

h

jk wN

jk

N

jkw 2

,2, )2/

sin()2

sin(

This means that the kth mode on Ωh becomes the kth mode on Ω2h, which means that in passing from the fine grid to the coarse grid, a mode becomes

more oscillatory

14

15

16

The Multigrid MethodNow that we know the smooth modes in the fine grid are oscilatory in the

next coarse grids, what shall we do?

How about doing some iterations of the Jacobi in the fine grid to get a smoother solution and then pass the residual every second point to the

coarser grid and do more smoothing there?

What shall we do when we are done with the smoothing on the coarse grid?

Shall we return to the fine grid? How? We don‘t have the same dimension!

Shall we go to the next coarser grid? Why?

We continue going down to the next coarser grid till we are in the coarsetst grid where we only have the lowest frequency mode left

And then we need to find a way to go back to the finer grids and check our stopping criteria

17

Going back to the fine gridThe next step is to go from the coarsest grid one level up to the next

fine grid,

But for this step we need to generate the intermediate points which we lost while going to corase grid

The interpolation techniques might help us through this

We can regenerate the intermediate points by interpolating them using the 2 neighboring points

gridcoarse

j

gridfine

j

gridcoarse

j

gridfine

j

gridcoarse

j

gridcoarse

jgridfine

j uuuuuu

u 1222

1

12 ,,2

Now when we are one level finner, we will do some more iterations of the Jacobi method to avoid the high frequency oscillations occured from

going to coarse grid and coming back and continue going up using the same steps till we are in the finest grid

18

Back in the finest grid

Now we will check our resedual and if the resedual is what we expect we stop the solver and we have the solution, else we have to redo the same

operation till we reach our expected residual

Now that we are in the finest (original) grid what shall we do?

Are we done with solving the system?

Is the solution good enough?

One multigrid cycle is also called a V cycle since we go down and come back up

19

Multigrid code1. Pre-smoothing: Perform some iterations of Modified Jacobi to get a rather

smooth solution

2. Construct the residue

3. Restrict your residue to every second point to go to the coarse grid

4. Do step 1

5. Goto step 2 till you are in the coarsest grid

6. Interpolate your results and go back to the finer grid

7. Add the result from the coarse grid to the previous results

8. Post-smoothing: perform some iteration of Modified Jacobi

9. Goto step 5 till you are in the finest grid

10. Check the residual

11. If condition not satisfied start from 1

12. else return the solution

20

The expected residual achieved!!

21

References

• Prof. Bungartz, Numerical Programming course, TUM (CSE), WS 2007

•Dr. Bader, Scientific Computing Course, TUM (CSE), WS 2007

•Dr. Mehl, Scientific Computing Lab Course, TUM (CSE), WS 2007

•William L. Briggs, A Multigirid Tutorial (1987)

22

Thank you for attending this talk