iterative methods for solving system of linear equations · gauss-seidel method •the gauss-seidel...
TRANSCRIPT
Iterative Methodsfor Solving
System of Linear Equations
Iterative Solution Methods• Starts with an initial approximation for the solution vector (x0)
• At each iteration updates the x vector by using the sytem Ax=b
• During the iterations A, matrix is not changed so sparcity is preserved
• Each iteration involves a matrix-vector product
Different Iterative methods
• Jacobi Iteration
• Gauss-Seidel Iteration
Gauss - Jacobi Method
])([ )( )]([
xDAbDxxDAbDxbxDAD
a
a
a
D
aaa
aaa
aaa
AbAx
1
33
22
11
333231
232221
131211
00
00
00
*
/
/
/
3
2
1
3231
2321
1312
3
2
1
33
22
11
3
2
1
0
0
0
100
010
001
x
x
x
aa
aa
aa
b
b
b
a
a
a
x
x
x
33
1
232
1
13133
22
1
323
1
12122
11
1
313
1
21211
a
xaxabx
a
xaxabx
a
xaxabx
kkk
kkk
kkk
Jacobi iteration
nnnnnn
nn
nn
bxaxaxa
bxaxaxa
bxaxaxa
2211
22222121
11212111
0
0
2
0
1
0
nx
x
x
x
)(1 0
1
0
2121
11
1
1 nnxaxaba
x
1
1 1
1 1 i
j
n
ij
k
jij
k
jiji
ii
k
i xaxaba
x
)(1 0
11
0
22
0
11
1
nnnnnn
nn
n xaxaxaba
x
)(1 0
2
0
323
0
1212
22
1
2 nnxaxaxaba
x
Jacobi Iteration - Example
19832
12283
738
zyx
zyx
zyx
0000 zyx
nnn
nnn
nnn
yxz
zxy
zyx
32198
1
23128
1
378
1
1
1
1
1st iteration:
2nd iteration:
Jacobi Iteration - Example
1 0 0
1 0 0
1 0 0
1 77 3 0.875
8 8
1 1212 3 2 1.5
8 8
1 1919 2 3 2.375
8 8
x y z
y x z
z x y
2 1 1
2 1 1
2 1 1
1 17 3 7 1.5 3 2.375 0.2031
8 8
1 112 3 2 12 3 0.875 2 2.375 0.578125
8 8
1 119 2 3 19 2 0.875 3 1.5 1.59375
8 8
x y z
y x z
z x y
Jacobi Iteration - Example
• x(3) = -0.427734375
• y(3) = 1.177734375
• z(3) = 2.876953125
• x(4) = -0.351
• y(4) = 0.620
• z(4) = 1.814
Actual Solution:
x =0
y =1
z =2
Iteration X1 X2 X3
0 0 0 0
1 0.875 1.5 2.375
2 -0.2031 0.5781 1.59375
3 -0.4277 1.1777 2.8769
4 -0.351 0.620 1.814
Jacobi Iteration
• Questions:1. How many iterations do we need?2. What is our stopping criteria?3. Is it faster than applying Gaussian Elimination?4. Are there round-off errors or other precision and
robustness issues?
Example 1 Perform five Jacobi Iteration for
15
21
7
512
184
114
3
2
1
x
x
x
0
0
00x
5
215
8
421
4
7
0
2
0
11
3
0
3
0
11
2
0
3
0
21
1
xxx
xxx
xxx
0.35
15
625.28
21
75.14
7
Example 1 continued...
5
215
8
421
4
7
1
2
1
12
3
1
3
1
12
2
1
3
1
22
1
xxx
xxx
xxx
175.35
625.275.1215
875.38
375.1421
65625.14
3625.27
8875.25
875.365625.1215
5
215
85.38
175.365625.1421
8
421
6625.14
175.3875.37
4
7
2
2
2
13
3
2
3
2
13
2
2
3
2
23
1
xxx
xxx
xxx
Example 1 continued...
5
215
8
421
4
7
3
2
3
14
3
3
3
3
14
2
3
3
3
24
1
xxx
xxx
xxx
895.25
85.36625.1215
817.38
8875.26625.1421
99.14
8875.285.37
033.35
817.399.1215
5
215
979.38
869.299.1421
8
421
981.14
895.2817.37
4
7
4
2
4
15
3
4
3
4
15
2
4
3
4
25
1
xxx
xxx
xxx
Example 2
7
21
15
114
184
512
3
2
1
x
x
x
0
0
00x
0
2
0
1
1
3
0
3
0
11
2
0
3
0
21
1
47
8
421
2
515
xxx
xxx
xxx
0.7
625.28
21
5.72
15
Example 2 continued...
625.39625.25.747
25.08
75.7421
3125.112
75625.215
1
3
1
2
1
1
x
x
x
The residual term is increasing at each iteration, so the iterations
are diverging.
Note that the matrix is not diagonally dominant
Iterative Method: PitfallWhat went wrong?
Even though done correctly, the answer is not converging to the
correct answer.
This example illustrates a pitfall of the iterative method: not all
systems of equations will converge.
Is there a fix?
One class of system of equations always converges: One with
a diagonally dominant coefficient matrix.
Diagonally dominant: [A] in [A] [X] = [B] is diagonally dominant
if:
n
jj
ijaa
i1
ii
n
ijj
ijii aa1
for all ‘i’
andfor at least one ‘i’
Example
1373
351
5312
A
Check if the coefficient matrix A is
diagonally dominant,
43155 232122 aaa
10731313 323133 aaa
8531212 131211 aaa
The inequalities are all true and at least one row
is strictly greater than.
Therefore: The solution should converge using
the Iterative Method.
Example
7
21
15
114
184
512
3
2
1
x
x
x
51488 232122 aaa
51411 323133 aaa
65122 131211 aaa
The inequalities are all not true although one row
is strictly greater than.
Therefore: The solution should not converge
using the Iterative Method.
Example 2 (Rearranged)
15
21
7
512
184
114
3
2
1
x
x
x
0
0
00x
5
215
8
421
4
7
0
2
0
11
3
0
3
0
11
2
0
3
0
21
1
xxx
xxx
xxx
0.3
625.28
21
75.14
7
Same as Example 1.Hence it will Converge to x = [2 4 3]’.
Gauss-Seidel Method
• The Gauss-Seidel method is a commonly used iterative method.
• It is same as Jacobi technique except with one important difference:
A newly computed x value (say xk) is substituted in the subsequent equations (equations k+1, k+2, …, n) in the same iteration.
•First, choose initial guesses for the x’s.
•A simple way to obtain initial guesses is to assume that they are all zero.
•Compute new x1 using the previous iteration values.
•New x1 is substituted in the equations to calculate x2 and x3
•The process is repeated for x2, x3, …newold
newnewnew
oldnewnew
oldoldnew
XX
a
xaxabx
a
xaxabx
a
xaxabx
}{}{
33
23213133
22
32312122
11
31321211
Gauss-Seidel Iteration
11
)(
313
)(
2121
)1(
1 /)( axaxabx iii
• Improved version of Jacobi Iteration
• Observation: If x's converge, then after computing
x1(i+1) should be a better approximation of x1 than x1
(i).
Thus the updating formula of Gauss-Seidel Iteration becomes
11
)(
313
)(
2121
)1(
1 /)( axaxabx iii
22
)(
323
)1(
1212
)1(
2 /)( axaxabx iii
33
)1(
232
)1(
1313
)1(
3 /)( axaxabx iii
Convergence• A simple way to determine the convergence is to
inspect the diagonal elements.
• All of the diagonal elements must be non-zero.
• Convergence is guaranteed if the system is diagonally dominant.
n
ijj
ijii aa,1
• What if the matrix is not diagonally dominant?
Example 3 Perform Gauss Seidel Iteration for
15
21
7
512
184
114
3
2
1
x
x
x
0
0
00x
5
215
8
421
4
7
211
3
311
2
321
1
newnew
oldnew
oldold
xxx
xxx
xxx
35
5.3)75.1(215
5.38
)75.1(421
75.14
7
Example 3 continued...
5
215
8
421
4
7
212
3
312
2
322
1
newnew
oldnew
oldold
xxx
xxx
xxx
9525.25
9375.3)85.1(215
9375.38
3)85.1(421
85.14
35.37
0001.35
9922.3)99625.1(215
5
215
9922.38
9525.2)99625.1(421
8
421
99625.14
9525.29375.37
4
7
213
3
313
2
323
1
newnew
oldnew
oldold
xxx
xxx
xxx
Example 3 continued...
5
215
8
421
4
7
214
3
314
2
324
1
newnew
oldnew
oldold
xxx
xxx
xxx
999.25
999.3)998.1(215
999.38
0001.3)998.1(421
998.14
0001.39922.37
00002.35
99988.3)2(215
5
215
99988.38
999.3)2(421
8
421
24
999.2999.37
4
7
215
3
315
2
325
1
newnew
oldnew
oldold
xxx
xxx
xxx
Example 4 Solve the following system using Gauss Seidel Method.-3x1+ x2+ 12x3 = 50 6x1- x2 - x3 = 36x1 + 9x2 + x3 = 40
The system can be re-arranged to the following6x1 - x2 - x3 = 36x1 + 9x2 + x3 = 40-3x1 + x2 + 12x3 = 50
12
350
9
640
6
3
213
312
321
newnew
oldnew
oldold
xxx
xxx
xxx
Iteration X1 X2 X3
0 0 0 0
1 0.5 4.1111 3.9491
2 1.8434 2.7768 4.3961
3 1.6955 2.8257 4.3551
4 1.6968 2.8294 4.3551
Example 5 Solve the following system using Gauss Seidel Method.
The system can be re-arranged to the following
3.19
85.7
4.71
3.071.0
2.01.03
102.03.0
3
2
1
x
x
x
4.71
3.19
85.7
102.03.0
3.071.0
2.01.03
3
2
1
x
x
x
Iteration X1 X2 X3
0 0 0 0
1 2.6167 -2.7945 7.0056
2 2.9906 -2.4996 7.0003
3 3.00003 -2.49999 6.9999
4 2.9999 -2.50000 7.0000
0,Initially
10/)2.03.04.71(
7/)3.01.03.19(
3/)2.01.085.7(
32
213
312
321
xx
xxx
xxx
xxx
newnew
oldnew
oldold