lec05 systems
TRANSCRIPT
-
8/9/2019 Lec05 Systems
1/36
SOLVING LINEAR SYSTEMS OF EQUATIONS
See Chapter 3 of text
Background on linear systems
Gaussian elimination and the Gauss-Jordan algorithms
The LU factorization
Gaussian Elimination with pivoting
5-1 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
2/36
Background: Linear systems
The Problem: Ais an n n matrix, and ba vector ofRn.
Find x such that:
Ax=b
x is theunknownvector, b theright-hand side, and A
is thecoefficient matrix
Example:
2x1 + 4x2 + 4x3 = 6
x1 + 5x2 + 6x3 = 4x1 + 3x2 + x3 = 8
or2 4 41 5 6
1 3 1
x1x2x3
= 648
Solution of above system ?5-2 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
3/36
Standard mathematical solution by Cramers rule:
xi = det(Ai)/ det(A)
Ai = matrix obtained by replacing i-th column by b.
Note: This formula is useless in practice beyond n= 3
orn= 4.
Three situations:
1. The matrix A is nonsingular. There is a unique solutiongiven by x=A1b.
2. The matrix A is singular and b Ran(A). There are
infinitely many solutions.3. The matrix A is singular and b /Ran(A). There are no
solutions.
5-3 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
4/36
Example: (1) Let A = 2 0
0 4 b =
1
8 . A is non-
singular a unique solution x=
0.5
2
.
Example: (2) Case whereAis singular &b Ran(A):
A= 2 0
0 0 , b= 1
0 . infinitely many solutions: x() =
0.5
.
Example: (3) Let A same as above, but b=
1
1
.
No solutions since 2nd equation cannot be satisfied
5-4 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
5/36
Triangular linear systems
Example:
2 4 40 5 20 0 2
x1x2x3
= 214
One equation can be trivially solved: the last one.x3 = 2
x3 is known we can now solve the 2nd equation:
5x2 2x3 = 1 5x2 2 4 = 1 x2 = 1
Finally x1 can be determined similarly:
2x1+ 4x2+ 4x3 = 2 ... x1 =5
5-5 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
6/36
ALGORITHM : 1 Back-Substitution algorithm
Fori =n:1 : 1 do:t :=biForj =i+ 1 :n do
t := t (ai,i+1:n, xi+1:n)= t an inner productt:=t aijxjEndxi =t/aii
End
We must require that each aii = 0
Operation count?
Round-off error (use previous results for (, ))?
5-6 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
7/36
Backward error analysis for the triangular solve
The computed solution xof the triangular systemU x=b computed by the previous algorithm satisfies:
(U+E)x=b
with|E| n u |U| +O(u 2)
Backward error analysis. Computed x solves a slightly
perturbed system.
Backward error not large in general. It is said that
triangular solve is backward stable.
5-7 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
8/36
Column version of back-substitution:
Back-Substitution algorithm. Column versionForj =n :1 : 1 do:
xj =bj/ajj
Fori = 1 :j 1 dobi := bi xj aij
End
End
Justify the above algorithm [Show that it does indeed
compute the solution]
See text for analogous algorithms for lower triangular
systems.
5-8 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
9/36
Linear Systems of Equations: Gaussian Elimination
Back to arbitrary linear systems.
Principle of the method: Since triangular systems are
easy to solve, we will transform a linear system into
one that is triangular. Main operation: combine rows so
that zeros appear in the required locations to make the
system triangular.
Notation: use a Tableau:
2x1 + 4x2 + 4x3 = 2
x1 + 3x2 + 1x3 = 1
x1 + 5x2 + 6x3 = 6
tableau:2 4 4 21 3 1 1
1 5 6 6
5-9 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
10/36
Main operation used: scaling and adding rows.
Example: Replace row2 by: row2 - 1
2*row1:2 4 4 2
1 3 1 1
1 5 6 6
2 4 4 2
0 1 1 0
1 5 6 6 This is equivalent to:
1 0 0
12
1 0
0 0 1
2 4 4 2
1 3 1 1
1 5 6 6
=
2 4 4 2
0 1 1 0
1 5 6 6
The left-hand matrix is of the form
M =I veT1 with v =
012
0
5-10 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
11/36
Linear Systems of Equations: Gaussian Elimination
Go back to original system. Step 1 must transform:
2 4 4 2
1 3 1 1
1 5 6 6
into:
x x x x
0 x x x
0 x x x
row2 :=row2 12
row1: row3 := row3 12
row1:
2 4 4 2
0 1 1 0
1 5 6 6
2 4 4 2
0 1 1 0
0 3 4 7
5-11 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
12/36
Equivalent to
1 0 0
12
1 0
12
0 1
2 4 4 2
1 3 1 1
1 5 6 6
=
2 4 4 2
0 1 1 0
0 3 4 7
[A, b][M1A, M1b]; M1 =I v(1)eT1 ; v
(1) =
012
12
New system A1x=b1. Step 2 must now transform:
2 4 4 20 1 1 0
0 3 4 7
into:x x x x0 x x x
0 0 x x
-
8/9/2019 Lec05 Systems
13/36
row3 := row3 3 row2 :
2 4 4 2
0 1 1 00 0 7 7
Equivalent to
1 0 0
0 1 0
0 3 1
2 4 4 2
0 1 1 0
0 3 4 7
=
2 4 4 2
0 1 1 0
0 0 7 7
Second transformation is as follows:
[A1, b1][M2A1, M2b1] M2 =I v(2)eT2 v(2) =00
3
Triangular system Solve.
5-13 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
14/36
Row k
Pivot
A =k
5-14 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
15/36
ALGORITHM : 2 Gaussian Elimination
1. For k = 1 :n 1 Do:2. For i=k+ 1 :n Do:
3. piv :=aik/akk4. For j :=k+ 1 :n+ 1 Do :
5. aij :=aij piv akj6. End
6. End
7. End
Operation count:
T =n1k=1
ni=k+1
[1+n+1
j=k+1
2] =n1k=1
ni=k+1
(2(n k)+3) =...
Complete the above calculation. Order of the cost?5-15 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
16/36
The LU factorization
Now ignore the right-hand side from the transformations.
Observation: Gaussian elimination is equivalent to n
1 successive Gaussian transformations, i.e., multiplica-
tions with matrices of the formMk =I v(k)eTk , where
the first k components ofv(k) equal zero.
Set A0 A
A M1A0 =A1 M2A1 =A2 M3A2 =A3
Mn1An2 =An1 U
Last Ak U is an upper triangular matrix.
5-16 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
17/36
At each step we have: Ak =M1k+1Ak+1. Therefore:
A0 = M11 A1= M11 M
12 A2
= M11 M12 M
13 A3
= . . .= M11 M
12 M
13 M
1n1An1
L=M11 M12 M
13 M
1n1
Note: L is Lower triangular, An1 is upper triangular
LU decomposition : A=LU
5-17 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
18/36
How to get L?
L=M11 M
12 M
13 M
1n1
Consider only the first 2 matrices in this product.
Note M
1
k = (I v
(k)
e
T
k )
1
= (I+v
(k)
e
T
k ). So:
M11 M12 = (I+ v
(1)eT1 )(I+ v(2)eT2 ) =I+ v
(1)eT1 + v(2)eT2
Generally,
M11 M12 M
1k =I+v
(1)eT1 +v(2)eT2 + v
(k)eTk
-
8/9/2019 Lec05 Systems
19/36
The L factor is a lower triangular matrix with ones on
the diagonal. Column k of L, contains the multipliers
lik used in the k-th step of Gaussian elimination.
5-19 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
20/36
A matrix A has an LU decomposition if
det(A(1 :k, 1 :k))= 0 for k = 1, , n 1.
In this case, the determinant ofA satisfies:
det A = det(U) =
ni=1
uii
If, in addition, A is nonsingular, then the LU factoriza-
tion is unique.
5-20 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
21/36
Show how to obtain L directly from the multipliers
Practical use: Show how to use the LU factorization tosolve linear systems with the same matrix A and different
bs.
LU factorization of the matrixA =2 4 41 5 6
1 3 1
?
Determinant ofA
? True or false: Computing the LU factorization of
matrix A involves more arithmetic operations than solving
a linear system Ax=b by Gaussian elimination.
5-21 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
22/36
Gauss-Jordan Elimination
Principle of the method: We will now transform thesystem into one that is even easier to solve than triangular
systems, namely a diagonal system. The method is very
similar to Gaussian Elimination. It is just a bit moreexpensive.
Back to original system. Step 1 must transform:
2 4 4 2
1 3 1 1
1 5 6 6
into:
x x x x
0 x x x
0 x x x
5-22 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
23/36
row2 :=row20.5row1: row3 :=row30.5row1:
2 4 4 2
0 1 1 0
1 5 6 6
2 4 4 2
0 1 1 0
0 3 4 7
Step 2:2 4 4 20 1 1 0
0 3 4 7
into:x 0 x x0 x x x
0 0 x x
row1 :=row1 4 row2: row3 :=row3 3 row2:
2 0 8 2
0 1 1 0
0 3 4 7
2 0 8 2
0 1 1 0
0 0 7 7
5-23 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
24/36
There is now a third step:
To transform: 2 0 8 20 1 1 0
0 0 7 7
into: x 0 0 x0 x 0 x
0 0 x x
row1 :=row1 87 row3: row2 :=row2 1
7 row3:
2 0 0 10
0 1 1 0
0 0 7 7
2 0 0 10
0 1 0 1
0 0 7 7
Solution: x3 =1; x2 =1; x1 = 5
5-24 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
25/36
ALGORITHM : 3 Gauss-Jordan elimination
1. For k = 1 :n Do:
2. For i= 1 :n and ifi! =k Do :
3. piv :=aik/akk4. For j :=k+ 1 :n+ 1 Do :
5. aij :=aij piv akj6. End
6. End
7. End
Operation count:
T =n
k=1
n1i=1
[1 +n+1
j=k+1
2] =n1k=1
n1i=1
(2(n k) + 3) =
Complete the above calculation. Order of the cost?
How does it compare with Gaussian Elimination?5-25 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
26/36
function x = gaussj (A, b)%---------------------------------------------------% function x = gaussj (A, b)% solves A x = b by Gauss-Jordan elimination%---------------------------------------------------
n = size(A,1) ;A = [A,b];for k=1:n
for i=1:nif (i ~= k)
piv = A(i,k) / A(k,k) ;A(i,k+1:n+1) = A(i,k+1:n+1) - piv*A(k,k+1:n+1);
endend
endx = A(:,n+1) ./ diag(A) ;
5-26 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
27/36
Gaussian Elimination: Partial Pivoting
Consider again Gaussian Elimination for the linear system
2x1 + 2x2 + 4x3 = 2
x1 + x2 + x3 = 1
x1 + 4x2 + 6x3 = 5
Or:
2 2 4 2
1 1 1 1
1 4 6 5row2 :=row2
12
row1: row3 := row3 12
row1:
2 2 4 2
0 0 1 0
1 4 6 5
2 2 4 2
0 0 1 0
0 3 4 6
Pivot a22 is zero. Solution : permute rows 2 and 3:2 2 4 2
0 3 4 6
0 0 1 05-27 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
28/36
Gaussian Elimination with Partial Pivoting
Partial Pivoting
General situation:
Largest a ik
Perm
uterows
akk
Row k
Always permute row k with row l such that|alk|= maxi=k,...,n |aik|
More stable algorithm.
5-28 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
29/36
function x = gaussp (A, b)%---------------------------------------------------
% function x = guassp (A, b)% solves A x = b by Gaussian elimination with% partial pivoting/%---------------------------------------------------
n = size(A,1) ;A = [A,b]for k=1:n-1
[t, ip] = max(abs(A(k:n,k)));ip = ip+k-1 ;
%% swaptemp = A(k,k:n+1) ;A(k,k:n+1) = A(ip,k:n+1);A(ip,k:n+1) = temp;
%%
for i=k+1:npiv = A(i,k) / A(k,k) ;A(i,k+1:n+1) = A(i,k+1:n+1) - piv*A(k,k+1:n+1);
endend
x = backsolv(A,A(:,n+1));5-29 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
30/36
Pivoting and permutation matrices
A permutation matrix is a matrix obtained from theidentity matrix by permuting its rows
For example for the permutation = {3, 1, 4, 2} we
obtain
P =
0 0 1 0
1 0 0 00 0 0 1
0 1 0 0
Important observation: the matrix P A is obtained from
A by permuting its rows with the permutation
(P A)i,: =A(i),:
5-30 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
Wh i h i P A h
-
8/9/2019 Lec05 Systems
31/36
What is the matrix P A when
P =
0 0 1 0
1 0 0 0
0 0 0 1
0 1 0 0
A=
1 2 3 4
5 6 7 8
9 0 1 2
3 4 5 6
?
Any permutation matrix is the product of interchange
permutations, which only swap two rows ofI.
Notation: Eij = Identity with rows i and j swapped
5-31 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
32/36
Example: To obtain ={3, 1, 4, 2} from ={1, 2, 3, 4}
we need to swap (2) (3) then (3) (4) and
finally (1) (2). Hence:
P =
0 0 1 0
1 0 0 00 0 0 1
0 1 0 0
=E1,2 E3,4 E2,3
In the previous example where
>> A = [ 1 2 3 4; 5 6 7 8; 9 0 -1 2 ; -3 4 -5 6]
Matlab gives det(A) =896. What is det(P A)?
5-32 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
At h t f G E ith ti l i ti
-
8/9/2019 Lec05 Systems
33/36
At each step of G.E. with partial pivoting:
Mk+1Ek+1Ak =Ak+1
whereEk+1encodes a swap of rowk+1with rowl > k+1.
Notes: (1) E1i = Ei and (2) M1
j Ek+1 = Ek+1
Mj1
for k j, where Mj has a permuted Gauss vector:
(I+v(j)eTj)Ek+1 = Ek+1(I+Ek+1v(j)eTj)
Ek+1(I+ v(j)eTj)
Ek+1 Mj
Here we have used the fact that above row k+ 1, the
permutation matrix Ek+1 looks just like an identity matrix.
5-33 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
Res lt
-
8/9/2019 Lec05 Systems
34/36
Result:
A0 = E1M11 A1
= E1M11 E2M
12 A2 =E1E2
M11 M12 A2
= E1E2 M11 M
12 E3M
13 A3
= E1E2E3 M11 M
12 M
13 A3
= . . .
= E1 En1 M11 M
12 M
13 M
1n1 An1
In the end
P A=LU with P =En1 E1
5-34 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
-
8/9/2019 Lec05 Systems
35/36
Error Analysis
If no zero pivots are encountered during Gaussian elimi-nation (no pivoting) then the computed factors L and U
satisfyL
U =A+H
with
|H| 3(n 1) u
|A| + |L| |U|
+O(u 2)
Solution x computed via Ly =b and Ux= y is s. t.(A+E)x =b with
|E| nu 3|A| + 5 |L| |U|+O(u 2)
5-35 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems
Backward error estimate
-
8/9/2019 Lec05 Systems
36/36
Backward error estimate.
|L| and |U| are not known in advance they can be
large.
What if partial pivoting is used?
Permutations introduce no errors. Equivalent to stan-dard LU factorization on matrix P A.
|L| is small since lij 1. Therefore, only U is uncer-
tain
In practice partial pivoting is stable i.e., it is highly
unlikely to have a very large U.
5-36 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems