chapter 12 gaussian elimination (ii)
DESCRIPTION
Chapter 12 Gaussian Elimination (II). Speaker: Lung-Sheng Chien. Reference book: David Kincaid, Numerical Analysis Reference lecture note: Wen-wei Lin, chapter 2, matrix computation http://math.ntnu.edu.tw/~min/matrix_computation.html. OutLine. - PowerPoint PPT PresentationTRANSCRIPT
Chapter 12 Gaussian Elimination (II)
Speaker: Lung-Sheng Chien
Reference book: David Kincaid, Numerical Analysis
Reference lecture note: Wen-wei Lin, chapter 2, matrix computation
http://math.ntnu.edu.tw/~min/matrix_computation.html
OutLine
• Weakness of A=LU- erroneous judgment: A is invertible but A=LU does not exist- unstable, A is far from LU
• A=LU versus PA=LU• Pivoting strategy• Implementation of PA=LU• MATLAB usage
Fail of LU: singular of leading principal minor
1
2
0 1 11 1 2
xx
10
0 1 1 0 0 11 1 1 0 ?
A
11 0, cannot continuea LU
Question: How about interchanging row 1 and row 2?
1
2
0 1 11 1 2
xP P
x
2,1P
1
2
1 1 20 1 1
xx
1 10 1
U
1 00 1
L
1
2
11
xx
and
unstable of LU: near singular of leading principal minor [1]
1
2
1 11 1 2
xx
1 02 2
2
1 1 11
1 2 1 1x
O Ox
0
1
2
0 1 11 1 2
xx
1
2
11
xx
0 ?
Theoretical:
Numerical: 1 0 111 11 1 1 0 1
A
if 0 then 21 22, L U
Problem: for double-precision, we only have 16 digit-accuracy, we cannot accept
1
2
1 1 1 11 11 2 1 21 1
xx
, why?
1
2
12
xAx
1
2
1 1 0 111 1 120 1 1 2
xx
1 1
2
12
xU Lx
backward substitution 2
1 11 2x
02
2 1/ 11 1/
x
step 1:
step 2: 1 2 1x x 21
1 0xx
22
2 1/ 1/ 1 2 11 1/ 1/ 1
x O
Question: why not
01 1/ 1/ due to rounding error02 1/ 1/ due to rounding error
unstable of LU: near singular of leading principal minor [2]
Case 1: 1410 Case 2: 1810
unstable of LU: near singular of leading principal minor [3]
wrong solution
Question: How about interchanging row 1 and row 2?
1
2
1 1 21 1
xx
1 1 1 0 1 11 1 0 1
A
LU-factorization
1
2
1 1 1 0 2 20 1 1 1 1 2
xx
1
2
12
xAx
1 1
2
12
xU Lx
backward substitution 21 1 2x 0
21 2 11
x
step 1:
step 2: 1 2 2x x 1 22 1x x
unstable of LU: near singular of leading principal minor [4]
Key observation:
without pivoting
02
2 1/ 11 1/
x
(rounding normal number)
pivoting
02
1 2 11
x
(rounding error does not occur for normal number)
Case 1: 1410 Case 2: 1810
unstable of LU: near singular of leading principal minor [5]
Why not 1?
1 ,max iji j n
A A
1810 without pivoting pivoting 1810
1610L U
1L U
(NOT stable)
(stable)
Objective: control growth rate of ,L U control multiplier
unstable of LU: cause and controllability
define largest component of matrix A
OutLine
• Weakness of A=LU• A=LU versus PA=LU
- controllability of lower triangle matrix L
• Pivoting strategy• Implementation of PA=LU• MATLAB usage
6 -2 2 4
12 -8 6 10
3 -13 9 3
-6 4 1 -18
121 36
6
6 -2 2 4
0 -4 2 2
0 -12 8 1
0 2 3 -14
1
2 1
0.5 1
-1 1
1L 2A
12,1 2 1L since 1
11 6A is not maximum among 1: 4,1A
6 -2 2 4
0 -4 2 2
0 -12 8 1
0 2 3 -14
12124
1
1
3 1
-0.5 1
2L
6 -2 2 4
0 -4 2 2
0 0 2 -5
0 0 4 -13
3A 2A
1A A
23,2 3 1L since 2
22 4A is not maximum among 2 2 : 4,2A
Recall LU example in chapter 11 [1]
Recall LU example in chapter 11 [2]
6 -2 2 4
0 -4 2 2
0 0 2 -5
0 0 4 -13
3A
42
1
1
1
2 1
3L
6 -2 2 4
0 -4 2 2
0 0 2 -5
0 0 0 -3
4A
34,3 2 1L since 3
33 2A is not maximum among 3 3: 4,3A
6 -2 2 4
12 -8 6 10
3 -13 9 3
-6 4 1 -18
A
1
2 1
0.5 3 1
-1 -0.5 2 1
L
6 -2 2 4
-4 2 2
2 -5
-3
U
18A 3L
6U
Question: How can we control L , is uniform bound possible?
6 2 2 412 8 6 103 13 9 36 4 1 18
A
2
3
4
1
2,3, 4,1
T
T
T
T
ee
Pee
PA = LU in MATLAB
12 8 6 103 13 9 36 4 1 18
6 2 2 4
PA
12 -8 6 10
3 -13 9 3
-6 4 1 -18
6 -2 2 4
31 6
126
12 -8 6 10
0 -11 7.5 0.5
0 0 4 -13
0 2 -1 -1
1
0.25 1
-0.5 1
0.5 1
1L 2A
1 2 : 4,1 1L
since 111 12A is maximum among 1 1: 4,1A
1PA A
PA = LU: assume P is given [1]
12 -8 6 10
0 -11 7.5 0.5
0 0 4 -13
0 2 -1 -1
2A
12 -8 6 10
0 -11 7.5 0.5
0 0 4 -13
0 0 4/11 -10/11
1
1
0 1
-2/11 1
2L 3A
01211
2 3: 4,2 1L
since 222 11A is maximum among 2 2 : 4,2A
PA = LU: assume P is given [2]
12 -8 6 10
0 -11 7.5 0.5
0 0 4 -13
0 0 0 3/11
4A
1 44 11
12 -8 6 10
0 -11 7.5 0.5
0 0 4 -13
0 0 4/11 -10/11
3A1
1
1
1/11 1
3L
3 1 111
L since 333 4A is maximum among 3 3: 4,3A
PA L U
12 -8 6 10
3 -13 9 3
-6 4 1 -18
6 -2 2 4
1
0.25 1
-0.5 0 1
0.5 -2/11 1/11 1
12 -8 6 10
-11 7.5 0.5
4 -13
3/11
18A 0.5 1L
13U
Observation: though proper permutation, we can control 1L
Question: in fact, we cannot know permutation matrix in advance, how can we do?
1 1 111 12 1
1 1 11 21 22 2
1 1 11 2
n
n
n n nn
a a a
a a aA A
a a a
1 1 1 111 12 13 1
2 2 222 23 2
2 2 2 232 33 3
2 2 22 3
0
0
0
n
n
n
n n nn
a a a a
a a aA a a a
a a a
1 2A L A
1 2 1k kA L L L A
1 111 1
1 11, 1 1,
, ,
0
0
0 0
n
k kk k k k n
k kk k k n
k knk nn
a a
a aA
a a
a a
Sequence of matrices during LU-decomposition
1 2 1 10
kL L LI
k row
k col
OutLine
• Weakness of A=LU• A=LU versus PA=LU• Pivoting strategy
- partial pivoting (we adopt this formulation)- complete pivoting
• Implementation of PA=LU• MATLAB usage
max : ,k kpkA A k n k , 1, ,p k k n (1) find a such that
(2) swap row ,1:kA k n and row ,1:kA p n
Partial pivoting and complete pivoting
PA LUthen with 1L
Partial pivoting
max : , :k kpqA A k n k n , , 1, ,p q k k n (1) find a such that
(2) swap row ,1:kA k n and row ,1:kA p n
PA LU then with 1L
(2) swap column 1: ,kA n k and column 1: ,kA n q
complete pivoting
1
3
2
1 0 01,3,2 0 0 1
0 1 0
T
T
T
eP e
e
Define permutation matrix
1 1 1
3 3 3
2 2 2
T T
T T
T T
e e x xPx e x e x x
e e x x
Recall permutation matrix
1 2 37 8 94 5 6
PA
1 2 34 5 67 8 9
A
interchange row 2, 3
1
3 1 3 2
2
T
TT T
xx P Px x x x x
x
1 3 24 6 57 9 8
TAP
interchange column 2, 3
Let
1 2 3
4 5 6
7 8 9
interchange
row 2, 3
1 2 3
7 8 9
4 5 6
interchange
column 2, 3
1 3 27 9 84 6 5
A PA TPA PSymmetric permutation:
TA PAP
x1
x2 x3
1
2 3
4
56
7
89
Meaning of matrix coefficient
ijA relationship between node and node i j3
1
:ij j ij
A x b
constraint on node i
node is named
1,3,2P
x1
x3 x2
1
2 3
4
56
7
89
i ix
change node 2 to node 3 and node 3 to node 2
write constraint for now node index
1 3 27 9 84 6 5
x1
x1
x2
x3
x2 x3
1 2 3
4 5 6
7 8 9
x1 x2 x3
x1
x2
x3
Identity matrix is invariant under symmetric permutation
x1
x2 x3
1
1
1
1
1
1
x1
x1
x2
x3
x2 x3
1,3,2P change node 2 to node 3 and node 3 to node 2
x1
x3 x2
1
1
1
1
1
x1
x1
x2
x3
x2 x31
PA = LU: partial pivoting [1]
6 -2 2 4
12 -8 6 10
3 -13 9 3
-6 4 1 -18
1A A
1 121 max 1: 4,1A A
1 2,1,3, 4P
12 -8 6 10
6 -2 2 4
3 -13 9 3
-6 4 1 -18
1 11PA A
12 -8 6 10
6 -2 2 4
3 -13 9 3
-6 4 1 -18
1A
61 3
126
1
0.5 1
0.25 1
-0.5 1
1L
12 -8 6 10
0 2 -1 -1
0 -11 7.5 0.5
0 0 4 -13
2A
12 -8 6 10
0 2 -1 -1
0 -11 7.5 0.5
0 0 4 -13
2A
2 232 max 2 : 4,2A A
2 1,3, 2, 4P
12 -8 6 10
0 -11 7.5 0.5
0 2 -1 -1
0 0 4 -13
2 22P A A
PA = LU: partial pivoting [2]
1 1 2 1 2 1 2 1 21 2 2 2 2 2
T T TPA A L A L P P A L P P A L P A
1 2 1 22 1 2 2
TP PA P L P A L A 2 1 1,3,2, 4 2,1,3, 4 2,3,1,4P P where
1
0.5 1
0.25 1
-0.5 1
1L
2 1,3,2,4P
12TL P
Interchange columns 2, 3
1
0.5 1
0.25 1
-0.5 1
12TL P
12 2
TP L P
Interchange rows 2, 3
1
0.25 1
0.5 1
-0.5 1
12 2
TP L P
12 -8 6 10
3 -13 9 3
6 -2 2 4
-6 4 1 -18
2 1P PA
1
0.25 1
0.5 1
-0.5 1
1L
12 -8 6 10
0 -11 7.5 0.5
0 2 -1 -1
0 0 4 -13
2A
verify
PA = LU: partial pivoting [3]
12 -8 6 10
0 -11 7.5 0.5
0 2 -1 -1
0 0 4 -13
2A
1
1
-2/11 1
0 1
2L
21011
12 -8 6 10
0 -11 7.5 0.5
0 0 4/11 -10/11
0 0 4 -13
3A
12 -8 6 10
0 -11 7.5 0.5
0 0 4/11 -10/11
0 0 4 -13
3A
3 343 max 3: 4,3A A
3 1,2,4,3P
12 -8 6 10
0 -11 7.5 0.5
0 0 4 -13
0 0 4/11 -10/11
3 33P A A
1 2 1 2 3 1 2 3 1 2 3 1 2 32 1 3 3 3 3 3
T T TP PA L A L L A L L P P A L L P P A L L P A
1 2 3 2 33 2 1 3 3
TPP PA P L L P A L A 3 2 1 1,2,4,3 2,3,1,4 2,3,4,1P P P where
1
0.25 1
0.5 -2/11 1
-0.5 1
1 2L L
3 1, 2,4,3P
1 23TL L P
Interchange columns 3,4
1
0.25 1
0.5 -2/11 1
-0.5 0 1
1 23TL L P
1 23 3
TP L L P
Interchange rows 3.4
1
0.25 1
-0.5 0 1
0.5 -2/11 1
1 23 3
TP L L P
PA = LU: partial pivoting [4]
12 -8 6 10
3 -13 9 3
-6 4 1 -18
6 -2 2 4
3 2 1PP PA
1
0.25 1
-0.5 0 1
0.5 -2/11 1
2L
12 -8 6 10
0 -11 7.5 0.5
0 0 4 -13
0 0 4/11 -10/11
3A
verify
PA = LU: partial pivoting [5]
12 -8 6 10
0 -11 7.5 0.5
0 0 4 -13
0 0 4/11 -10/11
3A
1
1
1
1/11 1
3L
1 44 11
12 -8 6 10
0 -11 7.5 0.5
0 0 4 -13
0 0 0 3/11
4A
2 3 2 3 43 2 1PP PA L A L L A
3 2 1 2,3, 4,1P PP P where
PA LU
2 3L L L
1
0.25 1
-0.5 0 1
0.5 -2/11 1/11 1
4U A
12 -8 6 10
-11 7.5 0.5
4 -13
3/11
we have
0.5 1L
OutLine
• Weakness of A=LU• A=LU versus PA=LU• Pivoting strategy• Implementation of PA=LU
- commutability of GE matrix and permutation- recursive formula
• MATLAB usage
1 2 1 22 1 2 2
TP PA P L P A L A
1
0.5 1
0.25 1
-0.5 1
1L
2 1, 3, 2, 4P Interchange rows and columns 2, 3
1L
1
0.25 1
0.5 1
-0.5 1
12 2
TP L P
1 2 3 2 33 2 1 3 3
TPP PA P L L P A L A
1
0.25 1
0.5 -2/11 1
-0.5 1
1 2L L
3 1,2, 4,3P
1 23 3
TP L L P
Interchange rows and columns 3, 4
1
0.25 1
-0.5 0 1
0.5 -2/11 1
2L
Implementation issue: commutability of GE matrix and permutation [1]
1 12 2P L L P
1 2 22 2P L L L P
Implementation issue: commutability of GE matrix and permutation [2]
2 1 2 1 1k k k k kTk kL L P L L P L Observation: If we define 0L I , then
where
2 1k k W OL L
M I
1k
1k 1 1
1
1
k kW R
1,2, 1, , 1, , 1, , 1, ,kP k p k p k p n interchanges rows or columns
,k p 1,2,3, ,p k
,k p
11
1
1
11
1k
1k
k
p
k p
and
11
1
11
1
2 1k k Tk kP L L P
interchange
k th p th
Implementation issue: commutability of GE matrix and permutation [3]
11
1
1
11
1k
1k
k
p
k p
11
1
1
11
partial rows (k and p)
interchange
11
1
1
11
symmetric permutation on an Identity matrix
1: 1k n
we have compute 1 2 1 ,k k k kP A L L A 11 2 2 1
kk kP P P P P
1 111 1
1 11, 1 1,
, ,
0
0
0 0
n
k kk k k k n
k kk k k n
k knk nn
a a
a aA
a a
a a
update original matrix A
2 1k k W OL L
M I
1k
1k stores in lower triangle matrix L
Algorithm ( PA = LU ) [1]
Given full matrix n nA R , construct initial lower triangle matrix L I
for
use permutation vector P to record permutation matrix kP
let 1 :A A , 1 0L L I 0 1,2,3, ,P n and
1, 2, 1, , 1, , 1, , 1, ,kP k p k p k p n k k
kA P A
Algorithm ( PA = LU ) [2]
define permutation matrix
then after swapping rows ,k p , we have
00
k TA dc B
k row
k col
max c
compute 1k kkP P P
compute 1 2 1k k k Tk kL P L L P by swapping ,1: 1L k k and ,1: 1L p k
then 1 2 1k k k kP A L L A 1k k kP A L A
for 0 1, 2,3, ,P n
if 1k then
endif
swap row , :kA k k n and row , :kA p k n
max : ,k kpkA A k n k , 1, ,p k k n find a such that1
2
3
4
Algorithm ( PA = LU ) [3]
endfor
1 0 , 0 0
Tk T cdA d B B
B
0 10 /
k
IL
c I
k row
k col
by updating 1: , /L k n k c
by updating matrix 1: , 1:TcdA k n k n
then 1 2 1k k k kP A L L A 1 1k k k kP A L L A
decompose 1k k kA L A where5
(recursion is done)
Question: recursion implementation
• Initialization- check algorithm holds for k=1
• Recursion formula- check algorithm holds for k, if k-1 is true
• Termination condition- check algorithm holds for k=n
• Breakdown of algorithm- check which condition PA=LU fails
• Exception: algorithm works only for square matrix?
normal
abnormal
Extension of algorithm
n
n n n
n n1 2 3 1 1 2 34 5 6 4 1 3 67 8 9 7 2 1 0
A
m
m m m
m m
p p
1 2 3 4 5 1 1 2 3 4 56 7 8 9 10 6 1 5 10 15 201 19 13 14 17 1 3.4 1 24 41 56
A
Exception: algorithm works only for square matrix? [1]
m nCase 1:
n m p m Case 2:
n
n
p
n
n
p I
O
n
n
O p
pm n p n Case 3:
Exception: algorithm works only for square matrix? [2]
1 6 1 1 1 6 12 7 19 2 1 5 173 8 13 3 2 1 244 9 14 4 3 1.7 15 10 17 5 4 2.3 1
A
B L O U LU LA U
C M I O MU M
we can simplify it as
Question: what is termination condition of case 3 ?
OutLine
• Weakness of A=LU• A=LU versus PA=LU• Pivoting strategy• Implementation of PA=LU• MATLAB usage
- abs- max
max : ,k kpkA A k n k
Command “abs”: take absolute value
In PA=LU algorithm, we need to take absolute value of one
column vector for pivoting
abs also works for matrix
Command “max”: take maximum value
5 3 max 1: 3A A
3 1 max 1: 3A A