gaussian elimination · 2010. 2. 18. · 3.3 towards gauss transforms (ge, take 3) we start by...

Chapter 3Gaussian Elimination

In this chapter, we again motivate matrix notation and the matrix-vector multiplicationoperation, but this time by looking at how we solve systems of linear equations.

3.1 Solving a System of linear Equations via Gaussian Elim-ination (GE, Take 1)

Consider the system of linear equations

2x + 4y− 2z =−104x− 2y + 6z = 206x− 4y + 2z = 18

Notice that x, y, and z are just variables, for which we can pick any name we want. To beconsistent with the notation we introduced previously for naming components of vectors, weuse the names χ0, χ1, and and χ2 instead of x, y, and z, respectively:

2χ0 + 4χ1− 2χ2 =−104χ0− 2χ1 + 6χ2 = 206χ0− 4χ1 + 2χ2 = 18

Solving this linear system relies on the fact that its solution does not change if

1. Equations are reordered (not actually used in this example); and/or

2. An equation in the system is modified by subtracting a multiple of another equationin the system from it; and/or

3. Both sides of an equation in the system are scaled by a nonzero.

The following steps are knows as Gaussian elimination. They transform a system of linearequations to an equivalent upper triangular system of linear equations:

93

94 Chapter 3. Gaussian Elimination

• Subtract λ10 = (4/2) = 2 times the first equation from the second equation:

Before After

2χ0 + 4χ1− 2χ2 =−104χ0− 2χ1 + 6χ2 = 206χ0− 4χ1 + 2χ2 = 18

2χ0 + 4χ1− 2χ2 =−10− 10χ1 + 10χ2 = 40

6χ0− 4χ1 + 2χ2 = 18

• Subtract λ20 = (6/2) = 3 times the first equation from the third equation:

Before After

2χ0 + 4χ1− 2χ2 =−10− 10χ1 + 10χ2 = 40

6χ0− 4χ1 + 2χ2 = 18

2χ0 + 4χ1− 2χ2 =−10− 10χ1 + 10χ2 = 40− 16χ1 + 8χ2 = 48

• Subtract λ21 = ((−16)/(−10)) = 1.6 times the second equation from the third equa-tion:

Before After

2χ0 + 4χ1− 2χ2 =−10− 10χ1 + 10χ2 = 40− 16χ1 + 8χ2 = 48

2χ0 + 4χ1− 2χ2 =−10− 10χ1 + 10χ2 = 40

− 8χ2 =−16

This now leaves us with an upper triangular system of linear equations.

Remark 3.1 In the above Gaussian elimination procedure, λ10, λ20, and λ21 are calledthe multipliers.

The equivalent upper triangular system of equations is now solved via back substitution:

• Consider the last equation, − 8χ2 =−16. Scaling both sides by by 1/(−8) we findthat

χ2 = −16/(−8) = 2.

• Next, consider the second equation, − 10χ1 + 10χ2 = 40. We know that χ2 = 2, whichwe plug into this equation to yield − 10χ1 + 10(2) = 40. Rearranging this we find that

χ1 = (40− 10(2))/(−10) = −2.

• Finally, consider the first equation, 2χ0 + 4χ1− 2χ2 =−10 We know that χ2 = 2and χ1 = −2, which we plug into this equation to yield 2χ0 + 4(−2)− 2(2) =−10.Rearranging this we find that

χ0 = (−10− (4(−2) + (−2)(−2)))/2 = 1.

3.2. Matrix Notation (GE, Take 2) 95

Gaussian elimination (transform to upper triangular system of equations)

Initial system of equations: Subtract λ10 = (4/2) = 2 times the firstrow from the second row:

2χ0 + 4χ1− 2χ2 =−104χ0− 2χ1 + 6χ2 = 206χ0− 4χ1 + 2χ2 = 18

2χ0 + 4χ1− 2χ2 =−10− 10χ1 + 10χ2 = 40

6χ0− 4χ1 + 2χ2 = 18Subtract λ20 = (6/2) = 3 times the firstrow from the third row:

Subtract λ21 = ((−16)/(−10)) = 1.6 timesthe second row from the third row:

2χ0 + 4χ1− 2χ2 =−10− 10χ1 + 10χ2 = 40− 16χ1 + 8χ2 = 48

2χ0 + 4χ1− 2χ2 =−10− 10χ1 + 10χ2 = 40

− 8χ2 =−16

Back substitution (solve upper triangular system of equations)

− 8χ2 =−16 implies χ2 = (−16)/(−8) = 2− 10χ1 + 10χ2 = 40 implies− 10χ1 + 10(2) = 40 and hence

χ1 = (40− 10(2))/(−10) = −22χ0 + 4χ1− 2χ2 =−10 implies2χ0 + 4(−2)− 2(2) =−10 and hence

χ0 = (−10− 4(−2) + 2(2))/(2) = 1Solution equals x =

χ0

χ1

χ2

=

1−2

2

Check the answer (by plugging χ0 = 1, χ1 = −2, and χ2 = 2 into the original system)2(1) + 4(−2)− 2(2) =−10 !4(1)− 2(−2) + 6(2) = 20 !6(1)− 4(−2) + 2(2) = 18 !

Figure 3.1: Summation of the steps that yield the solution via Gaussian elimination andbackward substitution for the example in Section 3.1.

Thus, the solution is the vector x =

χ0

χ1

χ2

=

1−2

2

. All the above discussed steps are

summarized in Figure 3.1.

3.2 Matrix Notation (GE, Take 2)

Now, in the above example, it becomes very cumbersome to always write the entire equation.The information is encoded in the coefficients in front of the χi variables, and the values tothe right of the equal signs. Thus, we could just let

2 4 −2 −104 −2 6 206 −4 2 18

represent2χ0 + 4χ1− 2χ2 =−104χ0− 2χ1 + 6χ2 = 206χ0− 4χ1 + 2χ2 = 18

Then Gaussian elimination can simply work with this array of numbers as illustrated inFigure 3.2.


Gaussian elimination (transform to upper triangular system of equations)


2 4 −2 −104 −2 6 206 −4 2 18

2 4 −2 −100 −10 10 406 −4 2 18

Subtract λ20 = (6/2) = 3 times the firstrow from the third row:


2 4 −2 −100 −10 10 400 −16 8 48

2 4 −2 −100 −10 10 400 0 −8 −16


The last row is shorthand for− 8χ2 =−16 which implies

χ2 = (−16)/(−8) = 2

The second row is shorthand for− 10χ1 + 10χ2 = 40 which implies− 10χ1 + 10(2) = 40 and hence

χ1 = (40− 10(2))/(−10) = −2The first row is shorthand for2χ0 + 4χ1− 2χ2 =−10 which implies2χ0 + 4(−2)− 2(2) =−10 and hence

χ0 = (−10− 4(−2) + 2(2))/(2) = 1

Solution equals x =

χ0

χ1

χ2

=

1−2

2


Figure 3.2: Summation of the steps that yield the solution via Gaussian elimination andbackward substitution for the example in Section 3.1. In this figure, a shorthand (matrixnotation) is used to represent the system of linear equations. Compare and contrast toFigure 3.1.

The above discussion motivates storing only the coefficients of a linear system (the num-bers to the left of the |) as a two dimensional array and the numbers to the right as a onedimension array. We recognize this two dimensional array as a matrix: A ∈ Rm×n is the twodimensional array of scalars

A =

α0,0 α0,1 · · · α0,n−1

α1,0 α1,1 · · · α1,n−1...

.... . .

...αm−1,0 αm−1,1 · · · αm−1,n−1

,

where αi,j ∈ R for 0 ≤ i < m and 0 ≤ j < n. It has m rows and n columns. Note that theparentheses are simply there to delimit the array rather than having any special meaning.

3.3. Towards Gauss Transforms (GE, Take 3) 97

We similarly recognize that the one dimensional array is a (column) vector x ∈ Rn where

x =

χ0

χ1...

χn−1

.

The length of the vector is n.Now, given A ∈ Rm×n and vector x ∈ Rn, the notation Ax stands for

α0,0χ0+ α0,1χ1+ · · ·+ α0,n−1χn−1

α1,0χ0+ α1,1χ1+ · · ·+ α1,n−1χn−1...

......

...αm−1,0χ0+ αm−1,1χ1+ · · ·+ αm−1,n−1χn−1

(3.1)

which is itself a vector of length m. We already encountered this operation: it is the by nowfamiliar matrix-vector multiply or matrix-vector product.

Now a linear system of m equations in n unknowns can be written as Ax = y, whereA ∈ Rm×n, x ∈ Rn, and y ∈ Rm. For example, the problem in Section 3.1 could be writtenas

2 4 −24 −2 66 −4 2

χ0

χ1

χ2

=

−10

2018

3.3 Towards Gauss Transforms (GE, Take 3)

We start by showing that the various steps in Gaussian elimination can be instead presentedas multiplication by a carefully chosen matrix. Compare and contrast the examples inExample 3.2 and Exercise 3.3 with the steps of Gaussian elimination in Figure 3.2.

Example 3.2

1 0 0−2 1 0

0 0 1

2 4 −24 −2 66 −4 2

=

2 4 −20 −10 106 −4 2

.

Notice that multiplication by the given matrix is the same as performing an axpy with the

negative of the given multiplier, λ1,0 = 2, the first row, and the second row.


Gaussian elimination, Take 3


2 4 −2 −104 −2 6 206 −4 2 18

1 0 0−2 1 0

0 0 1

2 4 −2 −104 −2 6 206 −4 2 18

=

2 4 −2 −100 −10 10 406 −4 2 18

Subtract λ20 = (6/2) = 3 times the firstrow from the third row:


1 0 00 1 0−3 0 1

2 4 −2 −100 −10 10 406 −4 2 18

=

2 4 −2 −100 −10 10 400 −16 8 48

1 0 00 1 00 −1.6 1

2 4 −2 −100 −10 10 400 −16 8 48

=

2 4 −2 −100 −10 10 400 0 −8 −16


The last row is shorthand for− 8χ2 =−16 which implies

χ2 = (−16)/(−8) = 2

The second row is shorthand for− 10χ1 + 10χ2 = 40 which implies− 10χ1 + 10(2) = 40 and hence

χ1 = (40− 10(2))/(−10) = −2The first row is shorthand for2χ0 + 4χ1− 2χ2 =−10 which implies2χ0 + 4(−2)− 2(2) =−10 and hence

χ0 = (−10− 4(−2) + 2(2))/(2) = 1

Solution equals x =

χ0

χ1

χ2

=

1−2

2


Figure 3.3: Summation of the steps that yield the solution via Gaussian elimination andbackward substitution for the example in Section 3.1. In this figure, we show how the stepsof Gaussian Elimination can be formulated as multiplications by the special matrices Li,j.

3.3. Towards Gauss Transforms (GE, Take 3) 99

Exercise 3.3 Compute

1 0 00 1 0−3 0 1

2 4 −20 −10 106 −4 2

.

How can this be described as an axpy operation?

Theorem 3.4 Let i > j and let Li,j be the matrix that equals the identity, except thatthe (i, j) element has been replaced with −λi,j. Then Li,jA equals the matrix A exceptthat the ith row is modified by subtracting λi,j times the jth row from it.

Proof: Let

Li,j =

Ij 0 0 0 00 1 0 0 00 0 Ii−j−1 0 00 −λi,j 0 1 00 0 0 0 In−i−1

A =

A0:j−1

aTj

Aj+1:i−1

aTi

Ai+1:m−1

,

where Ik equals a k×k identity matrix, As:t equals the matrix that consists of rows s throught from matrix A, and aT

k equals the kth row of A. Then

Li,jA =

Ij 0 0 0 00 1 0 0 00 0 Ii−j−1 0 00 −λi,j 0 1 00 0 0 0 In−i−1

A0:j−1

aTj

Aj+1:i−1

aTi

Ai+1:m−1

=

A0:j−1

aTj

Aj+1:i−1

−λi,j aTj + aT

i

Ai+1:m−1

=

A0:j−1

aTj

Aj+1:i−1

aTi − λi,j aT

j

Ai+1:m−1

.

Example 3.5 The steps of Gaussian elimination already given in Figures 3.1 and 3.2 arenow repeated with the insights in this section, in Figure 3.3.


Exercise 3.6 Verify that in Figure 3.3

1 0 00 1 00 −1.6 1

1 0 00 1 0−3 0 1

1 0 0−2 1 0

0 0 1

2 4 −2 −104 −2 6 206 −4 2 18

=

2 4 −2 −100 −10 10 400 0 −8 −16

In Figure 3.4, we use the insights in this section to show that one can perform GaussianElimination with just the matrix first and then on the right-hand side. This is important,because often one has multiple linear systems of equations that differ only in the vector onthe right-hand side. The important insight is that the multipliers can be stored in the placein which a new zero is introduced at each step.


1 0 00 1 00 −1.6 1

1 0 00 1 0−3 0 1

1 0 0−2 1 0

0 0 1

2 4 −24 −2 66 −4 2

=

2 4 −20 −10 100 0 −8

and

1 0 00 1 00 −1.6 1

1 0 00 1 0−3 0 1

1 0 0−2 1 0

0 0 1

−10

2018

=

−10

40−16

.

3.4 Gauss Transforms (GE, Take 4)

Next, we show how eliminating (zeroing) all elements in the matrix below the diagonal canbe combined into a single step. Compare and contrast the following example with those inExample 3.2 and Exercise 3.3.

Example 3.8

1 0 0−2 1 0−3 0 1

2 4 −24 −2 66 −4 2

=

2 4 −20 −10 100 −16 8

3.4. Gauss Transforms (GE, Take 4) 101

Gaussian elimination, Take 3b,separating the factorization of the matrix from the updating of the right-hand side

Initial matrix:Subtract λ10 = (4/2) = 2 times the firstrow from the second row, overwriting theelemented that is “zeroed”:

2 4 −24 −2 66 −4 2

1 0 0−2 1 0

0 0 1

2 4 −24 −2 66 −4 2

=

2 4 −22 −10 106 −4 2

Subtract λ20 = (6/2) = 3 times the firstrow from the third row, overwriting the el-emented that is “zeroed”:

Subtract λ21 = ((−16)/(−10)) = 1.6 timesthe second row from the third row, over-writing the elemented that is “zeroed”:

1 0 00 1 0−3 0 1

2 4 −22 −10 106 −4 2

=

2 4 −22 −10 103 −16 8

1 0 00 1 00 −1.6 1

2 4 −22 −10 103 −16 8

=

2 4 −22 −10 103 1.6 −8

Forward substitution (apply the transformations to the right-hand side vector)

Initial right-hand side vector: Subtract λ10 = 2 times the first row fromthe second row:

−10

2018

1 0 0−2 1 0

0 0 1

−10

2018

=

−10

4018

Subtract λ20 = 3 times the first row fromthe third row:

Subtract λ21 = 1.6 times the second rowfrom the third row:

1 0 00 1 0−3 0 1

−10

4018

=

−10

4048

1 0 00 1 00 −1.6 1

−10

4048

=

−10

40−16

Back substitution (solve upper triangular system of equations)As before

Check the answer (by plugging χ0 = 1, χ1 = −2, and χ2 = 2 into the original system)As before

Figure 3.4: Summation of the steps that yield the solution via Gaussian elimination andbackward substitution for the example in Section 3.1. In this figure, we show how thesteps of Gaussian Elimination can be formulated as multiplications by the special matricesLi,j. This time, Gaussian elimination is first applied to the matrix, and then applied to theright-hand side. Compare and contrast to Figure 3.3.


Gaussian elimination, Take 4,using Gauss transforms

Initial matrix:

Subtract λ10 = (4/2) = 2 times thefirst row from the second row and λ20 =(6/2) = 3 times the first row from the thirdrow, overwriting the elemented that is “ze-roed”:

2 4 −24 −2 66 −4 2

1 0 0−2 1 0−3 0 1

2 4 −24 −2 66 −4 2

=

2 4 −22 −10 103 −16 8

Subtract λ21 = ((−16)/(−10)) = 1.6 times the second row from the third row, over-writing the elemented that is “zeroed”:

1 0 00 1 00 −1.6 1

2 4 −22 −10 103 −16 8

=

2 4 −22 −10 103 1.6 −8

Forward substitution (apply the transformations to the right-hand side vector)

Initial right-hand side vector:Subtract λ10 = 2 times the first row fromthe second row and Subtract λ20 = 3 timesthe first row from the third row:

−10

2018

1 0 0−2 1 0−3 0 1

−10

2018

=

−10

4048

Subtract λ21 = 1.6 times the second row from the third row:

1 0 00 1 00 −1.6 1

−10

4048

=

−10

40−16

Back substitution (solve upper triangular system of equations)As before

Check the answer (by plugging χ0 = 1, χ1 = −2, and χ2 = 2 into the original system)As before

Figure 3.5: Summation of the steps that yield the solution via Gaussian elimination andbackward substitution for the example in Section 3.1. In this figure, we show how thesteps of Gaussian Elimination can be formulated as multiplications by the special matricesLi,j. This time, Gaussian elimination is first applied to the matrix, and then applied to theright-hand side. Compare and contrast to Figure 3.4.

3.4. Gauss Transforms (GE, Take 4) 103

We notice that this one multiplication has the same net effect as the two multiplications in

Example 3.2 and Exercise 3.3, performed in succession.

Theorem 3.9 Let Lj be a matrix that equals the identity, except that for i > jthe (i, j)elements (the ones below the diagonal in the jth column) have been replaced with −λi,j:

Lj =

Ij 0 0 0 · · · 00 1 0 0 · · · 00 −λj+1,j 1 0 · · · 00 −λj+2,j 0 1 · · · 0...

......

.... . .

...0 −λm−1,j 0 0 · · · 1

.

Then LjA equals the matrix A except that for i > j the ith row is modified by subtractingλi,j times the jth row from it. Such a matrix Lj is called a Gauss transform.

Proof: Let

Lj =

Ij 0 0 0 · · · 00 1 0 0 · · · 00 −λj+1,j 1 0 · · · 00 −λj+2,j 0 1 · · · 0...

......

.... . .

...0 −λm−1,j 0 0 · · · 1

A =

A0:j−1

aTj

aTj+1

aTj+2...

aTm−1

,

where Ik equals a k×k identity matrix, As:t equals the matrix that consists of rows s throught from matrix A, and aT

k equals the kth row of A. Then

LjA =

Ij 0 0 0 · · · 00 1 0 0 · · · 00 −λj+1,j 1 0 · · · 00 −λj+2,j 0 1 · · · 0...

......

.... . .

...0 −λm−1,j 0 0 · · · 1

A0:j−1

aTj

aTj+1

aTj+2...

aTm−1

=

A0:j−1

aTj

−λj+1,j aTj + aT

j+1

−λj+2,j aTj + aT

j+2...

−λm−1,j aTj + aT

m−1

=

A0:j−1

aTj

aTj+1 − λj+1,j aT

j

aTj+2 − λj+2,j aT

j...

aTm−1 − λm−1,j aT

j

.


Example 3.10 The steps of Gaussian elimination already given in Figures 3.1 –3.4 are nowrepeated with the insights in this section, in Figure 3.5.


1 0 00 1 00 −1.6 1

1 0 0−2 1 0−3 0 1

2 4 −24 −2 66 −4 2

=

2 4 −20 −10 100 0 −8

and

1 0 00 1 00 −1.6 1

1 0 0−2 1 0−3 0 1

−10

2018

=

−10

40−16

.

3.5 Gauss Transforms Continued (GE, Take 5)

Example 3.12 Now were are going to turn things around: Consider

1 0 0

−λ10 1 0−λ20 0 1

2 4 −24 −2 66 −4 2

=

2 4 −2

4− λ10(2) −2− λ10(4) 6− λ10(−2)6− λ20(2) −4− λ20(4) 2− λ20(−2)

(3.2)

The question we now ask is “how should λ10 and λ20 be chosen so that zeroes are introducedbelow the diagonal in the first column?”. Examining 4− λ10(2) and 6− λ20(2) we concludethat our familiar λ10 = 4/2 = and λ20 = 6/2 = 3 have the desired property.

Example 3.13 Alternatively, we can write (3.2) as

1

(0 0

)

−(

λ10

λ20

) (1 00 1

)

2

(4 −2

)(

46

) (−2 6−4 2

)

3.5. Gauss Transforms Continued (GE, Take 5) 105

=

2

(4 −2

)

−(

λ10

λ20

)2 +

(46

)−

(λ10

λ20

) (4 −2

)+

(−2 6−4 2

)

and note that to zero the elements below the diagonal in the first column we must ensure

that −(

λ10

λ20

)2 +

(46

)=

(00

)which means that

(λ10

λ20

)=

(46

)/2 =

(23

).

Now let us try to generalize this insight: Let A(0) ∈ Rn×n be a given matrix and L(0) aGauss transform which is to be determined. Partition

A(0) →(

α(0)11 a(0) T

12

a(0)21 A(0)

22

)and L(0) →

(1 0

−l(0)21 I

).

Then (1 0

−l(0)21 I

) (α(0)

11 a(0) T12

a(0)21 A(0)

22

)=

(α(0)

11 a(0) T12

a(0)21 − l(0)

21 α(0)11 A(0)

22 − l21a(0) T12

).

Now,

• We want to choose l(0)21 so that a(0)21 − l(0)

21 α(0)11 = 0, introducing zeroes below the diagonal

in the first column. Clearly, l(0)21 = a(0)21 /α(0)

11 has this property. In other words, l(0)21

results from dividing a(0)21 by 1/α(0)

11 .

• Moreover, we notice that the “rest of the matrix”, A(0)22 , is transformed to A(0)

22 −l(0)21 a(0) T

12 .

In other words, an outer product is subtracted from A(0)22 , which we before called a rank-

1 update (ger).

• After choosing l(0)21 as prescribed, let

A(1) :=

(1 0

−l(0)21 I

) (α(0)

11 a(0) T12

a(0)21 A(0)

22

)

=

(α(0)

11 a(0) T12

a(0)21 − l(0)21 α(0)

11 A(0)22 − l21a

(0) T12

)=

(α(1)

11 a(1) T12

0 A(1)22

).

Exercise 3.14 Identify α(0)11 , a(0)

21 , aT (0)12 , A(0)

22 , L(0), l(0)21 , and A(0)22 − l(0)21 a(0) T

12 in Example 3.12.


Example 3.15 Consider

1 0 0

0 1 00 −λ21 1

2 4 −2

0 −10 100 −16 8

=

2 4 −2

0 −10 100 −16− λ21(−10) 8− λ20(10)

The question we now ask is “how should λ21 be chosen so that zeroes are introduced belowthe diagonal in the second column?”. Examining −16 − λ21(−10) we conclude that ourfamiliar λ21 = −16/(−10) = 1.6 has the desired property. Alternatively, we notice that,viewed as a vector,

(λ21

)=

(−16

)/(−10).

Continuing our more general discussion, partition

A(1) →

A(1)

00 a(1)01 A(1)

02

0 α(1)11 a(1) T

12

0 a(1)21 A(1)

22

and L(1) →

I1 0 0

0 1 0

0 −l(1)21 I

.

Then

I1 0 0

0 1 0

0 −l(1)21 I

A(1)

00 a(1)01 A(1)

02

0 α(1)11 a(1) T

12

0 a(1)21 A(1)

22

=

A(1)

00 a(1)01 A(1)

02

0 α(1)11 a(1) T

12

0 a(1)21 − l(1)

21 α(1)11 A(1)

22 − l(1)21 a(1) T12

.

Now,

• We want to choose l(1)21 so that a(1)21 − l(1)

21 α(1)11 = 0, introducing zeroes below the diagonal

in the first column. Clearly, l(1)21 = a(1)21 /α(1)

11 has this property. In other words, l(1)21

results from dividing a(1)21 by 1/α(1)

11 .

• Moreover, we notice that the “rest of the matrix”, A(1)22 , is transformed to A(1)

22 −l(1)21 a(1) T

12 .

In other words, an outer product is subtracted from A(1)22 , which we before called a rank-

1 update (ger).

• After choosing l(1)21 as prescribed, let

A(2) =

I1 0 0

0 1 0

0 −l(1)21 I

A(1)

00 a(1)01 A(1)

02

0 α(1)11 a(1) T

12

0 a(1)21 A(1)

22

=

A(1)

00 a(1)01 A(1)

02

0 α(1)11 a(1) T

12

0 0 A(2)22

Exercise 3.16 Identify A(1)00 , . . . , A(1)

22 , L(1), l(1)21 , and A(1)

22 − l(1)21 a(1) T12 in Example 3.15.

3.5. Gauss Transforms Continued (GE, Take 5) 107

More general yet, assume that this progression of computations has proceded to where

A(k) →

A(k)

00 a(k)01 A(k)

02

0 α(k)11 a(k) T

12

0 a(k)21 A(k)

22

and L(k) →

Ik 0 0

0 1 0

0 −l(k)21 I

,

where A(k)00 and Ik are k × k matrices. Then

Ik 0 0

0 1 0

0 −l(k)21 I

A(k)

00 a(k)01 A(k)

02

0 α(k)11 a(k) T

12

0 a(k)21 A(k)

22

=

A(k)

00 a(k)01 A(k)

02

0 α(k)11 a(k) T

12

0 a(k)21 − l(k)

21 α(k)11 A(k)

22 − l(k)21 a(k) T

12

.

Now,

• We want to choose l(k)21 so that a(k)

21 − l(k)21 α(k)

11 = 0, introducing zeroes below the diagonal

in the first column. Clearly, l(k)21 = a(k)

21 /α(k)11 has this property. In other words, l(k)

21

results from dividing a(k)21 by 1/α(k)

11 .

• Moreover, we notice that the “rest of the matrix”, A(k)22 , is transformed to A(k)

22 −l(k)21 a(k) T

12 .

In other words, an outer product is subtracted from A(k)22 , which we before called a rank-

1 update (ger).

• After choosing l(k)21 as prescribed, let

A(k+1) =

I1 0 0

0 1 0

0 −l(k)21 I

A(k)

00 a(k)01 A(k)

02

0 α(k)11 a(k) T

12

0 a(k)21 A(k)

22

=

A(k)

00 a(k)01 A(k)

02

0 α(k)11 a(k) T

12

0 0 A(k+1)22

Now, if A ∈ Rn×n, then A(n) = L(n−1) · · · L(1)L(0)A = U , an upper triangular matrix. Also,to solve Ax = b, we note that

Ux = (L(n−1) · · · L(1)L(0)A)x = L(n−1) · · · L(1)L(0)b︸︷︷︸

b

.

The right-hand size of this we recognize as forward substitution applied to vector b. We willlater see that solving Ux = b where U is upper triangular is equivalent to back substitution.

The reason why we got to this point as “GE Take 5” is so that the reader, hopefully, nowrecognizes this as just Gaussian elimination. The insights in this section are summarized inthe algorithm in Figure 3.6, in which the original matrix A is overwritten with the uppertriangular matrix that results from Gaussian elimination and the strictly lower triangularelements are overwritten by the multipliers.


Algorithm: A := GE Take5 (A)

Partition A→(

ATL ATR

ABL ABR

)

where ATL is 0× 0while m(ATL) < m(A) do

Repartition(

ATL ATR

ABL ABR

)→

A00 a01 A02

aT10 α11 aT

12

A20 a21 A22

where α11 is 1× 1

a21 := a21/α11 (= l21)A22 := A22 − a21aT

12 (= A22 − l21aT12)

Continue with(

ATL ATR

ABL ABR

)←

A00 a01 A02

aT10 α11 aT

12

A20 a21 A22

endwhile

Figure 3.6: Algorithm for applying Gaussian elimination to matrix A, overwriting the matrixwith U on and above the diagonal, and the multipliers below the diagonal.

3.6 Toward the LU Factorization (GE Take 6)

Definition 3.17 (Inverse of a matrix) Let A ∈ Rn×n and B ∈ Rn×n have the propertythat AB = BA = I. Then B is said to be the inverse of matrix A and is denoted by A−1.

Later we will discuss when a matrix has an inverse and show that for square A and B itis always the case that if AB = I then BA = I, and that the inverse of a matrix is unique.

Example 3.18 Let

L =

1 0 0−2 1 0

0 0 1

and L =

1 0 02 1 00 0 1

.

Then

LL =

1 0 02 1 00 0 1

1 0 0−2 1 0

0 0 1

=

1 0 00 1 00 0 1

.

Notice that this should be intuitively true: LA subtracts two times the first row of A

3.6. Toward the LU Factorization (GE Take 6) 109

from the second row. LA adds two times the first row of A from the second row. Thus,LLA = L(LA) = A: the first transformation subtracts two times the first row from thesecond row, and the second transformation adds two times the first row back to the result,undoing the first transformation. Two transformations that always undo each other areinverses of each other.

Exercise 3.19 Compute

1 0 0−2 1 0−3 0 1

1 0 02 1 03 0 1

and reason why this should be intuitively true.

Theorem 3.20 Let

L =

Ik 0 0

0 1 00 −l21 I

be a Gauss transform, where I is a k × k identity matrix. Then

L =

Ik 0 0

0 1 00 l21 I

(itself a Gauss transform) is its inverse: LL = LL = I.

Proof:

LL =

Ik 0 0

0 1 00 −l21 I

Ik 0 0

0 1 00 l21 I

=

Ik 0 0

0 1 00 −l21 + Il21 I

=

Ik 0 0

0 1 00 0 I

= I.

Similarly LL = I. (Notice that when we use I without indicating its dimensions, it has thedimensions that are required to fit the situation.)

Exercise 3.21 To complete the above proof, show that LL = I.


Exercise 3.22 Recall that

1 0 00 1 00 −1.6 1

1 0 0−2 1 0−3 0 1

2 4 −24 −2 66 −4 2

=

2 4 −20 −10 100 0 −8

.

Show that

2 4 −24 −2 66 −4 2

=

1 0 02 1 03 0 1

1 0 00 1 00 1.6 1

2 4 −20 −10 100 0 −8

.

Exercise 3.23 Show that

1 0 02 1 03 0 1

1 0 00 1 00 1.6 1

=

1 0 02 1 03 1.6 1

so that

2 4 −24 −2 66 −4 2

=

1 0 02 1 03 1.6 1

2 4 −20 −10 100 0 −8

.

Theorem 3.24 Let L(0), · · · , L(n−1) be the sequence of Gauss transforms that transforman n× n matrix A to an upper triangular matrix:

L(n−1) · · · L(0)A = U.

ThenA = L(0) · · ·L(n−2)L(n−1)U,

where L(j) = L(j)−1, the inverse of L(j).

Proof: IfL(n−1)L(n−2) · · · L(0)A = U.

then

A = L(0) · · · L(n−2) L(n−1)L(n−1)︸︷︷︸

I

L(n−2)

︸︷︷︸I

· · · L(0)

︸︷︷︸I

A = L(0) · · ·L(n−2)L(n−1)U.

3.6. Toward the LU Factorization (GE Take 6) 111

Lemma 3.25 Let L(0), . . . , L(n−1) be the sequence of Gauss transforms that transformsa matrix A into an upper triangular matrix U :

L(n−1) · · · L(0)A = U

and let L(j) = L(j)−1. Then L(k) = L(0) · · ·L(k−1)L(k) has the structure

L(k) =

(L(k)

TL 0

L(k)BL I

)

where L(k)TL is a (k + 1)× (k + 1) unit lower trianglar matrix.

Proof: Proof by induction on k.

Base case: k = 0. L(0) = L(0) =

(1 0

l(0)21 I

)meets the desired criteria since 1 is a trivial

unit lower triangular matrix.

Inductive step: Assume L(k) meets the indicated criteria. We will show that then L(k+1)

does too. Let

L(k) =

(L(k)

TL 0

L(k)BL I

)=

L(k)

00 0 0

l(k) T10 1 0

L(k) T20 0 I

where LTL (and hence L00) are unit lower triangular matrices of dimension (k + 1)×(k + 1). Then

L(k+1) = L(k)L(k+1) =

L(k)

00 0 0

l(k) T10 1 0

L(k) T20 0 I

Ik+1 0 0

0 1 0

0 l(k+1)21 I

=

L(k)

00 0 0

l(k) T10 1 0

L(k) T20 l(k+1)

21 I

=

(L(k)

00 0

l(k) T10 1

)0

(L(k) T

20 l(k+1)21

)I

=

(L(k+1)

TL 0

L(k+1)BL I

),

which meets the desired criteria since L(k+1)TL is unit lower triangular.

By the Principle of Mathematical Induction the result holds for L(j), 0 ≤ j < n− 1.


Corollary 3.26 Under the conditions of Lemma 3.25, L = L(n−1) is a unit lower trian-gular matrix the strictly lower triangular part of which is the sum of all the strictly lowertriangular parts of L(0), . . . , L(n−1):

L = L(n−1) =

1 0

l(0)21

1 0

l(2)21

. . .1 0

l(n−2)21 1

.

(Note that l(n−1)21 is a vector of length zero, so that the last step of involving L(n−1) is

really a “no op”.)

Example 3.27 A consequence of this corollary is that the fact that

1 0 02 1 03 0 1

1 0 00 1 00 1.6 1

=

1 0 02 1 03 1.6 1

in Exercise 3.23 is not a coincidence: For these matrices all you have to do find the strictlylower triangular parts of the right-hand side is to move the nonzeroes below the diagonalin the matrices on the left-hand side to the corresponding elements in the matrix on theright-hand side of the equality sign.

Exercise 3.28 The order in which the Gauss transforms appear is important. In particular,verify that

1 0 00 1 00 1.6 1

1 0 02 1 03 0 1

'=

1 0 02 1 03 1.6 1

.

Theorem 3.29 Let L(0), . . . , L(n−1) be the sequence of Gauss transforms that transformsan n×n matrix A into an upper triangular matrix U : L(n−1) · · · L(0)A = U and let L(j) =L(j)−1. Then A = LU , where L = L(0) · · ·L(n−1) is a unit lower triangular matrix and canbe easily obtained from L(0), . . . , L(n−1) by the observation summarized in Corollary 3.26.

3.7. Coding Up Gaussian Elimination 113

Remark 3.30 Notice that Theorem 3.29 does not say that for every square matrix Gaus-sian elimination is well-defined. It merely says that if Gaussian elimination as presentedthus far completes, then there is a unit lower triangular matrix L and upper triangularmatrix U such that A = LU .

3.7 Coding Up Gaussian Elimination

Let us look at GE Take 3b, which overwrites a matrix with the upper triangular matrix thatresults from Gaussian elimination while overwriting the lower triangular part of that matrixwith the multipliers. Assuming we start with an n×n matrix A, the algorithm and M-scriptcode are given in Figure 3.7.

• Think of the loop indexed by j as tracking the diagonal element of the matrix: αj,j

(A(j,j)) identifies how many columns we have already processed by zeroing elementsbelow the diagonal.

• The loop indexed by i computes the multiplier λi,j, multiplies the jth row by thatmultiplier, and then subtracts this from the ith row. Notice that this only is done withthe part of the row to the right of the jth column, since to the left we already haveintroduced zeroes and have overwritten those zeroes with the multipliers.

• The loop indexed by p performs the “multiply the jth row by λi,j (which has overwrittenαi,j).

Now, we can recognize the operation “multiply the jth row by a constant and subtractit from the ith row” as an axpy operation. In Figure 3.8 we illustrate how the loop indexedby p implements that axpy operation. Next, in Figure 3.9 we rearrange the algorithm andcode in Figure 3.7 to compute the multipliers (which overwrite ai,j) first, before moving onto update rows j+1, j+2, . . . with the double nested loops indexed with i and p. This allowsus to then notice that order of the double-nested loop indexed by i and p can be reversed,in Figure 3.10. Again, the inner-most loop can then be recognized as an axpy operation, inFigure 3.11. Finally, we notice that the inner loops indexed by i and p in fact implement arank-1 update, in Figure 3.12.

We would like to think that it is easier to reason about the algorithm and code in Fig-ures 3.6 and 3.13, which are an alternative representation for the algorithms in Figure 3.12.


for j = 0, . . . , n− 1for i = j + 1, . . . , n− 1

αi,j := αi,j/αj,j

for p = j + 1, . . . , n− 1αi,p := αi,p − αi,jαj,p

endforendfor

endfor

for j=1:nfor i=j+1:nA(i,j) = A(i,j)/A(j,j);for p=j+1:n

A(i,p) = A(i,p) + A(i,j) * A(j,p);end

endend

Figure 3.7: Algorithm and M-script for GE Take 3b.

for j = 0, . . . , n− 1for i = j + 1, . . . , n− 1


αi,j+1:n−1 := αi,j+1:n−1 − αi,j ∗ αj,j+1:n−1

endforendfor

for j=1:nfor i=j+1:nA(i,j) = A(i,j)/A(j,j);A(i,j+1:n) = ...

SLAP_Axpy( - A(i,j), A(j,j+1:n), ...A(i,j+1:n));

endend

Figure 3.8: Algorithm and M-script for GE Take 3b, with the inner loop (indexed by p)identified as an axpy operation.

for j = 0, . . . , n− 1for i = j + 1, . . . , n− 1


endforfor i = j + 1, . . . , n− 1

for p = j + 1, . . . , n− 1αi,p := αi,p − αi,jαj,p

endforendfor

endfor

for j=1:nfor i=j+1:nA(i,j) = A(i,j)/A(j,j);

endfor i=j+1:nfor p=j+1:n


endend

Figure 3.9: The algorithm and M-script for GE Take 3b in Figure 3.7, but with the multiplierscomputed before the rest of the rows are updated. Notice that now the inner two loops(indexed by i and p) can be done in either order.

3.7. Coding Up Gaussian Elimination 115

for j = 0, . . . , n− 1for i = j + 1, . . . , n− 1


endforfor p = j + 1, . . . , n− 1

for i = j + 1, . . . , n− 1αi,p := αi,p − αi,jαj,p

endforendfor

endfor

for j=1:nfor i=j+1:nA(i,j) = A(i,j)/A(j,j);

endfor p=j+1:nfor i=j+1:n


endend

Figure 3.10: The algorithm and M-script for GE Take 3b in Figure 3.9, with the order ofthe inner two loops (indexed by i and p) reversed.

for j = 0, . . . , n− 1αj+1:n−1,j := αj+1:n−1,j/αj,j

for p = j + 1, . . . , n− 1αj+1:n−1,p := αj+1:n−1,p − αj,pαj+1:n−1,j

endforendfor

for j=1:nSLAP_Scal( 1/A(j,j), A(j+1:n,j) );for p=j+1:nA(j+1:n,p) = ...

SLAP_Axpy( A(j,p), A(j+1:n,j), ...A(j+1:n,p) );

endend

Figure 3.11: The algorithm and M-script for GE Take 3b in Figure 3.10, recognizing thatfirst loop indexed by i is a scal operation and the second loop indexed by i is an axpyoperation.

for j = 0, . . . , n− 1αj+1:n−1,j := αj+1:n−1,j/αj,j

αj+1:n−1,j+1:n−1 :=αj+1:n−1,j+1:n−1 − αj+1:n−1,jαj,j+1:n−1

endfor

for j=1:nSLAP_Scal( 1/A(j,j), A(j+1:n,j) );A(j+1:n,j+1:n) = SLAP_Ger( ...

-1, A(j+1:n,j), A(j,j+1:n), ...A(j+1:n,j+1:n) );

endend

Figure 3.12: The algorithm and M-script for GE Take 3b in Figure 3.11, recognizing that theloop indexed by p, which calls the axpy operations, implements the updating of the indi-cated matrix by subtracting off an outer product (combined this is called a rank-1 update).Compare and contrast also with the algorithm in Figure 3.6.


function [ A_out ] = GE_Take5( A )

[ ATL, ATR, ...ABL, ABR ] = FLA_Part_2x2( A,

0, 0, ’FLA_TL’ );

while ( size( ATL, 1 ) < size( A, 1 ) )[ A00, a01, A02, ...

a10t, alpha11, a12t, ...A20, a21, A22 ] = ...

FLA_Repart_2x2_to_3x3( ATL, ATR, ...ABL, ABR, 1, 1, ’FLA_BR’ );

%-----------------------------------------------------%% a21 = a21 / alpha11;a21 = SLAP_Scal( 1 / alpha11, a21 );

% A22 = A22 - a21 * a12t;A22 = SLAP_Ger( -1, a21, a12t, A22 );%-----------------------------------------------------%[ ATL, ATR, ...

ABL, ABR ] =FLA_Cont_with_3x3_to_2x2( A00, a01, A02, ...

a10t, alpha11, a12t, ...A20, a21, A22,’FLA_TL’ );

end

A_out = [ ATL, ATRABL, ABR ];

return

Figure 3.13: FLAME@lab code for algorithm in Figure 3.6. Compare and contrast also withthe code in Figure 3.12. Two loops are hidden inside of the call to FLA Ger.

gaussian elimination · 2010. 2. 18. · 3.3 towards gauss transforms (ge, take 3) we start by...

Documents