math5316 lecture notes chapter 3 { least squares 3.1 the

MATH5316 Lecture NotesDaniel R. Reynolds, SMU MathematicsSpring 2019

Chapter 3 – Least Squares

We’ll now focus on what appears to be a simple problem in data fitting from Introductionto Scientific Computing; however, the techniques that are used here arise frequently in awide range of algorithms for linear algebra.

3.1 The Discrete Least Squares Problem

Suppose that we have data points {(ti, yi)}mi=1, that we believe adhere to a functionalrelationship

x1ϕ1(ti, yi) + x2ϕ2(ti, yi) + . . .+ xnϕn(ti, yi) = b(ti, yi), i = 1, . . . ,m

⇔n∑j=1

xjϕj(ti, yi) = b(ti, yi), i = 1, . . . ,m

⇔ ϕ1(t1, y1) ϕ2(t1, y1) · · · ϕn(t1, y1)ϕ1(t2, y2) ϕ2(t2, y2) · · · ϕn(t2, y2)

......

...ϕ1(tm, ym) ϕ2(tm, ym) · · · ϕn(tm, ym)

x1x2...xn

=

b(t1, y1)b(t2, y2)

...b(tm, ym)

⇔

Ax = b,

for A ∈ �m×n, b ∈ �m and x ∈ �n.

Notes:

• Perhaps the simplest example is the line of best fit:

ϕ1(t, y) = t, ϕ2(t, y) = 1, b(t, y) = y.

• Other polynomials may also be easily constructed:

ϕj(t, y) = tj−1, b(t, y) = y.

MATH5316 Lecture Notes Chapter 3 – Least Squares

• Other possibilities exist as well, e.g.,

y = eat+b ⇔ ln(y) = at+ b

⇒ϕ1(t, y) = t, ϕ2(t, y) = 1, b(t, y) = ln(y).

• Typically, m� n, i.e., there is more data than parameters, so the system is overde-termined.

• In science and engineering, such problems typically arise when performing experi-ments, so even if the model is perfect, experimental error will ensure that there isno “solution” to the overdetermined linear system.

Our goal is then to fin the set of coefficients x ∈ �n so that the model “best fits” thedata. To this end, we define the linear residual as usual,

r = b− Ax. (1)

While we would like to find x ∈ �n such that r = 0, this is typically not possible, soinstead we choose to find x ∈ �n that minimizes ‖r‖2, i.e.,

x = argminx∈�n ‖r‖2 = argminx∈�n ‖r‖22 = argminx∈�n1

2‖r‖22. (2)

While other norms are possible to use, the resulting optimization problems typicallybecome much more difficult.

The problem (2) is called the discrete least squares problem – it is “discrete” because wehave finite data, {(ti, yi)}mi=1.

3.2 Orthogonal Matrices, Rotators, and Reflectors

For now we’ll consider real-valued problems, although the techniques in this chapter maybe extended to the complex case as well.

Definition 3.2.1 (Inner product). An inner product on the vector space �n is a function〈·, ·〉 : �n×�n → � satisfying the following properties for all x, y, z ∈ �n and all α ∈ �:

(i) 〈x, x〉 ≥ 0, and 〈x, x〉 = 0 iff x = 0.

(ii) 〈x, y〉 = 〈y, x〉.

(iii) 〈αx, y〉 = α 〈x, y〉.

(iv) 〈x+ y, z〉 = 〈x, z〉+ 〈y, z〉.

Note:

D.R. Reynolds, SMU Mathematics 67


• The standard Euclidean inner product is the dot product

〈x, y〉 = xTy = yTx =n∑i=1

xi yi = x · y.

This inner product will be used throughout the remainder of this course, unlessspecified otherwise.

• The Cauchy-Schwarz inequality maybe equivalently written:

| 〈x, y〉 | ≤ ‖x‖ ‖y‖

for any norm that is “induced” from the inner product, ‖x‖ = 〈x, x〉1/2. Clearly,the 2-norm is induced from the Euclidean inner product above.

• An equivalent definition of the Euclidean inner product is given in Physics courses:Let θ be the angle between two vectors x and y, then

cos(θ) =〈x, y〉‖x‖2 ‖y‖2

,

or equivalently,

θ = cos−1〈x, y〉‖x‖2 ‖y‖2

,

where if either x = 0 or y = 0 the angle is defined to be θ = π2.

• If x and y are orthogonal, then θ = π2

and 〈x, y〉 = 0.

Definition 3.2.2 (Orthogonal). A matrix Q ∈ �n×n is called orthogonal iff QQT =

QTQ = In.

Note that QTQ = In states that the columns of Q are themselves orthonormal, i.e.,〈qi, qj〉 = δij.

Theorem 3.2.3. If Q ∈ �n×n is orthogonal, then for all x, y ∈ �n,

(a) 〈Qx, Qy〉 = 〈x, y〉, and

(b) ‖Qx‖2 = ‖x‖2.

Proof. Both results follow directly from the definition of orthogonal matrix, along withthe definition of the Euclidean inner product. For (a), we have

〈Qx, Qy〉 = (Qx)T (Qy) = xTQTQy = xT Iny = xTy = 〈x, y〉 .

Similarly, for part (b) we have

‖Qx‖2 = 〈Qx, Qx〉1/2 = 〈x, y〉1/2 = ‖x‖2.



Lemma 3.2.4.

cos−1〈Qx, Qy〉‖Qx‖2 ‖Qy‖2

= cos−1〈x, y〉‖x‖2 ‖y‖2

.

Notes:

• Part (b) above states that length is preserved under orthogonal transformations(multiplication by orthogonal matrices).

• The Lemma above states that angles are preserved under orthogonal transforma-tions.

• Our least squares problem may therefore be rewritten:

x = argminx∈�n

1

2‖r‖22 = argminx∈�n

1

2‖Qr‖22 (3)

A few more properties of orthogonal matrices that we’ll use in this chapter; the proofs ofwhich are left for homework/exams:

Lemma 3.2.5. If Q ∈ �n×n is orthogonal, then so is QT .

Lemma 3.2.6. If P,Q ∈ �n×n are orthogonal, then so is the product, PQ.

Since orthogonal transformations have such nice properties, we like to build algorithmsout of them! To that end, we have three “typical” orthogonal transformations, each areoptimal for different purposes: permutations, rotations and reflections.

3.2.1 Rotators

For the moment, let’s consider �2. The matrix Q =

[cos θ − sin θsin θ cos θ

]is a standard rotation

matrix (studied in Intro. Linear Algebra), since it takes any vector x ∈ �2 and rotates itan angle θ around the origin via the transformation y = Qx.

For example, for θ = π4, we have

Q =

[√22−√22√

22

√22

], x =

[10

], x =

[0−1

]⇒ Qx =

[√22√22

], Qx =

[ √22

−√22

].

Furthermore, note that by its structure and the fact that cos is “even” while sin is “odd”,QT is also a rotator, that rotates by an angle of −θ.

We typically use rotators for the following type of operation: given a vector x ∈ �2, wemay determine an angle θ such that the product QTx is parallel to the x1 axis.



WLOG, let x =

[x1x2

]with x2 6= 0, and let Q =

[cos θ − sin θsin θ cos θ

]. We therefore wish to

find θ so that QTx =

[y0

]. Since

QTx =

[cos θ sin θ− sin θ cos θ

] [x1x2

]=

[x1 cos θ + x2 sin θ−x1 sin θ + x2 cos θ

],

then we obtain our result iff x1 sin θ = x2 cos θ. Choosing

cos θ =x1√x21 + x22

, and sin θ =x2√x21 + x22

,

we have that

cos2 θ + sin2 θ =x11 + x22x11 + x22

= 1,

i.e., θ is a valid angle, and

−x1 sin θ + x2 cos θ =−x1x2 + x1x2√

x21 + x22= 0

i.e., θ satisfies our desired goal.

Expanding our horizons slightly, let’s now consider the following related problem. Given

a matrix A =

[a1,1 a1,2a2,1 a2,2

], we wish to find a rotation matrix Q ∈ �2×2 such that QTA =

R =

[r1,1 r1,20 r2,2

]. From our previous example, we see that the matrix

Q =1√

a11,1 + a22,1

[a1,1 −a2,1a2,1 a1,1

]

should do the trick. Moreover, we have that QTA =

[r1,1 r1,20 r2,2

], so

r1,1 =1√

a11,1 + a22,1

[a1,1 a2,1

] [a1,1a2,1

]=

a11,1 + a22,1√a11,1 + a22,1

=√a11,1 + a22,1.

Moreover, since Q is orthogonal, then

QTA = R ⇔ A = QR,

this is called the QR-decomposition of A. Moreover, given this decomposition, we mayeasily solve the linear system Ax = b:

Ax = b ⇔ QRx = b ⇔ Rx = QT b,

so we require only the two steps

(1) Compute c = QT b (matrix-vector product)

(2) Solve Rx = c for x (upper-triangular linear system)



3.2.1.1 Extending to larger matrices

Obviously, there is more to life than �2, so how can we extend this idea to larger matrices,A ∈ �n×n?

Again, we may consider an angle 0 ≤ θ < 2π. For brevity, let’s denote c = cos θ ands = sin θ. We may then write a matrix Q ∈ �n×n with the form

Q =

1. . .

1c −s

1. . .

1s c

1. . .

1

← i

← j

↑ ↑i j

i.e., this is just In with a 2 × 2 rotator matrix embedded within rows/columns i andj.

Notes:

• This is called a plane rotator or a Givens rotator or a Jacobi rotator.

• For any x ∈ �n, we can transform y = Qx so that yj = 0 using the above matrixQ, and where we select

c =xi√x2i + x2j

, s =xj√x2i + x2j

.

• Left multiplication: the matrix products QA and QTA only alter the ith and jthrows of A (since the other rows of Q and QT correspond to In).

• Right multiplication: the matrix products BQ and BTQ only alter the ith and jthcolumns of B (again, since the other columns of Q and QT correspond to In).

• Geometrically, if we consider only the xixj-plane embedded in �n, this matrix Qrotates a vector within only that plane (by an angle of θ).

Theorem 3.2.7 (QR decomposition). Let A ∈ �n×n. Then there exists an orthogonalmatrix Q ∈ �n×n and an upper-triangular matrix R ∈ �n×n such that A = QR.



While we’ll wait to prove the above theorem until later in this section, we can sketch outthe proof/algorithm using rotators.

Consider the first column of A, given by

a1,1...an,1

. As shown above, we may construct a

rotator on the x1x2-plane, Q2,1, such that

QT2,1

a1,1a2,1a3,1

...an,1

=

#0a3,1

...an,1

.

In the same manner, we may construct Q3,1, . . . , Qn,1 such that

QTn,1 · · ·QT

3,1QT2,1

a1,1a2,1

...an,1

=

#0...0

.We may similarly transform the second column of A so that

QTn,2 · · ·QT

4,2QT3,2

# a1,20 a2,20 a3,2...

...0 an,2

=

# a1,20 #0 0...

...0 0

.

Note: since the rotators Q3,2, . . . , Qn,2 only have c and s values in rows 2 through n, noneaffect row 1 (and hence cannot create additional nonzeros in column 1).

Proceeding in this same manner, we may eventually obtain

R = QTn,n−1Q

Tn,n−2 · · ·QT

3,1QT2,1A,

where R is upper-triangular. Defining

QT = QTn,n−1Q

Tn,n−2 · · ·QT

3,1QT2,1

⇔Q = Q2,1Q3,1 · · ·Qn,n−2Qn,n−1,

then from our Lemmas 3.2.5 and 3.2.6, we see that Q is orthogonal. Hence,

R = QTA ⇔ A = QR.



3.2.2 Reflections

While rotation matrices are excellent, it turns out that reflection matrices are more fre-quently used in constructing the QR decomposition of a matrix.

Let’s again consider the simple case of x ∈ �2 to begin, and assume there is a line L ⊂ �2

that intersects the origin. We wish to build an orthogonal matrix Q ∈ �2×2 that willreflect any vector x ∈ �2 across the line L.

v

L

Qx=−au+bv

u

x=au+bv

Geometry : let v ∈ �2 lie along L, and letu ∈ �2 be a unit vector orthogonal to L.Then we may write x = αu + βv, and thereflection Qx = −αu + βv, since the com-ponent along L is retained, but the compo-nent perpendicular to L is reversed.

Hence, based on the geometry we have

Q(αu+ βv) = −αu+ βv,

but algebraically, we haveQ(αu+ βv) = αQu+ βQv.

Hence for both of these equations to hold simultaneously, we need that Q ∈ �2×2 satisfythe properties

Qu = −u and Qv = v. (4)

So how can we build Q? For starters, let’s consider the [projection] matrix P = uuT .Since u is a unit vector, then

Pu = (uuT )u = u(uTu) = u‖u‖22 = u,

andPv = (uuT )v = u(uTv) = u(0) = 0,

We’re not quite there at meeting the properties (4), but we’re close. For Pu we need−u instead of u, and for Pv we need v instead of 0. To that end, note that Iu = u andIv = v, so the choice

Q = I − 2P = I − 2uuT (5)

will work:

Qu =(I − 2uuT

)u = u− 2u(uTu) = u− 2u = −u,

Qv =(I − 2uuT

)v = v − 2u(uTv) = v − 0 = v.



3.2.2.1 Extending to larger matrices

Theorem 3.2.8. Let u ∈ �n with ‖u‖2 = 1, and define P = uuT . Then

(a) Pu = u,

(b) Pv = 0 if 〈u, v〉 = 0,

(c) P T = P , and

(d) P 2 = P .

(proof left for homework/exams)

Notes:

• Any matrix P ∈ �n×n that satisfies P 2 = P is called a projector, or projection matrix.

• A symmetric projector is also called an orthoprojector.

• Since here P = uuT has rank 1, then P is called a rank-1 orthoprojector.

Theorem 3.2.9. Let u ∈ �n with ‖u‖2 = 1, and define Q = I − 2uuT . Then

(a) Qu = −u,

(b) Qv = v if 〈u, v〉 = 0,

(c) QT = Q,

(d) QT = Q−1, and

(d) Q−1 = Q (i.e., Q is called an involution).

(proof left for homework/exams)

Notes:

• A matrix of the formQ = I−2uuT where ‖u‖2 = 1 is called a Householder transformationor Householder reflector.

• For any u ∈ �n, u 6= 0, H = {v ∈ �n : 〈u, v〉 = 0} is an (n − 1)-dimensionalsubspace of �n, called a hyperplane (recall in Calculus 3 – a plane that intersectsthe origin may be uniquely defined by it’s normal vector).

• Q maps each vector x ∈ �n to its reflection across this hyperplane H.

Lemma 3.2.10. Let u ∈ �n, u 6= 0, and let γ = 2‖u‖22

, and define the matrix Q = I−γuuT .

Then Q is a Householder transformation that satisfies Qu = −u and Qv = v if 〈u, v〉 = 0.



Proof. Let u = u‖u‖2 be the unit vector in the same direction as u. Then

Q = I − γuuT = I − 2

‖u‖22uuT = I − 2uuT ,

so we may use the previous Theorem 3.2.9 to conclude that Q is a Householder transfor-mation, and that Qv = v for any vector v ∈ �n such that 〈u, v〉 = 〈u, v〉 = 0, since uand u are parallel.

All that remains is to show that Qu = −u (since the previous theorem only shows thiswhen u is a unit vector).

Qu =(I − γuuT

)u = u− γu(uTu) = u− γu‖u‖22 = u− 2‖u‖22

‖u‖22u = u− 2u = −u.

Theorem 3.2.11. Let x, y ∈ �n with x 6= y, and let ‖x‖2 = ‖y‖2. Then ∃! Householdertransformation Q such that Qx = y.

Proof. (existence here; uniqueness is left for homework/exam)Let u = x−y, and note that u 6= 0 since x 6= y. Then defining γ = 2

‖u‖22and Q = I−γuuT ,

from Lemma 3.2.10 we know that Q is a Householder transformation that satisfies Qu =−u and Qv = v for all v ∈ �n such that 〈u, v〉 = 0.

Note that we may equivalently write x as

x =1

2(x+ x) +

1

2(y − y) =

1

2(x− y) +

1

2(x+ y).

Recalling that u = x− y, and defining v = x+ y, we may equivalently write

x =1

2u+

1

2v. (6)

Moreover, since ‖x‖2 = ‖y‖2 we have that

〈u, v〉 = 〈x− y, x+ y〉 = 〈x, x〉+ 〈x, y〉 − 〈y, x〉 − 〈y, y〉 = 0,

and so this vector v satisfies the desired property above. Therefore,

Qx =1

2Qu+

1

2Qv [from (6)]

= −1

2u+

1

2v [since Qu = −u and Qv = v]

= −1

2(x− y) +

1

2(x+ y) [def. u and v]

=1

2(x− x) +

1

2(y + y) = y [algebra]



Note: while the above theorem used u = x − y to construct the matrix Q = I − γuuT ,since γ = 2

‖u‖22includes the magnitude of u, we may in fact use any nonzero multiple of u

and obtain the same matrix Q.

Corollary 3.2.12. Let x ∈ �n with x 6= 0. Then ∃! reflector Q such that Qx =

#0...0

.

Proof. Let y =

−τ0...0

with τ = ±‖x‖2, and we may choose the +/− to guarantee that

x 6= y. Then ‖x‖2 = ‖y‖2, and Theorem 3.2.11 guarantees ∃!Q such that Qx = y.

As before with rotators, once we know how to construct an orthogonal matrix that elimi-nates entries below the leading nonzero in a column, we may perform this process repeat-edly to construct A = QR.

Putting this into practice:

• Given a nonzero vector x ∈ �m (e.g., the first column of A ∈ �m×n), computeτ = ‖x‖2 sign(x1) – note that this guarantees that x1 + τ 6= 0. While we “imagine”

the vector y =

−τ0...0

, we do not explicitly construct it.

• Compute u = x−yx1+τ

=

1

x2/(x1 + τ)...

xm/(x1 + τ)

. Note that by construction

‖u‖22 =(x1 + τ)2 + x22 + · · ·+ x2m

(x1 + τ)2=τ 2 + 2τx1 + ‖x‖22

(x1 + τ)2,

so since τ 2 = ‖x‖22, this simplifies to

‖u‖22 =2τ(τ + x1)

(x1 + τ)2=

2τ

x1 + τ.

• Compute

γ =2

‖u‖22=τ + x1τ

.



While we again “imagine” the matrix Q = I − γuuT , its low rank structure allowsus to store the relevant components using only (m+ 1) numbers (u and γ) insteadof actually storing Q (m2 numbers).

• When performing the matrix-matrix product B = QA, it is more efficient to directlyuse the form

(I − γuuT

)A:

B = QA =(I − γuuT

)A = A− γuuTA

with the steps:

(a) compute vT = uTA

(b) compute wT = γvT

(c) compute B = A− uwT ,

requiring a total of ≈ 4mn flops (instead of O(m2n) for construction of Q and densematrix-matrix multiplication QA).

With this approach (including relevant implementation details) in place, we may nowconsider a proof of Theorem 3.2.7 that shows how to construct A = QR using reflectors.Moreover, this ‘proof’ is really a second algorithm for computing the QR decompositionof a matrix!

Proof of Theorem 3.2.7, using induction on the matrix size n:

First, we recall the statement of the Theorem: Let A ∈ �n×n. Then ∃ an orthogonalmatrix Q ∈ �n×n and an upper-triangular matrix R ∈ �n×n such that A = QR.

Base case (n = 1): Here A =[a1,1], so we may define Q =

[1]

and R = A. SinceQTQ = QQT = 1 and R is upper triangular, we are finished.

Inductive step: assume that for all matrices A ∈ �(n−1)×(n−1), ∃ an orthogonal matrixQ ∈ �(n−1)×(n−1) and an upper-triangular matrix R ∈ �(n−1)×(n−1) such that A = QR.If the first column of A is all zeros then let Q1 = In; otherwise let Q1 ∈ �n×n be areflector that zeros out the first column of A below the diagonal (guaranteed to exist fromCorollary 3.2.12). In either case, we have

Q1

a1,1a2,1

...an,1

=

−τ0...0

where τ = ±‖a1‖2. Then since Q1 = QT

1 , we have

QT1A =

−τ a1,2 · · · a1,n0... A0



By our inductive hypothesis, A = QR with Q orthogonal and R upper triangular.Let

Q2 =

1 0 · · · 00... Q0

⇒ QT2Q2 =

1 0 · · · 00... I0

,and so Q2 is also orthogonal. Moreover,

QT2Q

T1A =

1 0 · · · 00... Q0

−τ a1,2 · · · a1,n0... A0

=

−τ a1,2 · · · a1,n0... R0

= R.

Finally, let QT = QT2Q

T1 ⇔ Q = Q1Q2; since Q is a product of orthogonal matrices then

it is also orthogonal, so QTA = A ⇔ A = QR.

The corresponding algorithm for computing the QR decomposition of A ∈ �n×n re-quires

• 4n3

3operations to construct the unassembled version (where we store only the parts

to build Q), or

• 8n3

3operations to construct the assembled version.

Suppose that you have computed A = QR with Q unassembled, i.e., Q = Q1Q2 · · ·Qn−1.Then to solve Ax = b ⇔ QRx = b ⇔ Rx = QT b, you need to take the steps:

(1) Compute c = QT b = QTn−1Q

Tn−2 · · ·QT

2QT1 b = Qn−1Qn−2 · · ·Q2Q1b (since reflectors

are symmetric):

(1a) c = Q1b

(1b) c = Q2c

...

(2) Solve Rx = c using backwards substitution.

The overall cost of step 1 is ≈ 2n2, while the overall cost of step 2 is ≈ n2.

3.2.3 Uniqueness of the QR decomposition

Notice that Theorem 3.2.7 stated nothing about the QR decomposition being unique.This is because in that theorem we considered general A ∈ �n×n (i.e., A could possiblybe singular, and hence the proof step where we allowed the possibility of a zero column inA). If we assume a bit more about A, then the QR decomposition is in fact unique.



Theorem 3.2.13. Let A ∈ �n×n be nonsingular. Then ∃!Q,R ∈ �n×n where Q isorthogonal, R is upper-triangular, ri,i > 0, and A = QR.

Proof. We are guaranteed the existence of an orthogonal matrix orthogonal Q ∈ �n×nand an upper triangular matrix R ∈ �n×n such that A = QR, due to our earlier Theorem3.2.7. Moreover, since A is nonsingular then R must also be nonsingular, so ri,i 6= 0.

Define the diagonal matrix D ∈ �n×n such that di,i = sign(ri,i). Note that since D =DT = D−1, then D is orthogonal.

We then define the matricesQ = QD−1 and R = DR. SinceQ is the product of orthogonalmatrices then it is orthogonal. Since R is the product of upper-triangular matrices thenit is upper-triangular. Moreover, since left-multiplication by D scales the rows of R tocreate R, then ri,i = sign(ri,i)ri,i = |ri,i| > 0 (since ri,i 6= 0). Finally,

QR = QD−1DR = QR = A,

so we have proven existence of the QR factorization with positive diagonal in R.

For uniqueness, suppose that A = Q1R1 = Q2R2, where both Q1 and Q2 are orthogonal,and both R1 and R2 are upper triangular with strictly positive entries on their diagonals.

Since A is nonsingular, then ATA is SPD. So from Theorem 1.4.7, ∃! upper-triangularmatrix R with strictly positive diagonal entries, such that ATA = RTR.

However, inserting our separate QR factorizations in for A in this formula, we have

ATA = (Q1R1)T (Q1R1) = RT

1QT1Q1R1 = RT

1R1

and

ATA = (Q2R2)T (Q2R2) = RT

2QT2Q2R2 = RT

2R2.

Since both R1 and R2 are upper triangular with strictly positive diagonal entries, thenthey must both equal the unique Cholesky factor for ATA, and hence R1 = R2 = R.

Finally, since now A = Q1R = Q2R, and R is nonsingular, then

Q1 = AR−1 = Q2.

3.2.4 Stability of the QR decomposition

Suppose that Q ∈ �n×n is either a rotator or reflector, and we compute the product QA.Wilkinson showed that

fl(QA) = Q(A+ E) with‖E‖2‖A‖2

� 1,



so QA is normwise backwards stable. Now consider Q2Q1A for two rotators/reflectors Q1

and Q2:

fl(Q2Q1A) = fl(Q2 fl(Q1A)) = fl(Q2(Q1(A+ E1))) = Q2(Q1(A+ E1) + E2)

= Q2(Q1(A+ E1) +Q1QT1E2) = Q2Q1(A+ E1 +QT

1E2)

= Q2Q1(A+ E),

with E = E1 +QT1E2. Since

‖E‖2 = ‖E1 +QT1E2‖ ≤ ‖E1‖2 + ‖QT

1 ‖2‖E2‖2 = ‖E1‖2 + ‖E2‖2,

then we again have,‖E‖2‖A‖2

� 1,

and Q2Q1A is normwise backwards stable. Continuing in this manner, we see thatR = QA = QpQp−1 · · ·Q2Q1A may be computed with relative error � 1, resulting ina normwise backwards stable algorithm.

3.2.5 The QR decomposition in complex arithmetic

We must first define a few items, before stating the main theorems for A ∈ �n×n.

Definition 3.2.14 (Complex inner product). For vectors x, y, z ∈ �n, a complex inner product〈·, ·〉 : �n × �n → � is any function that satisfies the properties:

(i) 〈x, x〉 ∈ �, 〈x, x〉 ≥ 0, and 〈x, x〉 = 0 iff x = 0,

(ii) 〈x, y〉 = 〈y, x〉,

(iii) 〈αx, y〉 = α 〈x, y〉,

(iv) 〈x, αy〉 = α 〈x, y〉, and

(v) 〈x+ y, z〉 = 〈x, z〉+ 〈y, z〉.

Note that the standard Euclidean complex inner product for vectors x, y ∈ �n is definedas

〈x, y〉 = y∗x = x∗y =n∑i=1

xiyi.

Definition 3.2.15 (Unitary). A matrix U ∈ �n×n is called unitary if U∗U = UU∗ = I(i.e., the complex-valued equivalent of “orthogonal”).

Theorem 3.2.16. Let A ∈ �n×n. Then there exist matrices Q,R ∈ �n×n, Q unitary, Rupper triangular with ri,i ∈ �, such that A = QR.

(the proof follows similar steps as our result in �n)



Theorem 3.2.17. Let A ∈ �n×n be nonsingular. Then there exist unique Q,R ∈ �n×n,Q unitary, R upper triangular with ri,i ∈ � and ri,i > 0, such that A = QR.

(the proof follows similar steps as our result in �n)

3.3 Solving the Least Squares Problem

We now return to our main goal of this chapter.

Consider an overdetermined linear system Ax = b, with A ∈ �m×n, b ∈ �m, and m > n.We have already shown in equations (2) and (3) that in real arithmetic,

x = argminx∈�n

1


1

2‖QT r‖22,

for any orthogonal matrix Q; the equivalent result in complex arithmetic is

x = argminx∈�n1


1

2‖Q∗r‖22 (7)

for any unitary matrix Q. Let’s name the vector Q∗r as s. Recalling that we defined thelinear residual as r = b− Ax, we have

s = Q∗r = Q∗b−Q∗Ax.

So to solve the problem (7), we must find an x ∈ �n such that ‖s‖22 is minimized.

Our primary tool in solving this problem will be a QR-like factorization for tall/skinnyrectangular matrices.

Theorem 3.3.1. Let A ∈ �m×n with m > n. Then ∃ a unitary matrix Q ∈ �m×m and a

matrix R =

[R0

]∈ �m×n, with R ∈ �n×n upper triangular, such that A = QR.

Proof. Let A =[A B

]∈ �m×m, with B ∈ �m×(m−n) arbitrary. Then from Theorem

3.2.16, there exist matrices Q and R in �m×m such that Q is unitary, R is upper-triangularwith ri,i ∈ �, and A = QR.

Partitioning R =[R T

]with R ∈ �m×n, then

A = QR ⇔[A B

]= Q

[R T

]⇒ A = QR.

Finally, since R is upper-triangular, and R contains only the first n columns of R, it must

have the structure R =

[R0

], with R ∈ �n×n upper triangular.



Note: we can compute this “tall-skinny QR” (TSQR) decomposition by just performingthe first n steps of the m× n QR decomposition on A. Hence B need never be defined,stored, or used, since the first n steps only touch the first n columns of A (i.e., justA).

Since m > n there are two possibilities for this TSQR decomposition: (a) rank(A) = n(i.e., A has full column rank), or (b) rank(A) < n (the “rank deficient” case).

3.3.1 Full-rank case

Assume that rank(A) = n. Let A = QR from Theorem 3.3.1.

Recalling that we want to solve

Ax = b ⇔ QRx = b ⇔ Rx = Q∗b,

we first compute the vector c = Q∗b. Although we would like to solve Rx = c, sinceR ∈ �m×n with m > n (i.e., it is not square), then the system Rx = c is likely to beinconsistent.

More specifically, consider the partition c =

[cd

]with c ∈ �n. Then our modified least-

squares residual can be written

s = Q∗r = Q∗(b− Ax) = Q∗b−Q∗QRx = c−Rx =

[cd

]−[R0

]x =

[c− Rxd

].

Therefore‖s‖22 = ‖c− Rx‖22 + ‖d‖22.

Since ‖d‖22 is independent of x, then the minimum value of ‖s‖22 may be obtained exactly

when ‖c− Rx‖22 is minimized.

Since rank(A) = n and A = QR with Q unitary, then rank(R) = rank(R) = n, and thusR is nonsingular. In other words, the linear system

Rx = c

is always consistent, and we can compute the [unique] solution to the discrete least-squaresproblem.

In summary, this requires 3 steps:

(1) Compute the TSQR decomposition of A.

(2) Multiply Q∗b to generate the vector c

(3) Extract the first n rows from both c and R, c and R respectively, and solve Rx = cfor x using backwards substitution.

Note: from the above derivation, it is clear that ‖d‖2 is a measure of how well the modelfits the data – if ‖d‖2 = 0 then the model provides a perfect fit.



3.3.2 Rank-deficient case

Note: most discrete least-squares problems have full rank, or if they do not yet havefull rank, this may be rectified by obtaining more data points {(ti, yi)}mi=1 (i.e., increasem).

The optimal solution to the rank-deficient case will come in chapter 4. Our book discussesan alternate algorithm here, the QR decomposition with column pivoting. While thisapproach can indeed work (and should take less computational effort than what we’lldiscuss in chapter 4), it requires knowledge of rank(A). Unfortunately, in floating-pointarithmetic, the actual rank of a matrix can be difficult/impossible to discern. As such, Ido not plan to discuss this portion of chapter 3, and would prefer to dedicate our timetogether to focus on more ‘useful’ algorithms.

3.4 Gram-Schmidt

While it would seem that we’ve already “solved” the least squares problem (or at least,when A ∈ �m×n has full column rank), in fact both of the preceding approaches forcomputing A = QR are only relevant when performing computations in serial (i.e., notin parallel). We therefore spend the rest of this chapter on an alternate algorithm forcomputing the QR decomposition. While the resulting approach is not as optimal forserial computations, it may easily be ported to very large scale parallel computations,and is therefore widely used in modern scientific computing.

Definition 3.4.1 (Orthonormal). A set of vectors {qi}ki=1 ⊂ �n is called orthonormal inan inner product 〈·, ·〉 iff

〈qi, qj〉 = δi,j =

{0, i 6= j

1, i = j

Theorem 3.4.2. A matrix Q ∈ �n×n (or in �n×n) is orthogonal (or unitary) iff itscolumns form an orthonormal set.

Similarly, a matrix Q ∈ �n×n (or in �n×n) is unitary (or orthogonal) iff its rows form anorthonormal set.

Proof. We will prove this for real arithmetic; the complex-valued case is essentially iden-tical.

Let Q =[q1 q2 · · · qn

]. Then

QTQ =

qT1...qTn

[q1 q2 · · · qn]

⇔ (QTQ)i,j = qTi qj.



First, assume that Q is orthogonal. Then QTQ = I, and so qTi qj = δi,j.

Second, assume that qTi qj = δi,j. Then (QTQ)i,j = δi,j and QTQ = I.

Finally, if Q is orthogonal, then so is QT (since QTQ = QQT = I), so we may repeat theabove arguments for QT instead of Q to show the result for Q’s rows.

Definition 3.4.3 (Isometric). A matrix Q ∈ �m×n (m ≥ n) is called isometric if itscolumns are orthonormal.

Hence QTQ = In, but if m > n then QQT 6= Im.

Theorem 3.4.4. Let A ∈ �m×n with m ≥ n. Then there exist matrices Q ∈ �m×n andR ∈ �n×n such that Q is isometric, R is upper triangular, and A = QR.

Proof. If m = n then this result was already proven in Theorem 3.2.7.

Assume that m > n. Then from Theorem 3.3.1 we know that there exist matrices Q ∈

�m×m and R ∈ �m×n such that Q is orthogonal and R =

[R0

]with R ∈ �n×n upper

triangular.

Choose Q as the first n columns of Q, so that Q =[Q Q

]. Then

A = QR =[Q Q

] [R0

]= QR.

As before, we can prove a stronger result if we know that A has full column rank.

Theorem 3.4.5. Let A ∈ �m×n with m ≥ n and rank(A) = n. Then there exist uniquematrices Q ∈ �m×n and R ∈ �n×n such that Q is isometric, R is upper triangular withri,i > 0, and A = QR.

(proof is left for homework/exam)

In this case of the “tall-skinny QR,” we can again use the decomposition to solve theleast-squares problem Ax = b:

Ax = b ⇔ QRx = b

⇒ QT QRx = QT b

⇔ Rx = QT b,

so we have the steps

(a) Compute c = QT b.

(b) Solve Rx = c for x (backwards substitution).



While this approach indeed solves the least-squares problem, it does not give us a freeresidual norm, ‖d‖2. This is a benefit and a detractor, since it eliminates additionalcomputation (e.g., QT b), but it does not provide residual information.

3.4.1 Brief review

Before deriving the main result of this section, let’s review a few topics from introductorylinear algebra:

Definition 3.4.6 (Subspace). S is a subspace of a vector space V iff

(i) αv ∈ S for all scalars α and all vectors v ∈ S.

(ii) (u+ v) ∈ S for all u, v ∈ S.

Note: these imply the existence of the zero vector in S.

Definition 3.4.7 (Linear combination). A linear combination of the vectors {vk}nk=1 is aformula of the form

α1v1 + . . .+ αnvn

for arbitrary scalars {αk}nk=1.

Note: if {vk}nk=1 are in a subspace S, then

α1v1 + . . .+ αnvn ∈ S.

Definition 3.4.8 (Span). The span of a set of vectors {vk}nk=1 is the set of all possiblelinear combinations of {vk}nk=1.

Notes:

• This is denoted span{v1, . . . , vn}

• If {vk}nk=1 are in a subspace S, then span{v1, . . . , vn} ⊆ S.

Definition 3.4.9 (Linear independence). A set of vectors {vk}nk=1 is called linearly independentiff

α1v1 + . . .+ αnvn = 0

can only be achieved if α1 = · · · = αn = 0.

Lemma 3.4.10. If a set of nonzero vectors {vk}nk=1 are orthogonal, then they are linearlyindependent.

Proof. Assume that {vk}nk=1 are orthogonal, and that

α1v1 + . . .+ αnvn = 0.



Then taking the inner product of both sides with vj, we have

〈α1v1 + . . .+ αnvn, vj〉 = 〈0, vj〉⇔

α1 〈v1, vj〉+ . . .+ αn 〈vn, vj〉 = 0.

Since orthogonality implies that each inner product is zero except 〈vj, vj〉 6= 0 (recallthat vj 6= 0), we may divide to see that αj = 0. However, since j was arbitrary, we havethat α1 = · · · = αn = 0.

Definition 3.4.11 (Spanning set). If span{v1, . . . , vn} = S then we call {vk}nk=1 aspanning set for S, or more briefly, that {vk}nk=1 spans S.

Definition 3.4.12 (Basis). A set of vectors {vk}nk=1 is a basis for a subspace S iff

(i) the set {vk}nk=1 is linearly independent, and

(ii) the set {vk}nk=1 spans S.

Definition 3.4.13 (Dimension). The dimension of a subspace S is the number of vectorsin every basis for S.

Definition 3.4.14 (Orthogonal complement). Let S be a subspace of a vector space V .Then the orthogonal complement of S in V , denoted S⊥, is defined as

S⊥ ={v ∈ V

∣∣ 〈v, w〉 = 0 ∀w ∈ S}.

Notes:

• S⊥ is always non-empty, since 0 ∈ S⊥.

• S⊥ is itself a subspace of V , and(S⊥)⊥

= S.

Theorem 3.4.15. Let S be a subspace of a vector space V . Then for every y ∈ V , thereexist unique vectors s ∈ S and p ∈ S⊥ such that y = s+ p.

Definition 3.4.16. Given subspaces S1 and S2 of a vector space V , we define the sum

S1 + S2 ={s1 + s2

∣∣ s1 ∈ S1 ∧ s2 ∈ S2

}.

Our Theorem 3.4.15 thus states that if S is a subspace of a vector space V , then V =S + S⊥. Moreover, the components s ∈ S and p ∈ S⊥ of y ∈ V are called theorthogonal projections of y into S and S⊥.

Definition 3.4.17 (Direct sum). The sum U = S1 +S2 is called a direct sum if for everyu ∈ U , the decomposition u = s1 + s2 where s1 ∈ S1 and s2 ∈ S2 is unique. This isdenoted with the special symbol

U = S1 ⊕ S2.



Hence if S is a subspace of V , then we can always write V = S ⊕ S⊥.

3.4.2 Gram-Schmidt Process

With these definitions fresh in our minds, we consider the following problem:

Given a set of basis vectors {vk}nk=1 for a subspace S, can we find an orthonormal set ofvectors {qk}nk=1 such that for each 1 ≤ m ≤ n,

span{v1, . . . , vm} = span{q1, . . . , qm}? (8)

The basic algorithm is covered in most introductory linear algebra classes, but it rarely“sticks” the first time.

Let’s proceed by first satisfying (8) for m = 1, then m = 2, and so on. When m = 1 wewant a unit vector q1 such that

span{v1} = span{q1} ⇔ v1 = αq1,

so the only two choices are

q1 = ± v1‖v1‖

Notes:

• We typically use the 2-norm here.

• All else equal, we typically use the positive of the two choices.

Hence, we choose

q1 =v1‖v1‖2

We continue with m = 2, and now seek q2 such that span{v1, v2} = span{q1, q2}, and〈q1, q2〉 = 0. To this end, we decompose v2 into two components, v2 = u + z, whereu ∈ span{q1} and z ∈ span{q1}⊥. Here, only the z component contains useful information,since u can be built using q1. In equations, these requirements become

u = αq1, 〈z, q1〉 = 0.

Since z = v2 − u = v2 − αq1, we have

0 = 〈v2 − αq1, q1〉 = 〈v2, q1〉 − α 〈q1, q1〉 = 〈v2, q1〉 − α,

since q1 is a unit vector. Hence, α = 〈v2, q1〉 and so we first compute

z = v2 − 〈v2, q1〉 q1,

and then normalize the result,

q2 =z

‖z‖2.



Note: 〈v2, q1〉 q1 is the projection of v2 onto the orthonormal basis span{q1}. More gen-

erally, 〈x, y〉〈y, y〉y is the projection of x onto a nonzero vector y.

To see the full pattern, we examine m = 3 before jumping to the general case. We now seekq3 such that span{v1, v2, v3} = span{q1, q2, q3}, and 〈q1, q3〉 = 〈q2, q3〉 = 0. Followingthe same idea, we decompose v3 = u + z where u ∈ span{q1, q2} and z ∈ span{q1, q2}⊥,or equivalently,

u = αq1 + βq2, 〈z, q1〉 = 〈z, q2〉 = 0.

Writing z = v3−αq1−βq2 and inserting into the two orthogonality relations, we have

0 = 〈v3 − αq1 − βq2, q1〉 = 〈v3, q1〉 − α 〈q1, q1〉 − β 〈q2, q1〉 = 〈v3, q1〉 − α,0 = 〈v3 − αq1 − βq2, q2〉 = 〈v3, q2〉 − α 〈q1, q2〉 − β 〈q2, q2〉 = 〈v3, q2〉 − β,

since {q1, q2} are orthonormal. Or equivalently,

α = 〈v3, q1〉 and β = 〈v3, q2〉 ,

so we computez = v3 − 〈v3, q1〉 q1 − 〈v3, q2〉 q2,

and then normalize,

q3 =z

‖z‖2.

By this point, the pattern is clear: at step m of the process, we compute

z = vm −m−1∑j=1

〈vm, qj〉 qj, (9a)

and the normalizeqm =

z

‖z‖2. (9b)

The equations (9a)-(9b) is called the Classical Gram-Schmidt Process.

Note: if at any step we obtain z = 0 then that vm is linear dependent on the previous{vk}m−1k=1 .

Moreover, if we instead denote the scalars in this process as rj,k, where at each step k wedo the steps:

rj,k = 〈vk, qj〉 , j = 1, . . . , k − 1

zk = vk −k−1∑j=1

rj,kqj

rk,k = ‖zk‖2qk =

zkrk,k

,



then we can rewrite these equations as

zk = rk,kqk

vk = zk +k−1∑j=1

rj,kqj =k∑j=1

rj,kqj.

Examining this last equation for increasing values of k, we have

v1 = q1r1,1

v2 = q1r1,2 + q2r2,2...

vn = q1r1,n + q2r2,n + . . .+ qnrn,n,

or equivalently,

[v1 v2 · · · vn

]=[q1 q2 · · · qn

]r1,1 r1,2 · · · r1,n

r2,2 · · · r2,n. . .

rn,n

,or A = QR where A =

[v1 v2 · · · vn

]. Here, Q has orthonormal columns (i.e., Q is

isometric), R is upper-triangular, and rk,k > 0 when {vk}nk=1 are linearly independent.Hence these matrices Q and R are unique from Theorem 3.2.13.

We therefore have three completely different methods for computing the QR decomposi-tion: rotators, reflectors, and Gram-Schmidt.

Of these, rotators and reflectors are well-conditioned due to their reliance on orthogo-nal transformations, but Gram-Schmidt can be sensitive to ill-conditioning in A. Morespecifically, if {vk}nk=1 are nearly linearly dependent, then we’ll get a vector zk at somepoint that is nearly zero (i.e., it is susceptible to catastrophic cancellation), so rk,k � 1and qk = zk

rk,kwill increase roundoff errors.

On the other hand, due to their use of specific matrix entries, both rotators and reflectorsare only suitable to serial programs, where the entire matrix A may be stored in memoryon a single computer. On the other hand, since Gram-Schmidt never actually investigatesmatrix entries, but instead relies purely on inner-products and vector arithmetic, it isideally suited to parallel computing.

As a result, there are a variety of “fixes” for Gram-Schmidt:

(1) Reorthogonalization: after computing zk, perform the orthogonalization again to en-sure that all remaining components not orthogonal to {q1, . . . , qk−1} are subtractedoff before normalizing.



(2) Modified Gram-Schmidt: instead of computing the projection coefficients all at once,compute each one after modifying the result, i.e., instead of first computing

k−1∑j=1

〈vk, qj〉 qj

and then subtracting,

z = vk −

(k−1∑j=1

〈vk, qj〉 qj

),

this performs these subtractions in a loop:

for j = 1, . . . , k − 1 dovk ← vk − 〈vk, qj〉 qj

end for

Since vk is updated at each iteration, the following iterations use different vk vectorsin their inner products.

3.5 Geometric Approach to Least-Squares

We end this chapter with an investigation of the geometry underlying the least-squaresproblem, which will be necessary in subsequent chapters.

Definition 3.5.1. Given A ∈ �m×n, we have the “four fundamental subspaces”:

C(A) ={b ∈ �m

∣∣ b = Ax for some x ∈ �n}

→ the column space of A

N(A) ={x ∈ �n

∣∣ Ax = 0}

→ the null space of A

C(AT ) ={b ∈ �n

∣∣ b = ATx for some x ∈ �m}

→ the row space of A

N(AT ) ={x ∈ �m

∣∣ ATx = 0}

→ the left null space of A

Notes:

• N(A) ⊆ �n, C(A) ⊆ �m, C(AT ) ⊆ �n, N(AT ) ⊆ �m.

• rank(A) = dim (C(A)).

Lemma 3.5.2. Let A ∈ �m×n. Then for all x ∈ �n and all y ∈ �m, 〈Ax, y〉 = 〈x, A∗y〉,where 〈·, ·〉 is the standard Euclidean inner product.

Proof. We start with the right side of the equality, and work our way backwards usingonly the basic properties of matrix arithmetic:

〈x, A∗y〉 = (A∗y)∗x = y∗ (A∗)∗ x = y∗(Ax) = 〈Ax, y〉 .



Theorem 3.5.3 (Fundamental Theorem of Linear Algebra).

(C(A))⊥ = N(A∗) and (N(A))⊥ = C(A∗).

Proof. We begin with the first of these, and note that to prove set equality we must showboth (C(A))⊥ ⊆ N(A∗) and N(A∗) ⊆ (C(A))⊥.

Let y ∈ (C(A))⊥. Then by definition of C(A) and the orthogonal complement, and byusing Lemma 3.5.2,

0 = 〈Ax, y〉 = 〈x, A∗y〉

for all x ∈ �n. Choosing x = A∗y, we then have

0 = 〈A∗y, A∗y〉 ⇔ A∗y = 0 ⇔ y ∈ N(A∗).

Now let y ∈ N(A∗), so by definition A∗y = 0. Hence, given an arbitrary x ∈ �n (andtherefore an arbitrary (Ax) ∈ C(A)), we have

〈Ax, y〉 = 〈x, A∗y〉 = 〈x, 0〉 = 0,

so y ∈ (C(A))⊥.

The other half of the theorem follows immediately by naming B = A∗ and applying thealready-proven portion to B and B∗.

We may therefore sum up the entire Fundamental Theorem of Linear Algebra in twolines:

�m = C(A)⊕N(A∗) and �

n = C(A∗)⊕N(A).

Moreover, replacing � with � and A∗ with AT in all of the above results gives the equiv-alent result in real arithmetic:

�m = C(A)⊕N(AT ) and �

n = C(AT )⊕N(A).

We now use the above results to return to the discrete least squares problem.

Theorem 3.5.4. Let S be a subspace of �n and let b ∈ �n. Then ∃!y ∈ S such that

‖b− y‖2 = mins∈S‖b− s‖2,

where y is the unique element in S such that b−y ∈ S⊥, i.e., y is the orthogonal projectionof b into S.

Proof. From the Theorem 3.4.15, there exists a unique decomposition b = y + z, wherey ∈ S and z ∈ S⊥, and where y is the orthogonal projection of b into S. Hence b − y =z ∈ S⊥.



Let s ∈ S be arbitrary. Then (y − s) ∈ S, and hence 〈b− y, y − s〉 = 0. Therefore,

‖b− s‖22 = ‖(b− y) + (y − s)‖22 [add zero]

= ‖b− y‖22 + ‖y − s‖22 [Pythagorean theorem].

Since ‖b − y‖22 is indepenent of s, and ‖y − s‖22 ≥ 0, then this distance is minimizedprecisely when y = s. Hence

‖b− y‖22 = mins∈S‖b− s‖22,

so taking the square root of both sides we arrive at our desired result.

So how can we use this for the discrete least-squares problem?

In that case, S = C(A), so

mins∈C(A)

‖b− s‖2 = minw∈�n

‖b− Aw‖2,

and thus we may directly apply this theorem to solve our problem!

We therefore seek the unique y ∈ C(A) such that

b− y ∈ (C(A))⊥ = N(AT )

⇔AT (b− y) = 0.

Moreover, since y ∈ C(A), then we are guaranteed the existence of some x ∈ �n such thatAx = y. Inserting this into our equation above, we have

AT (b− Ax) = 0 ⇔ ATAx = AT b, (10)

which are the so-called normal equations that you learned to use in previous courses!Finally, if N(A) = {0}, then since y is unique, we’ll also have that x is unique.

Note: In practice, you should not solve (10) directly, since κ(ATA) ∼ (κ(A))2 [when κ(A)is defined], so roundoff errors grow significantly when A is ill-conditioned. Instead, it ismuch more stable to use the QR factorization.


math5316 lecture notes chapter 3 { least squares 3.1 the

Documents