orthogonality inner or dot product in r : u v = u · v = u v1 + ···u
TRANSCRIPT
![Page 1: Orthogonality Inner or dot product in R : u v = u · v = u v1 + ···u](https://reader034.vdocuments.net/reader034/viewer/2022042507/588ef4711a28abd04e8b49f2/html5/thumbnails/1.jpg)
Math 52 0 - Linear algebra, Spring Semester 2012-2013Dan Abramovich
Orthogonality
Inner or dot product in Rn:
uTv = u · v = u1v1 + · · ·unvn ←examples
Properties:
• u · v = v · u• (u + v) ·w = u ·w + v ·w• (cv) ·w = c(v ·w)• u · u ≥ 0,
and u · u > 0 unless u = 0.
1
![Page 2: Orthogonality Inner or dot product in R : u v = u · v = u v1 + ···u](https://reader034.vdocuments.net/reader034/viewer/2022042507/588ef4711a28abd04e8b49f2/html5/thumbnails/2.jpg)
2
Norm = length:
‖v‖ =√v · v
u is a unit vector if ‖u‖ = 1.
If v 6= 0 then
u =1
‖v‖v
is a unit vector positively-proportionalto v.
examples→
![Page 3: Orthogonality Inner or dot product in R : u v = u · v = u v1 + ···u](https://reader034.vdocuments.net/reader034/viewer/2022042507/588ef4711a28abd04e8b49f2/html5/thumbnails/3.jpg)
3
We argue that u ⊥ v if and only if‖u− v‖ = ‖u− (−v)‖. ←perpendicular or or-
thogonal
This means
(u−v) · (u−v) = (u+v) · (u+v)
Expand and get
‖u‖2 + ‖v‖2 − 2u · v= ‖u‖2 + ‖v‖2 + 2u · v
So
u ⊥ v⇐⇒ u · v = 0
![Page 4: Orthogonality Inner or dot product in R : u v = u · v = u v1 + ···u](https://reader034.vdocuments.net/reader034/viewer/2022042507/588ef4711a28abd04e8b49f2/html5/thumbnails/4.jpg)
4
Exercise:u ⊥ v⇐⇒ ‖u‖2 + ‖v‖2 = ‖u + v‖2
Angles: We define the angle cosinebetween nonzero u and v to be
cos θ = u·v‖u‖ ‖v‖examples→
This fits with the law of cosines!
![Page 5: Orthogonality Inner or dot product in R : u v = u · v = u v1 + ···u](https://reader034.vdocuments.net/reader034/viewer/2022042507/588ef4711a28abd04e8b49f2/html5/thumbnails/5.jpg)
5
W ⊂ Rn a subspace.
we sayu ⊥ W
if for all v ∈ W we have
u ⊥ v.
W⊥ = {u ∈ Rn|u ⊥ W}.
![Page 6: Orthogonality Inner or dot product in R : u v = u · v = u v1 + ···u](https://reader034.vdocuments.net/reader034/viewer/2022042507/588ef4711a28abd04e8b49f2/html5/thumbnails/6.jpg)
6
Theorem: if {v1, . . . ,vk} spans Wthen
u ⊥ W
if and only if
u ⊥ vi for all i.
Theorem: W⊥ ⊂ Rn is a subspace.
Theorem: dimW + dimW⊥ = n
Theorem: (W⊥)⊥ = W .
![Page 7: Orthogonality Inner or dot product in R : u v = u · v = u v1 + ···u](https://reader034.vdocuments.net/reader034/viewer/2022042507/588ef4711a28abd04e8b49f2/html5/thumbnails/7.jpg)
7
Key examples:
Row (A)⊥ = Null (A).
Col (A)⊥ = Null (AT )
![Page 8: Orthogonality Inner or dot product in R : u v = u · v = u v1 + ···u](https://reader034.vdocuments.net/reader034/viewer/2022042507/588ef4711a28abd04e8b49f2/html5/thumbnails/8.jpg)
8
{u1, . . . ,uk} is an orthogonal setif ui ⊥ uj whenever i 6= j.
examples - fill in R3→
Theorem. If {u1, . . . ,uk} is anorthogonal set and ui are non-zero,then they are linearly independent.
prove it!→
![Page 9: Orthogonality Inner or dot product in R : u v = u · v = u v1 + ···u](https://reader034.vdocuments.net/reader034/viewer/2022042507/588ef4711a28abd04e8b49f2/html5/thumbnails/9.jpg)
9
An orthogonal set of non-zero vec-tors is a basis for its span.
Definition: An orthogonal basisof W is a basis which is an orthogo-nal set. ←just a change of per-
spective
Theorem. If {u1, . . . ,uk} is anorthogonal basis for W and we wantto decompose a vector y ∈ W as
y = c1u1 + · · · + ckukthen ←examples!!
cj =y · uiui · ui
.
![Page 10: Orthogonality Inner or dot product in R : u v = u · v = u v1 + ···u](https://reader034.vdocuments.net/reader034/viewer/2022042507/588ef4711a28abd04e8b49f2/html5/thumbnails/10.jpg)
10
Things get better if we normalize:
{u1, . . . ,uk} is an orthonormalset if it is an orthogonal set of unitvectors.
examples→
![Page 11: Orthogonality Inner or dot product in R : u v = u · v = u v1 + ···u](https://reader034.vdocuments.net/reader034/viewer/2022042507/588ef4711a28abd04e8b49f2/html5/thumbnails/11.jpg)
11
Theorem. A matrix U has or-thonormal columns if and only if
UTU = I.
Theorem. If U has orthonormalcolumns then
(Ux) · (Uy) = x · y
Corollary If U has orthonormalcolumns then
• ‖Ux‖ = ‖x‖• x ⊥ y⇐⇒ Ux ⊥ Uy
![Page 12: Orthogonality Inner or dot product in R : u v = u · v = u v1 + ···u](https://reader034.vdocuments.net/reader034/viewer/2022042507/588ef4711a28abd04e8b49f2/html5/thumbnails/12.jpg)
12
If U is a square matrix with or-thonormal columns then it is calledan orthogonal matrix.
Note: UTU = I, and U a squarematrix means UUT = I.
meaning the rows are also orthonor-mal!
![Page 13: Orthogonality Inner or dot product in R : u v = u · v = u v1 + ···u](https://reader034.vdocuments.net/reader034/viewer/2022042507/588ef4711a28abd04e8b49f2/html5/thumbnails/13.jpg)
13
Suppose {u1, . . . ,uk} is an orthog-onal basis for W but y 6∈ W . Whatcan we do?
Theorem. There is a uniqueexpression
y = yW + yW⊥
where yW ∈ W and yW⊥ ∈ W⊥.
In fact
yW = c1u1 + · · · + ckukwhere ←examples!!
cj =y · uiui · ui
.
![Page 14: Orthogonality Inner or dot product in R : u v = u · v = u v1 + ···u](https://reader034.vdocuments.net/reader034/viewer/2022042507/588ef4711a28abd04e8b49f2/html5/thumbnails/14.jpg)
14
If {u1, . . . ,uk} is orthonormal, theformula gets simpler:
yW = (y · u1)u1 + · · ·+ (y · uk)uk
Writing U = [u1 . . .uk] this gives
yW = UUTy
![Page 15: Orthogonality Inner or dot product in R : u v = u · v = u v1 + ···u](https://reader034.vdocuments.net/reader034/viewer/2022042507/588ef4711a28abd04e8b49f2/html5/thumbnails/15.jpg)
15
We call yW the perpendicular pro-jection of y on W .
Notation:
yW = projW y
So
y = projW y + projW⊥ y.
Then yW is the vector in W closestto W .
![Page 16: Orthogonality Inner or dot product in R : u v = u · v = u v1 + ···u](https://reader034.vdocuments.net/reader034/viewer/2022042507/588ef4711a28abd04e8b49f2/html5/thumbnails/16.jpg)
16
Gram-Schmidt Algorithm. Say{x1, . . . ,xk} is a basis for W .
Think about Wi = Span {x1,xi}We construct inductively vi as fol-
lows:examples→
v1 = x1
v2 = ProjW⊥1
x2 =x2−ProjW1x2
= x2 −x2 · v1
v1 · v1v1
...
vk = ProjW⊥k−1
xk
= xk −(xk·v1v1·v1
v1+···+ xk·vk−1vk−1·vk−1
vk−1
)
We can normalize: ui = 1‖vi‖
vi.
![Page 17: Orthogonality Inner or dot product in R : u v = u · v = u v1 + ···u](https://reader034.vdocuments.net/reader034/viewer/2022042507/588ef4711a28abd04e8b49f2/html5/thumbnails/17.jpg)
17
QR factorization
Matrix interpretation of Gram-Schmidt:sayA has linearly independent columns
A = [x1 · · ·xn]
and say Q = [u1 · · ·un].
Since Span (u1, . . . ,ui) = Span (x1, . . . ,xi)we have
xi = r1iu1 + · · · + riiui
meaning xi = Q
r1i...rii0...0
So A = [x1 · · ·xn] = Q[r1 · · · rn].
![Page 18: Orthogonality Inner or dot product in R : u v = u · v = u v1 + ···u](https://reader034.vdocuments.net/reader034/viewer/2022042507/588ef4711a28abd04e8b49f2/html5/thumbnails/18.jpg)
18
In other words A = QR, where Qhas orthonormal columns, and R isinvertible upper triangular.
Note: R = QTA
![Page 19: Orthogonality Inner or dot product in R : u v = u · v = u v1 + ···u](https://reader034.vdocuments.net/reader034/viewer/2022042507/588ef4711a28abd04e8b49f2/html5/thumbnails/19.jpg)
19
Least squares approximation.
Say Col A does not span b.
You still want to approximate
Ax ≈ b.
We’ll calculate x̂ such that
‖Ax̂− b‖is minimized. Then we’ll apply.
![Page 20: Orthogonality Inner or dot product in R : u v = u · v = u v1 + ···u](https://reader034.vdocuments.net/reader034/viewer/2022042507/588ef4711a28abd04e8b49f2/html5/thumbnails/20.jpg)
20
The closest an element Ax̂ of thecolumn space gets to b is
Gram-Schmidt?ouch!→
b̂ = projCol Ab.
Now b̂ ∈ Col A, so we can solve
Ax̂ = b̂.
This seems hard. But. . .
![Page 21: Orthogonality Inner or dot product in R : u v = u · v = u v1 + ···u](https://reader034.vdocuments.net/reader034/viewer/2022042507/588ef4711a28abd04e8b49f2/html5/thumbnails/21.jpg)
21
Ax̂ = b̂ = projCol Ab.
⇐⇒(b− b̂) ⊥ Col A
= Row AT
⇐⇒
AT (b− Ax̂) = 0
⇐⇒
ATA x̂ = ATb
![Page 22: Orthogonality Inner or dot product in R : u v = u · v = u v1 + ···u](https://reader034.vdocuments.net/reader034/viewer/2022042507/588ef4711a28abd04e8b49f2/html5/thumbnails/22.jpg)
22
Linear regression
Theoretically a system behaves like
y = β0 + β1x,
and you want to find βi.
You run experiments which give datapoints (xi, yi). They will not actu-ally lie of a line.
You get the approximate equations
β0 + x1β1 ≈ y1β0 + x2β1 ≈ y2
... ...β0 + xnβn ≈ yn
![Page 23: Orthogonality Inner or dot product in R : u v = u · v = u v1 + ···u](https://reader034.vdocuments.net/reader034/viewer/2022042507/588ef4711a28abd04e8b49f2/html5/thumbnails/23.jpg)
23
In matrix notation:1 x11 x2... ...1 xn
[β1β2
]≈
y1...yn
or
Xβ ≈ y.
![Page 24: Orthogonality Inner or dot product in R : u v = u · v = u v1 + ···u](https://reader034.vdocuments.net/reader034/viewer/2022042507/588ef4711a28abd04e8b49f2/html5/thumbnails/24.jpg)
24
To solve it we take
XTXβ = XTy
We can solve directly, or expand:[n
∑xi∑
xi∑x2i
] [β0β1
]=
[ ∑yi∑xiyi
]