linear algebra i - ucl · vectors • the most fundamental element in linear algebra is a vector...

Linear Algebra Part I - Linear Spaces

Simon Julier

Department of Computer Science, UCL

[email protected]

http://moodle.ucl.ac.uk/course/view.php?id=11547

GV01 - Mathematical Methods, Algorithms and Implementations: Linear Algebra Part I - Linear Spaces

Structure

• Introduction

• Tuples• Vectors and Vector Spaces

• Affine Spaces• Euclidean Spaces

2

Introduction

• Introduction



3

What is Linear Algebra?

4

What is Linear Algebra?

• Scaling and adding things together!

• The tools and techniques are central for many of the modules you are going to encounter

5

Linear Algebra in Image Processing

• An image that we look at is given in its "standard”basis

• Find new basis where different features (shapes, edges, noise) are represented as basis vectors

• Do signal processing• Turn back to the original

basis

6

Linear Algebra in Computer Graphics

• Linear algebra is frequently used to construct scene graphs:– The geometry of a scene

is composed of many parts connected together in rigid or non-rigid ways

– The algebra cascades the transformations through the scene graph

7

Example Application of Vector Spaces

• One problem with processing eye witness reports is getting an accurate representation of a perpetrator

• Verbal descriptions are ambiguous and inaccurate

• Identikits are often used to mock up faces

• However, can be slow and difficult to use

8

EigenFIT System

9

Interactive Genetic Algorithm

• One solution is to generate novel sequences of face images and let people “home in” on the correct face

10

GUI and User Selection of “Fit”

However, where do these faces come from? 11

Generating the Faces

• Faces are represented using a vector-space representation and are sampled

• A large set of training data of faces is collected– 150 standard landmarks identified– All the images warped so that these landmarks line up in

each image

• A principal component analysis (PCA) is carried out– This is something we’ll touch on with

eigendecompositions, but it basically means “find the most important vectors”

12

Generating the Faces

• Faces are generated using the expressions

Geometry (shape)

Texture (appearance) Mean values

Weighted vectors

13

Results

14

Linear Algebra in Computer Vision

• Most computer vision (and graphics) assumes pinhole camera models– Situation is described by projective geometry, which is

generalised from affine geometry

15

Coordinate -Free Approach

• Where possible, these lectures will emphasise a coordinate free approach to linear algebra

• The reason for using this abstraction is that it emphasises that we are dealing with geometric objects

• Just looking at the mechanics (coordinates only) can be misleading:– Points are not vectors, but they are often represented the

same way (e.g., array of 3 doubles)– Doesn’t easily generalise to more than 2 or 3 dimensions

16

Coordinate -Free Concepts in Java3D

17

Roadmap of the Material

Affine SpacePoints and geometric objects

Tuples Vector Space Relative displacement, subspaces, basis

General linear algebra concepts

Specific geometric applications

Angles, distances, areas, volumesEuclidean Space

The standard (x,y,z) coordinate systemCartesian Frame

18

Structure

• Introduction



19

Tuples

• Much of linear algebra is about manipulating tuples• A tuple is an ordered list of elements

• Ordered list -> the position matters• Element -> what sits in each position (can be another

tuple)• Vectors and points are special examples of tuples

which we’ll work with extensively

20

Tuples Describe Directed Graphs

• A directed graph consists of a set of vertices and edges

21

Tuples Describe Colour

• There are lots of ways to parameterise colour depending on the application, physics, etc.

• Most use 3 or 4 real numbers

Additive (RGB) Subtractive (CMYK) YUVHSV

22

Arithmetic Operations on Tuples

• Addition (when defined):

• Subtraction (when defined):

• Scalar multiplication and division (when defined):

23

Multiplication of Tuples

• The only general product between tuples is the Cartesian product

• The number of elements in each tuple does not have to be the same

• The type of elements in each tuple does not have to be the same

24

Tuple Mini-Quiz

• Consider the following set of tuples:

– The suits:

– The faces:

– The numbers:

25

Tuple Mini-Quiz

• Compute the following:

1. i+j2. 17i-3j

3. s × c4. i × s × c

26

Real-Valued Tuples

• We shall mostly be concerned with real-valued tuplesof the same length

• Therefore, we can also define the inner product

27

Properties of Tuple Products

• Commutative:

• Associative:

• Distributive:

28

Summary

• Tuples are the basic “building blocks” for linear algebra

• They are ordered lists of elements• Defined operations include:

– Addition (of the same type when it’s sensible)– Scaling (when it’s sensible)– Multiplication (can mix tuples of different types)

• For most of this module, we are concerned with real-valued tuples and so we can use the inner product

29

Vectors and Vector Spaces

• Introduction



30

Vectors

• The most fundamental element in linear algebra is a vector

• Vectors are special types of tuples which satisfy various types of scaling and addition operations

• We shall meet several kinds in this course:– Coordinate vector spaces– Functional vector spaces (perhaps)– Probabilistic vector spaces (perhaps)

• Vectors actually “live” in a vector space, but we can’t talk about this until we’ve look a bit more at the properties of vectors

31

Geometric Interpretation of Vectors

• In geometry, a vector encodes relative displacement– Length– Direction

• It does not encode absolute position

32

Equivalency of Vectors

• Two vectors are the same iff (if and only if):– They have the same direction– They have the same length

Equivalent vectors are parallel and of the same length

33

Scaling Vectors by a Positive Constant

• Length changed

• Direction unchanged34

Scaling Vectors by a Negative Constant

• Length changed

• Direction reversed

35

Addition and Subtraction of Vectors

• Summation or differencing of vectors “closes the triangle”

36

Other Properties of Vectors

• Commutativity

• Associativity

• Distributivity of addition over multiplication

• Distributivity of multiplication over addition

37

Vector Spaces

• A vector space has the following properties:– Addition and subtraction is defined, and the result is another

vector– The set is closed under linear combinations:

• Given and the combination

– There is a zero vector such that

38

Vector and Vector Space Mini-Quiz

• Which of the following are vector spaces?

1.A tuple of n real numbers:

2.The zero-vector:

3.The coefficients of quadratic polynomials:

4.The angle displayed on a compass:

39

Summary

• Vectors encode relative displacement, not absolute position

• Scaling a vector changes its length, but not its direction

• All vectors “live” in a vector space

• The vector space is closed under linear combinations and possesses a zero vector

4040

Subspaces, Basis, Dimensions and Coordinates

• The previous part showed that we can add vectors

• However, we can look at the structure of vector spaces in greater detail

• The means for doing this is, surprisingly enough, to consider a way of representing a vector as a set of coordinates with respect to a spanning set

4141

Coordinate Representation of a Vector

• Consider the following problem

• I am given a spanning set of vectors,

• I am given a vector

• I form the linear combination of vectors in S

42


• What properties must be obeyed by S if I can alwaysfind a unique set of coefficients so that

43


• From the closure property, we know that any linear combination

must automatically obey the property

• Therefore, we need to prove two things:– The coefficients are unique– Any vector in can be written in this form

4444

Arbitrary S Can Be Ambiguous

45

2D Vector Example

• Here,

• Therefore,

46

Polynomial Example

• Consider the vector space for the coefficients of quadratics,

• If our spanning set is

we compute the polynomial

4747

Linear Independence

• The number of vectors we can put into the spanning set S is unlimited

• However, if some of the vectors can be written in terms of other vectors, this leads to redundancy– The same vector can be written in an infinite number of ways

using the spanning set

• The way to eliminate this redundancy is to use vectors which are linearly independent

48

Linear Independence

• A set is linearly independent if

only for the special case

• In other words, we can’t write any member of the set as a linear combination of the other members of the set

49

Linearly Independent Vectors

50

Arbitrary S Might Not “Fill the Space ”

• The spanning set of vectors might be independent, but might not “point” in all the directions in

51

Polynomial Example

• For our polynomial,

• Suppose our spanning set is

• This is linearly independent, but we can only construct quadratics of the form

5252

Subspaces

• We formalise this by saying that given the spanning set

the linear combination

comprises the subspace

53

Dimension of a Subspace

• The dimension n of is the maximum number of linearly independent vectors in

• If the dimension of is less than that of then we are dealing with a subspace

• The coursework will ask you to prove a method which can be used to compute the number of dimensions from a spanning set

54

Basis of a Subspace

• Since the spanning set lies in the subspace , we can also say

• If the vectors in the spanning set are linearly independent, then S is a basis for

• In other words, any vector which lies in can be written using S and a unique set of coefficients

5555

Subspaces and Basis

• The first two vectors form a basis for the subspace S

• All three vectors form a basis for the (sub)space V

56

Vector Subspaces Mini-Quiz

• Consider the set of linearly independent vectors

• Are these spanning sets dependent or independent? What’s the dimension and basis of the subspace?

57

The Question …

• Recall the question – for any vector and

where

can we find a unique set of coefficients such that

58

The Answer…

• For S a basis of , we can always represent the vector uniquely and exactly

• For S a basis of just a subspace, we can only represent some of the vector exactly

• Expressing the “closest” approximation of the vector in the subspace is a kind of projection operation, which we’ll get back to later

59

The Answer…

60

Summary

• Vector spaces can be decomposed into a set of subspaces

• Each subspace is a vector space in its own right(closure; zero vector)

• The dimension of a subspace is the maximum numberof linearly independent vectors which can be constructed within that subspace

61

Summary

• A spanning set is an arbitrary set of vectors which comprise a subspace

• If the spanning set is linearly independent, it’s also known as a basis for that subspace

• The coordinate representation of a vector in a subspace is unique with respect to a basis for that subspace

6262

Changing Basis

• In the last few slides we said we could write the coordinates of a vector uniquely given a basis set

• However, for a given subspace the choice of a basis is not unique

• For some classes of problems, we can make the problem significantly easier by changing the basis to reparameterise the problem

63

Example of Changing Basis

• Earlier we considered the vector space of quadratics,

• One basis could be

• However, another basis is

64

Example of Changing Basis

• One way to represent position on the Earth is to use Earth Centered Earth Fixed (ECEF) Coordinates

• Locally at each point on the globe, however, it’s more convenient to use East-North-Up (ENU) coordinates

• If we move to a new location, the ENU basis has to change in the ECEF frame

Local Tangent Plane65

Changing Basis

• Suppose we would like to change our basis set from

to

where both basis span the same vector space

66

Changing Basis

67

Changing Basis

• Since the subspaces are the same, each vector in the original basis can be written as a linear combination of the vectors from the new basis,

68

Changing Basis

• Clanking through the algebra, we can repeat this for all the other vectors in the original basis,

69

Changing Basis

• Now consider the representation of our vector in the original basis,

• Substituting for just the first coefficient, we get

7070

Changing Basis

• Substituting for all the other coefficients gives

71

Changing Basis Mini-Quiz

• Earlier we considered the vector space of quadratics,

• Given a set of coordinates in the basis

• Compute the coordinates in the new basis

72

Summary of Changing Basis

• Sometimes changing a basis can make a problem easier

• We can carry this out if our original basis is a subspace of our new vector space

• The mapping is unique, and corresponds to writing the old basis in terms of the new basis

• (It’s much neater to do it with matrix multiplication)

73

So What’s the Problem with Vector Spaces?

• We have talked about vector spaces– They encode displacement– There are rules for adding vectors together and scaling them– We can define subspaces, dimensions and sets of basis

vectors– We can even change our basis

• However vector spaces leave a lot out!

74

The Bits Which Are Missing

• There are no points– There is no way to represent actual geometric objects

• There is no formal definition of what things like angles, or lengths mean– Therefore, we can’t consider issues like orthogonality

• We haven’t discussed the idea of an origin– Everything floats in “free space”

• Affine spaces start to redress this by throwing pointsinto the mix

75

Affine Spaces

• Introduction



76

Affine Spaces

• An affine space extends a vector space by adding a set of points to it

• The vectors are referenced by

• The points are referenced by

• The dimension of the vector space is the dimension of

77

Geometric Interpretation of Points

• In geometry a point encodes a “zero-dimensional”place– It contains no information about length, area, volume, etc.

• (This is still not absolute because we have no origin)

7878

Important Axioms

• The relationship between points and vectors are laid out using two important axioms:– The Coordinate Axiom (defines multiplicative

operations on points)– The Head-to-Tail Axiom (codifies point

subtraction and how points and vectors fit together)

79

The Coordinate Axiom

• Multiplying a point by 1 returns the original point,

• Multiplying a point by 0 returns the zero vector

• Multiplying a point by any other scalar is not allowed

80

Head-to-Tail Axiom: Part I

81

Head-to-Tail Axiom: Part II

82

Head-to-Tail Axiom: Part III

83

Head-to-Tail Axiom Mini-Quiz

• Prove the following identities:

84

Representing Lines Using Affine Combinations

• Now that we have points, we can represent objects as an (infinite) set of those points

• For example, a line of finite length consists of allthe points which lie between the start point Qand the end point P

8585

Representing Lines Using Affine Combinations

• The vector oriented along the line is

• A point R lying on the line can be written as

Useful abuse of notation!

86

Representing Planes Using Affine Combinations

• The spanning set for the plane is computed from three points O, P,Q which lie in the plane,

• Any point R on the plane can be written as

87

General Affine Combinations

• In general, the affine combination is written as

• The combination must obey the constraint

• The reason for this restriction will become apparent later

88

Affine Combination Mini-Quiz

• Which of these are valid affine combinations?

8989

General Affine Combination Mini-Quiz

• Consider three points O, P, Q and the affinecombination

• What 2D shape is described by this combination when

9090

Affine Frames

• So far we have not had to resort to any notion of an origin

• However, in many cases origins are useful• An affine frame for an affine space is formally

defined to consist of a basis and an origin,

where and are a basis for

91

Illustration of a 3D Affine Frame

The basis is linearly independent92

Vectors and Points in Affine Frames

• Given an affine frame, we can define points and vectors accordingly,

• The arithmetic operations from the head-to-tail axiom can be carried out directly in the affine frame

93

Frame-Based Addition of Vectors

• Vectors add in a completely straightforward manner

• Given two vectors,

• Then

9494

Frame-Based Scalar Multiplication of Vectors

• Scaling a vector by a scalar scales the coordinates of the vector

• This changes the length but not the direction of the vector

95

Frame-Based Point and Vector Arithmetic

• Adding a point to a vector gives a new point

• The position is the sum of the point position vector and the additive vector

96

Frame-Based Point Arithmetic

• Subtracting points leads, as we expect, to a vector

97

Frame-Based Point and Vectors Mini-Quiz

• So… What’s wrong with calculating P+Q?

9898

Frame-Based Affine Combinations

• The coordinate axiom also explains the seemingly arbitrary constraint on affine combinations that

• Recall that an affine combination is

• In turn, each point can be written as

99

Frame-Based Affine Combinations

• Substituting into the expression for the affinecombination,

• This is only a valid point if the coefficient on is 1, which can only happen if the coefficients sum to 1

100

Barycentric Coordinates

• So far we’ve treated the coordinate representation of points and vectors a bit differently

• The relationship between affine combinations

and

is no coincidence• It is possible to express both using barycentric

coordinates

101101


• We consider each point used in our affine combination as being used to form a spanning set,

102102


• A point is simply an affine combination of the “basis points” or the n-dimensional simplex,

where if P is a valid point then

103103

Barycentric Coordinates in Action

104104

Vectors in Barycentric Coordinates

• Now, a vector can be written as

but we’d like to similarly express it in barycentriccoordinates in the form

105105


• Substituting the expressions for the points in terms of the vectors,

What we want Additional point bits

106106


• We eliminate the point bits by setting the constraint

• In other words, vectors are represented as barycentriccoordinates whose values sum to zero

107107

Barycentric Coordinates Mini-Quiz

• How do you think affine subspaces can be represented using barycentric coordinates?

• How do you think affine independence can be represented using barycentric coordinates?

108108

Summary of Barycentric Coordinates

• We can use sets of points to define a basis• For an n-dimensional space we use n+1 points known

as an n-dimensional simplex• In this basis, points are represented by coordinates

that sum to 1• Vectors are represented by coordinates that sum to 0• Affine subspaces correspond to lower dimensional

simplexes• Affine independence happens when no point is an

affine combination of the other points

109109

So What’s the Prob lem with Affine Spaces?

• We’ve talked about affine spaces– We now have points as well as vectors– We have discussed the algebra which links points and

vectors together– We can describe geometric objects using affine

combinations and barycentric coordinates

• However, a few useful things are still missing!

110

More Missing Bits

• We still don’t have a notion of distances– Just ratios on lines between points

• We still don’t have a notion of angles

• We still don’t have an absolute origin

• These are all introduced in Euclidean geometry

111

Euclidean Spaces

• Introduction



112

Euclidean Spaces

• Euclidean spaces extend affine spaces by adding notions of length and angle

• The Euclidean structure is determined by the forms of the equations used to calculate these quantities

• We are going to just “give” these without proof• However, we can motivate the problem by considering

the problem of orthogonally projecting one vector onto another vector

• First, though, we need some additional vector notation

113

Some More Vector Notation

• Since we are going to define lengths and directions, we can now decompose a vector into the product of its length and an orthonormal vector which defines its direction

Length(+ve)

Orthonormal vector(length=1)

114

Projecting Vectors onto Vectors

• Consider two vectors and that occupy the sameaffine space

• What’s the orthogonal projection of onto ?

115

The Answer…

Projected vector

116

The Answer…

Projected vector117

Projected vector

Computing the Answer

• We need to compute both the direction and length of the projection vector

• The direction of the vector must be parallel to

• Therefore, we are going to define the orthogonal projection as

118

Scalefactor Orthonormal vector

parallel to the “right direction

Length of the Scaled Projection

• From basic trigonometry, the length of the parallel projection is

• From our assumed solution the length must be

The dodgy bit!

119

Special Case of the Scalar Product

• The analysis from the past few slides shows that

• Furthermore, cosθ gives us the “right” sign behaviour

• Therefore,

120

General Case of the Scalar Product

• If we now let both of our vectors be non-normalised, then

• The scalar product is a special case of an inner product (we might meet these again later)

121

Lengths, Angles and Distances

• Lengths and angles are (circularly) defined as

• The distance function (or metric) between two points is the length of the vector between them,

122

Properties of Scalar Products

• Bilinearity:

• Positive definiteness:

123

Properties of Scalar Products

• Commutativity:

• Distributivity of the dot product over vector addition,

• Distributivity of vector addition over the dot product,

124

Scalar Product and Projection Mini-Quiz

• What’s the value of

125

Summary

• The Euclidean structure is defined by the scalar product

• The scalar product is used to compute the orthogonal projection of one vector onto another vector

• The scalar product works in any number of dimensions• It has many useful properties including bilinearity

• However, it only defines a one dimensional quantity (length)

• Vector products generalise this

126

Vector Products

• The vector product is the familiar cross product and is defined to be

where is orthogonal to and

• The vector product is a special case of an exterior product and is a pseudovector (we won’t meet these again)

• The vector product is only defined in 3D127

Magnitude of the Vector Product

• The magnitude of the vector product

is the area of the parallelogram having the vectors as sides128

Direction of the Vector Product

• In general, in 3D we have 3 axes of rotation• However, if we rotate and such that they still lie in

the same plane, still points in the same direction• Therefore, encodes information about two axes of

rotation129

Sign of the Vector Product

• Given the plane defined by the vectors, there are two possible choices of the normal vector – out of or into the plane

• The choice is determined by the sign of the angle between the vectors– Anticlockwise is positive

130

Right Hand Rule

131

Properties of Vector Products

• Anticommutativity:

• Distributivity:

• Distributivity:

• Parallelism:

132

Scalar Triple Product and Volumes

• We have shown that:– Scalar products define lengths and angles– Vector products define areas and a sense of perpendicularity

• Therefore, is it possible to extend this, at least in 3D, to the notion of a volume?

• The scalar triple product computes the volume of the parallelepiped defined by three linearly independent vectors

133

Scalar Triple Product and Volumes

134

Volume of the Parallelepiped

• The volume is computed as follows

• This is the absolute value of the scalar triple product

135

Scalar Triple Product

• The term inside the absolute value is the pseudo-scalar

which is known as the scalar triple product• Pseudo-scalars reverse their sign under inversion,

• Fortunately we don’t need to look at them any further!

136136

Cartesian Frames

• So far we’ve said an affine frame is simply an origin point and a set of linearly independent vectors

• Now that we can talk about angles and lengths, we can define a Cartesian coordinate frame

• A Cartesian Frame

has the property that its basis vectors are orthonormal

137

Orthonormal Basis Set

• An orthonormal basis set has the property that its vectors:– Are orthogonal to one another– Are of unit length

• More compactly,

138

Illustration of the Cartesian Basis Set

The basis is orthonormal

139

Cartesian Frames in 3D (Finally!)

• The basis set in 3D is written using the familiar vectors,

• Any vector can be written as

140

Scalar and Vector Product in the Cartesian Frame

• The scalar product is given by

• The vector product is given by

141

Euclidean Space Summary

• Euclidean spaces add notions of length and angles through the scalar product

• These can be extended by the vector and vector triple products to give areas and volumes in 3D

• The Cartesian Coordinate frame is a special case with a dedicated origin and an orthonormal basis set

142

linear algebra i - ucl · vectors • the most fundamental element in linear algebra is a vector...

Documents