Introduction Intersections

THE LEAST SQUARES METHOD

Robert Joan Arinyo

Grup d’Informàtica a l’EnginyeriaEscola Tècnica Superior d’Enginyeria Industrial

Universitat Politècnica de Catalunya

Màster en Computació

Introduction Intersections

Overview

1. Introduction

2. Fitting problem formulation

3. The general linear problem

4. Intersecting n lines in 2D

5. Intersecting n planes in 3D

6. Fitting a plane to n given points in 3D

Introduction Intersections

Introduction (1/4)

• The method of least squares is a standard approach tothe approximate solution of overdetermined systems, i.e.sets of equations in which there are more equations thanunknowns.

• Least squares means that the overall solution minimizesthe sum of the squares of the errors made in solving everysingle equation.

• The most important application is in data fitting .

• The best fit in the least-squares sense minimizes the sumof squared residuals, a residual being the differencebetween an observed value and the fitted value providedby a model.

Introduction Intersections

Introduction (2/4)

• Depending on whether or not the residuals are linear in allunknowns, Least squares problems fall into two categories:

1. linear least squares

2. nonlinear least squares

Introduction Intersections

Introduction (3/4)

Linear least squares occurs in statistical regression analysis. Ithas a closed-form solution.

The approach is called linear least squares since the solutiondepends linearly on the data.

Introduction Intersections

Introduction (4/4)

• The nonlinear least squares problem has no closedsolution and is usually solved by iterative refinement.

• At each iteration the system is approximated by a linearone, thus the core calculation is similar in both cases.

Introduction Intersections

Fitting Problem Formulation (1/3)

Given N points located at positions x i in Rd with i ∈ [1..N]. We

wish to obtain a globally defined function f (x) thatapproximates the given scalar values fi at points x i in such away that minimizes the error functional

JLS =∑

i

||f (x i) − fi ||2

Introduction Intersections

Fitting Problem Formulation (2/3)Illustrative example:

• Experimental data. Points shown in red in the picture

x 1 2 3 4y 6 5 7 10

• It is desired to find a line y = ax + b that fits "best" these fourpoints. In other words, we would like to find the numbers a and bthat approximately solve the overdetermined linear system

a + b = 62 a + b = 53 a + b = 74 a + b = 10

Introduction Intersections

Fitting Problem Formulation (3/3)

• The least squares approach minimizes the sum of squares oferrors or residual values , that is, to find the minimum of thefunction

R(a, b) = (6−(a+b))2+(5−(2a+b))2+(7−(3a+b))2+(10−(4a+b))2

• The minimum is determined by calculating the partial derivativesof R(a, b) with respect to a and b and setting them to zero. Thisresults in a system of two equations in two unknowns, called thenormal equations.

• When solved, we have a = 1.4 and b = 3.5. Therefore, the liney = 1.4x + 3.5 is the best least squares fit.

Introduction Intersections

Solution to the first degree problem

The common computational procedure to find a first-degreepolynomial function approximation over n data points is asfollows.

• The slope is given by

a =n

∑

xy −∑

x∑

yn

∑

x2 − (∑

x)2

• The Y-intercept is given by

b =

∑

y∑

x2 −∑

x∑

xyn

∑

x2 − (∑

x)2

Introduction Intersections

General Problem (1/10)

Consider an overdetermined system of m linear equations eachwith n unknowns such that m > n,

n∑

j=1

ajx ij = b j , (i = 1, 2, . . . , m)

Written in matrix form

XA = Y

Introduction Intersections

General Problem (2/10)

Where

X =

x11 x12 · · · x1n

x21 x22 · · · x2n...

.... . .

...xm1 xm2 · · · xmn

A =

a1

a2...

an

Y =

y1

y2...

yn

Introduction Intersections

General Problem (3/10)

Such a system usually has no solution, and the goal is then tofind the coefficients A which fit the equations "best", in thesense of minimizing the residuals:

||Y − XA||2

Introduction Intersections

General Problem (4/10)

• This minimization problem has a unique solution, providedthat the n columns of the matrix X are linearly independent

• The solution is given by solving the normal equations

(X⊤X)A = X⊤Y

Introduction Intersections

General Problem (5/10)

• Solving the normal equations

(X⊤X)A = X⊤Y

entails inverting (X⊤X).

• However, for large values of m, matrix (X⊤X) is illconditioned and thus the computation is numericallyunstable.

Introduction Intersections

General Problem (6/10)

• It is preferable to apply orthogonal decomposition methods.

• The residuals are r = Y − XA• Apply QR decomposition to get X = QR

• Q is an m × m orthogonal matrix and R is an m × n whichis partitioned into an n × n upper triangular matrix block,say Rn, and a (m − n) × n zero block, say O.

(

Rn

O

)

Introduction Intersections

General Problem (7/10)

m m m

n nm

n

m − n

0

Rn

X Q

O

Introduction Intersections

General Problem (8/10)

• Therefore, residuals r = Y − QRA can be written as

Q⊤r = Q⊤Y − (Q⊤Q)RA

=

(

(Q⊤Y)n − RnA(Q⊤Y)m−n

)

=

(

uv

)

• v doesn’t depend on A. Then the minimum residual valueis attained when the upper block, u, is zero.

Introduction Intersections

General Problem (9/10)

• Therefore the parameters are found by solving

RnA =(

Q⊤Y)

n

• These equations are easily solved as Rn is uppertriangular.

Introduction Intersections

General Problem (10/10)

HOMEWORK

Search for numerical libraries to perform the compuations sofar discussed

Introduction Intersections

Intersecting n lines in 2D

• Set of given line equations

a1x + b1y + c1 = 0a2x + b2y + c2 = 0. . .

anx + bny + cn = 0

• Residuals are ri = aix + biy − ci , 1 ≤ i ≤ n

• Apply what has been said in the solution to the first degreeproblem considering as unknowns x and y .

Introduction Intersections

Intersecting n planes in 3D

• Set of given plane equations

a1x + b1y + c1z + d1 = 0a2x + b2y + c2z + d2 = 0. . .

anx + bny + cnz + dn = 0

• Residuals are ri = aix + biy + ciz − di , 1 ≤ i ≤ n

• Apply what has been said in the solution to the generalproblem considering as unknowns x , y and z.

Introduction Intersections

Fitting a plane to n points in 3D• Set of given 3D points

x1 y1 z1

x2 y2 z2

. . .

xn yn zn

• Plane equation to be fit ax + by + cz + d = 0

• We want to find values for a, b, c and d that approximately solvethe system of equations

x1a + y1b + z1c + d = 0x2a + y2b + z2c + d = 0. . .

xna + ynb + znc + d = 0

• Apply what has been said in the solution to the general problemconsidering as unknowns a, b, c and d .

Introduction Intersections

This is it concerning Least Squares fitting