lecture07 - least squares regression

8/9/2019 Lecture07 - Least Squares Regression

1/32

Curve Fitting and Interpolation

Lecture 7:

Least Squares Regression

MTH2212 Computational Methods and Statistics


2/32

Dr. M. HrairiDr. M. Hrairi MTH2212MTH2212 -- Computational Methods and StatisticsComputational Methods and Statistics 22

Objectives

Introduction

Linear regression

Polynomial regression

Multiple linear regression

General linear least squares

Nonlinear regression


3/32

Dr. M. HrairiDr. M. Hrairi MTHMTH22122212 -- Computational Methods and StatisticsComputational Methods and Statistics 33

Curve Fitting

Experimentation

Data available at discrete points or times

Estimates are required at points between the discrete values

Curves are fit to data in order to estimate the intermediate values


4/32


Curve Fitting

Two methods - depending on error in data

Interpolation

- Precise data

- Force through each data point

Regression- Noisy data

- Represent trend of the data

0

1

2

3

4

5

6

7

8

9

0 2 4 6 8 10 12 14 16

x

f(x)

0

20

40

60

80

100

120

0 1 2 3 4 5

Time (s)

Tem

perature(degF)


5/32


Least Squares Regression

Experimental Data

Noisy (contains errors or inaccuracies)

x values are accurate, y values are not

Find relationship betweenxand y = f(x)

Fit general trend without matching individual points

Derive a curve that minimizes the discrepancy between the data

points and the curve Least-squaresregression


6/32


x y

2.10 2.90

6.22 3.83

7.17 5.98

10.5 5.71

13.7 7.74

Linear Regression: Definition

0

1

2

3

4

5

6

7

8

9

0 2 4 6 8 10 12 14 16

x

f(x)

Straight line characterizes trend without

passing through particular point

Noisy Data From

Experiment

xaay10

!


7/32


Linear Regression: criteria for a best fit

How do we measure goodness of fit of the line to the data?

0

1

2

3

4

5

6

7

8

9

0 2 4 6 8 10 12 14 16

x

f(x)

y1 y2

y4

y5

y3

e 3

e2

Regression Model

y = a 0+ a 1x

Residual

e = y - (a 0+ a 1x)

Data points


8/32


Linear Regression: criteria for a best fit

Use the curve that minimizes the residual between the data

points and the line

Model:

Find the values of a0 and a1 that minimize Sr

xy10

!

ii xaay 10 !

iii xaa= ye 10

n

i=ii

n

i=ir xaay=e=S

1

210

1

20

1

2

3

4

5

6

7

8

9

0 2 4 6 8 10 12 14 16

x

f(x)

y1 y2

y4

y5

y3

e 3

e 2

Regression Model

y = a 0+ a 1x

Residual

e = y - (a 0+ a 1x)

Data points

x y

2.10 2.90

6.22 3.83

7.17 5.98

10.5 5.71

13.7 7.74


9/32


Linear Regression: Finding a0 and a1

Minimize Sr by taking

derivatives WRT

a0and a1,

First a0

? A

? A ? A

!

!

n

i=

n

i=

a

a

1 0

1

2

0

2 ..

.

x

x

x

x

-

n

i=

ii

r x=S

1

2

10

00 x

x

x

x

!

-

n

i=i

n

i=i yaxna

1

1

1

0

? A

0

12

1

10

)(xaayn

i=

ii

!

! Finally


10/32


Linear Regression: Finding a0 and a1

Finally

Minimize Sr by taking

derivatives WRT

a0and a1

Second a1

-

n

i ii

r aaa

a 1

2

10

11x

x

x

x

? A

? A ? A

!

!

n

i=

n

i=

a

a

1 1

1

2

1

2 ..

.

x

x

x

x

? A

0

2

1

10

)(i

iii

!

!

!

-

-

iii

ii

ii

1

1

1

20

1


11/32


Linear Regression: Normal equations

!

-

-

n

i=ii

n

i=i

n

i=i yxaxax

1

1

1

20

1

!

-

n

i=i

n

i=i yaxna

11

10

Set of two simultaneous linear equations with two unknowns( a0 and a1):


12/32


Linear Regression: Solution of normal equations

2

11

2

111

1

2

11

2

111

2

1

0

1

1

1

11

-

!

-

!

n

i=i

n

i=i

n

i=i

n

i=i

n

i=ii

n

i=i

n

i=i

n

i=ii

n

i=i

n

i=i

n

i=i

xn

x

yxn

yx

a

xn

x

yxxn

xyn

a

The normal equations can be solved simultaneously for:


13/32


Example 1

Fit a straight line to the values in the following table

x y

2.10 2.90

6.22 3.83

7.17 5.98

10.5 5.71

13.7 7.74


14/32


Example 1 - Solution

The intercept and the slope can be calculated with:

i xi yi xi2

xiyi

1 2.10 2.90 4.41 6.09

2 6.22 3.83 38.69 23.82

3 7.17 5.98 51.41 42.88

4 10.5 5.71 110.25 59.96

5 13.7 7.74 187.69 106.04

39.69 26.1

6

392.32 238.7

4

2

11

2

1111

2

11

2

111

2

10

1

1

1

11

-

!

-

!

n

i=i

n

i=i

n

i=i

n

i=i

n

i=ii

n

i=i

n

i=i

n

i=ii

n

i=i

n

i=i

n

i=i

xn

x

yxn

yx

a

xn

x

yxxn

xyn

a


15/32



? A

? A

4023.0

69.39

5

13.392

)16.26)(69.39(5

17.238

038.2

69.39513.392

)7.238)(69.39(5

1)3.392)(16.26(

5

1

21

2

0

!

!

!

!

a

a

x..y 402300382 !

The values of the intercept and the slope:

The equation of the straight line linear regression


16/32


Linear Regression: Quantification of error

Suppose we have data points (xi, yi) and modeled (orpredicted) points (xi, i) from the model = f(x).

Data {yi} have two types of variations;(i) variation explained by the model and

(ii) variation not explained by the model.

Residual sum of squares: variation not explained by themodel

Regression sum of squares: variation explained by themodel

The coefficient of determination r2

!

!n

iiir yyS

1

2)(

!

!n

i

it yyS1

2)(

t

rt

S

SSr

!2


17/32


Linear Regression: Quantification of error

For a perfect fit

Sr=0 and r=r2=1, signifying that the line explains 100% of

the variability of the data.

For r=r2

=0, Sr=St, the fit represents no improvement.

x1 x2

y1

y2

y

Total variation in y = Variation explained by the model + Unexplained variation (error)


18/32


Linear Regression: Another measure of fit

In addition to r2, r

Define

= standard error of the estimate- Represents the distribution of the residuals around the

regression line

- Large Sy|xlarge residuals

- Small Sy|xsmall residuals

2| n

SS rxy


19/32


Example 2

Compute the total standard deviation, the standard error of

the estimate, and the correlation coefficient for the data in

Example 1.

x y

2.10 2.90

6.22 3.83

7.17 5.98

10.5 5.71

13.7 7.74


20/32



The standard deviation is

The standard error of the estimate is

The correlation coefficient r is

9028.115

4819.14

1!

!

!

n

SS

t

y

8092.025

9643.1

2| !

!

!

n

SS

r

xy

8644.4819.14

9643.14819.142!

!

!

t

rtr

i xi yi (yi-y)2

(yi-a0-a1xi)2

1 2.1 2.9 5.4382 0.0003

2 6.22 3.83 1.9656 0.5045

3 7.17 5.98 0.5595 1.11834 10.5 5.71 0.2285 0.3049

5 13.7 7.74 6.2901 0.0363

39.69 26.1

6

14.481

9

1.9643

9297.08644.0 !!r


21/32


22/32


Linearization of Nonlinear Relationships


23/32


Polynomial Regression

Minimize the residual between the data points and the curve

-- least-squaresregression

Must find values ofa0 ,a1,a2, am

ii xaay 10 !Linear

2

210 iii xaxaay !Quadratic

2

210 iiii xaxaxaay !Cubic

Generalm

imiiii xxxxy .3

32

210


24/32



Residual

Sum of squared residuals

Minimize by taking derivatives

)(3

32

210

m

imiiiii xxxxe .

n

i

mm

n

iir aaaaaeS

1

233

2210

1

2 )]([ .


25/32



Normal Equations

-

!

-

-

n

i=i

mi

n

i=ii

n

i=ii

n

i=

i

mn

i=

mi

n

i=

mi

n

i=

mi

n

i=

mi

n

i=

mi

n

i=i

n

i=i

n

i=i

n

i=

mi

n

i=i

n

i=i

n

i=i

n

i=

mi

n

i=

i

n

i=

i

yx

yx

yx

y

a

a

aa

xxxx

xxxx

xxxx

xxxn

1

1

2

1

1

2

1

0

1

2

1

2

1

1

1

1

2

1

4

1

3

1

2

1

1

1

3

1

2

1

11

2

1

/

/

/1///


26/32


Example 3

Fit a third-order polynomial to the data given in the Table

below

x 0 1.0 1. 2.3 2. 4.0 5.1 6.0 6.5 7.0 8.1 9.0

y 0.2 0.8 2.5 2.5 3.5 4.3 3.0 5.0 3.5 2.4 1.3 2.0

x 9.3 11.0 11.3 12.1 13.1 14.0 15.5 16.0 17.5 17.8 19.0 20.0

y -0.3 -1.3 -3.0 -4.0 -4.9 -4.0 -5.2 -3.0 -3.5 -1.6 -1.4 -0.1


27/32



-

!

-

-

n

iii

n

iii

n

iii

n

ii

n

ii

n

ii

n

ii

n

ii

n

ii

n

ii

n

ii

n

ii

n

ii

n

ii

n

ii

n

ii

n

ii

n

ii

n

ii

a

a

a

a

n

1

3

1

2

1

1

3

2

1

0

1

6

1

5

1

4

1

3

1

5

1

4

1

3

1

2

1

4

1

3

1

2

1

1

3

1

2

1

-

!

-

-

3643

2603

316301

22351 11612 014252 3546342

12 014252 354634223060

252 354634223060622463422306062224

3

2

1

0

.

.

.

.

....

....

....

...

-

-

01210

35320

30512

35930

3

2

1

0

.

.

.

.

a

a

a

a


28/32



Regression Equation

y = - 0.359 + 2.305x - 0.353x2 + 0.012x3

-6

-4

-2

0

2

4

6

0 5 10 15 20 25

x

f(x)


29/32


Multiple Linear Regression

y = a0 + a1x1 + a2x2 + e

Again very similar.

Minimize e

Polynomial and multipleregression fall within

definition of General

Linear Least Squares.


30/32


General Linear Least Squares

_ a ? A_ a _ a

? A

_ a_ a

_ a residualsE

tscoefficienunknownA

variabledependenttheofvaluedobservedY

t variableindependentheofvaluesmeasuredat thefunctionsbasistheofvaluescalculatedtheofmatrix

functionsbasis1are10

221100

!

!

Z

EAZY

m, z,, zz

ezazazazay

m

mm

-

.

2

1 0

! !

!

n

i

m

j

jijir zayS

Minimized by taking its partial

derivative w.r.t. each of the

coefficients and setting the

resulting equation equal to zero


31/32


Nonlinear Regression

Not all equations can be broken down into General Linear

Least Squares model i.e.

Solve with nonlinear least squares using iterative methods

like Gauss-Newton method

Equation could possibly be transformed into linear form

Caveat: when fitting transformed data you minimize the residuals of the

data you are working with

May not give you the exact same fit as nonlinear regression on

untransformed data

eeayxa! )1( 10


32/32


Nonlinear Regression

eeayxa! )1( 10

lecture07 - least squares regression

Documents