w11 linear regression

8/2/2019 W11 Linear Regression

1/29

Centre for Computer Technology

ICT114Mathematics for

Computing

Week 11

Linear Regression


2/29

March 20, 2012March 20, 2012 Copyright Box Hill Institute

ObjectivesObjectives

Review week 10Review week 10

Curve FittingCurve Fitting

RegressionRegressionDependent/Independent variableDependent/Independent variable

Method of Least SquaresMethod of Least Squares

Least Squares lineLeast Squares line


3/29


Numerical DifferentiationNumerical Differentiation

Newtons Forward Difference FormulaNewtons Forward Difference Formula

ff//

(x)= (1/h) [ (x)= (1/h) [ 11

//22 2 +2 +

++11

//33 33

11//44

44 +.]+.]

Newtons Backward Difference FormulaNewtons Backward Difference Formulaff//(x)= (1/h) [(x)= (1/h) [ ++11//22

22++11//3333

++ 11//4444+..]+..]


4/29


Numerical IntegrationNumerical Integration

Trapezoid RuleTrapezoid Rule

= (= (hh//22) [ f(a) [ f(a00) + 2 f(a) + 2 f(a11) + 2f(a) + 2f(a22) +.) +.

+ 2 f(a+ 2 f(a

n-1n-1) + f(a) + f(a

nn)])]

Simpsons One Third Rule (the number ofSimpsons One Third Rule (the number ofintervals have to be even)intervals have to be even)

= (h/3)[ ( f(a= (h/3)[ ( f(a00) + f (a) + f (ann) )) )

+ 4 (f(a+ 4 (f(a11) + f (a) + f (a33) + f(a) + f(a55)+ f(a)+ f(an-1n-1) )) )

+ 2 (f(a+ 2 (f(a22) + f (a) + f (a44) + f(a) + f(a66) + .. f(a) + .. f(an-2n-2) ) ]) ) ]


5/29




6/29


Curve Fitting (1)Curve Fitting (1)

A relation between two variables isA relation between two variables is

expressed mathematically by an equationexpressed mathematically by an equation

connecting both.connecting both.For example if x and y are the height andFor example if x and y are the height and

weight of an individualweight of an individual

Then a sample of n individuals will haveThen a sample of n individuals will haveheight xheight x11, x, x22, .., x, .., xnn

weight yweight y11, y, y22, .., y, .., ynn


7/29



Plotting the points (xPlotting the points (x11, y, y11), (x), (x22,y,y22), .(x), .(xnn,y,ynn))

on a rectangular coordinate system willon a rectangular coordinate system will

result in aresult in a scatter diagram.scatter diagram.The data in the scatter diagram can beThe data in the scatter diagram can be

generallygenerally approximated by a smoothapproximated by a smooth

curve.curve.The resulting curve is called theThe resulting curve is called the

approximating curveapproximating curve..


8/29



9/29



If the data is approximated byIf the data is approximated bystraight line there is a linear relationshipstraight line there is a linear relationship

between the variablesbetween the variableselse, there is a nonlinear relationshipelse, there is a nonlinear relationship

Finding equations toFinding equations to approximating curvesapproximating curvesthat fit the given set of data is called curvethat fit the given set of data is called curve

fitting.fitting.


10/29



11/29



The general equation of a straight isThe general equation of a straight is

y = a +bx (linear relation)y = a +bx (linear relation)

The general equation for a parabola or aThe general equation for a parabola or a

quadratic equation isquadratic equation is

y = a + bx + cxy = a + bx + cx22 (nonlinear relation)(nonlinear relation)


12/29



13/29


Regression/Regression/Method of Least SquaresMethod of Least Squares


14/29


RegressionRegression

In curve fitting we need to determine oneIn curve fitting we need to determine one

of the variables (the dependent variable)of the variables (the dependent variable)

from the other (the independent variable)from the other (the independent variable)

This process of estimation is calledThis process of estimation is called

regressionregression


15/29

March 20, 2012March 20, 2012

Copyright Box Hill Institute

Dependent/Independent VariableDependent/Independent Variable

y = a + bxy = a + bx

determine the variables?determine the variables?

the corresponding equation is called thethe corresponding equation is called the

regression equation of y on xregression equation of y on xthe corresponding curve is called thethe corresponding curve is called theregression curve of y on xregression curve of y on x


16/29

March 20, 2012March 20, 2012


Method of Least Squares (1)Method of Least Squares (1)

It is a method to find the best fitting curveIt is a method to find the best fitting curve

for the data points (scatter diagram)for the data points (scatter diagram)

Let (xLet (x11, y, y11), (x), (x22,y,y22), .(x), .(xnn,y,ynn) be the data) be the datapoints.points.

LetLet C be the best fitting curveC be the best fitting curve

For a given value of x, xFor a given value of x, x11 there will be athere will be adifference between the corresponding ydifference between the corresponding y

and the value determined by C.and the value determined by C.


17/29

March 20, 2012March 20, 2012



18/29

March 20, 2012March 20, 2012Copyright Box Hill Institute


For the data points (xFor the data points (x11, y, y11), (x), (x22,y,y22), .), .

(x(xnn,y,ynn). Let the corresponding). Let the corresponding deviationsdeviations

be dbe d11, d, d22,, d,, dnn..The measure of the curve is provided byThe measure of the curve is provided by

D = dD = d1122 + d+ d22

22++d++dnn22

IfIfD is small the fit is goodD is small the fit is good

IfIfD is large the fit is badD is large the fit is bad


19/29



The curve having this property is said to fit theThe curve having this property is said to fit the

data indata in least squares senseleast squares sense

The curve is called aThe curve is called a least squares regressionleast squares regression

curve or a least squares curvecurve or a least squares curve

Dependent VarDependent Var yy

Independent VarIndependent Var xx

OffsetOffset verticalvertical


20/29


Least Squares LineLeast Squares Line

For the data points (xFor the data points (x11, y, y11), (x), (x22,y,y22), .), .

(x(xnn,y,ynn))

LetLet y = a+bxy = a+bx is the least squares lineis the least squares linea and b are determined by solving thea and b are determined by solving the

(normal) equations(normal) equations

y = an + b xy = an + b x

xy = a x + b xxy = a x + b x22


21/29


Least Squares Line (y on x)Least Squares Line (y on x)

( y) (x( y) (x22) ( x) (xy)) ( x) (xy)

a = ---------------------------------------------a = ---------------------------------------------

n xn x22 (x) (x)22

n xy ( x)( y)n xy ( x)( y)

b = ----------------------------------------------b = ----------------------------------------------

n xn x22 (x) (x)22


22/29


ExampleExample

The table below shows the respective heights xThe table below shows the respective heights x

and y of a sample of 12 fathers and their oldestand y of a sample of 12 fathers and their oldest

sons. Find the least squares regression line of ysons. Find the least squares regression line of y

on x.on x.

Father (x) 65 63 67 64 68 62 70 66 68 67 69 71

Son (y) 68 66 68 65 69 66 68 65 71 67 68 70


23/29

March 20, 2012March 20, 2012


x y x2 xy y2

65

6367

64

68

62

70

66

68

67

69

71

68

6668

65

69

66

68

65

71

67

68

70

4225

39694489

4096

4624

3844

4900

4356

4624

4489

4761

5041

4420

41584556

4160

4692

4092

4760

4290

4828

4489

4692

4970

4624

43564624

4225

4761

4356

4624

4225

5041

4489

4624

4900

x = 800 y = 811 x2 = 53418 xy = 54107 y2 = 54849


24/29

March 20, 2012March 20, 2012


the normal equations arethe normal equations are

12a + 800b = 81112a + 800b = 811

800a + 53418b = 54107800a + 53418b = 54107

solving the above equations we getsolving the above equations we get

a = 35.82, b = 0.4776a = 35.82, b = 0.4776y = 35.82 + 0.4776xy = 35.82 + 0.4776x is the equation for theis the equation for the

regression lineregression line


25/29

March 20, 2012March 20, 2012


QuestionQuestion

The table below shows theThe table below shows the Consider the

variation of the bulk modulus of Silicon Carbide

as a function of temperature. Find the leastFind the least

squares regression line of y (G) on x (T).squares regression line of y (G) on x (T).


26/29

March 20, 2012March 20, 2012


SummarySummary


A relation between two variables isA relation between two variables is

expressed mathematically by an equationexpressed mathematically by an equationconnecting both.connecting both.

Finding equations approximating curvesFinding equations approximating curves

that fit the given set of data is called curvethat fit the given set of data is called curvefitting.fitting.


27/29

March 20, 2012March 20, 2012


SummarySummary

Method of Least SquaresMethod of Least SquaresFor the data points (xFor the data points (x11, y, y11), (x), (x22,y,y22), .), .

(x(xnn,y,ynn)) If y = a+bx is the least squares lineIf y = a+bx is the least squares linea and b are determined by solving thea and b are determined by solving the

(normal) equations(normal) equations y = an + b xy = an + b x

xy = a x + b xxy = a x + b x22


28/29

March 20, 2012March 20, 2012


SummarySummary

Method of Least SquaresMethod of Least Squares

( y) (x( y) (x22) ( x) (xy)) ( x) (xy)

a = ---------------------------------------------a = ---------------------------------------------

n xn x22 (x) (x)22

n xy ( x)( y)n xy ( x)( y)b = ----------------------------------------------b = ----------------------------------------------

n xn x22 (x) (x)22


29/29

March 20, 2012March 20, 2012


ReferencesReferences H L Verma and C W Gross : Introduction toH L Verma and C W Gross : Introduction to

Quantitative Methods,John WileyQuantitative Methods,John Wiley JB Scarborough : Numerical MathematicalJB Scarborough : Numerical Mathematical

Analysis, Jon Hopkins Hall, New JerseyAnalysis, Jon Hopkins Hall, New Jersey Gerald W. Recktenwald, Numerical MethodsGerald W. Recktenwald, Numerical Methodswith MATLAB, Implementation and Application,with MATLAB, Implementation and Application,Prentice HallPrentice Hall

Murray Spiegel, John Schiller, Alu Srinivasan,Murray Spiegel, John Schiller, Alu Srinivasan,Probability and Statistics, Schaums easyProbability and Statistics, Schaums easyOutlinesOutlines

http://mathworld.wolfram.comhttp://mathworld.wolfram.com

w11 linear regression

Documents