the linear regression model: theory and application
DESCRIPTION
TRANSCRIPT
![Page 1: The linear regression model: Theory and Application](https://reader033.vdocuments.net/reader033/viewer/2022061207/5483fb995906b594158b46e6/html5/thumbnails/1.jpg)
Statistics Lab
Rodolfo Metulini
IMT Institute for Advanced Studies, Lucca, Italy
Lesson 4 - The linear Regression Model: Theory andApplication - 21.01.2014
![Page 2: The linear regression model: Theory and Application](https://reader033.vdocuments.net/reader033/viewer/2022061207/5483fb995906b594158b46e6/html5/thumbnails/2.jpg)
Introduction
In the past praticals we analyzed one variable.
For certain reasons, it is even usefull to analyze two or morevariables together.
The question we want to asnwer regards what are the relations,the causal effects determining changes in a variable. Analyze if acertain phenomenon is endogenous or exogenous.
In symbols, the idea can be represent as follow:
y = f (x1, x2, ...)
Y is the response, which is a function (it depends on) one or morevariables.
![Page 3: The linear regression model: Theory and Application](https://reader033.vdocuments.net/reader033/viewer/2022061207/5483fb995906b594158b46e6/html5/thumbnails/3.jpg)
Objectives
All in all, the regression model is the instrument used to:
I measure the entity of the relations between two or morevariables: 4Y /4 X ,
and to measure the causal direction (4X −→ 4Y orviceversa? )
I forecast the value of the variable Y in response to somechanges in the others X1,X2, ... (called explanatories),
or for some cases that are not considered in the sample.
![Page 4: The linear regression model: Theory and Application](https://reader033.vdocuments.net/reader033/viewer/2022061207/5483fb995906b594158b46e6/html5/thumbnails/4.jpg)
Simple linear regression model
The regression model is stochastic, not deterministic.
Giving two sets of values (two variables) from a random sample oflength n: x = {x1, x2, ..., xi , ..xn}; y = {y1, y2, ..., yi , ..yn}:
Deterministic formula:
yi = β0 + β1xi , ∀i = 1, .., n
Stochastic formula:
yi = β0 + β1xi + εi ∀i = 1, .., n
where εi is the stochastic component.
β1 define the slope in the relations between X and Y (See graph inchart 1)
![Page 5: The linear regression model: Theory and Application](https://reader033.vdocuments.net/reader033/viewer/2022061207/5483fb995906b594158b46e6/html5/thumbnails/5.jpg)
Simple linear regression model - 2
We need to find β = {β0, β1} as estimators of β0 and β1.
After β is estimated, we can draw the estimated regression line,which corresponds to the estimated regression model, asfollow:
yi = β0 + β1xi
Here, εi = yi − yi .
Where yi is the i-element of the estimated Y vector, and yi is thei-elements of the real Y vector. (see graph in chart 2)
![Page 6: The linear regression model: Theory and Application](https://reader033.vdocuments.net/reader033/viewer/2022061207/5483fb995906b594158b46e6/html5/thumbnails/6.jpg)
Steps in the Analysis
1. Study the relations (scatterplot, correlations) between two ormore variables.
2. Estimation of the parameters of the model β = {β0, β1}.
3. Hypotesis tests on the estimated β1 to verify the casualeffects between X and Y
4. Robustness check of the model.
5. Use the model to analyze the causal effect and/or to doforecasting.
![Page 7: The linear regression model: Theory and Application](https://reader033.vdocuments.net/reader033/viewer/2022061207/5483fb995906b594158b46e6/html5/thumbnails/7.jpg)
Why linear?
I It is simple to estimate, to analyze and to interpret
I it likely fits with most of empirical cases, in which therelations between two phenomenon is linear.
I There are a lot of implemented methods to transorm variablesin order to obtain a linear relationship (log transformation,normalization, etc.. )
![Page 8: The linear regression model: Theory and Application](https://reader033.vdocuments.net/reader033/viewer/2022061207/5483fb995906b594158b46e6/html5/thumbnails/8.jpg)
Model Hypotesis
In order the estimation and the utilization of the model to becorrect, certain hypotesis must hold:
I E (εi ) = 0,∀i −→ E (yi ) = β0 + β1xi
I Omoschedasticity: V (εi ) = σ2i = σ2,∀i
I Null covariance: Cov(εi , εj) = 0,∀i 6= j
I Null covariance among residuals and explanatories:Cov(xi , εi ) = 0, ∀i , since X is deterministic (known)
I Normal assumption: εi ∼ N(0, σ2)
![Page 9: The linear regression model: Theory and Application](https://reader033.vdocuments.net/reader033/viewer/2022061207/5483fb995906b594158b46e6/html5/thumbnails/9.jpg)
Model Hypotesis - 2
From the hypotesis above, follow that:
I V (yi ) = σ2,∀i . Y is stochastic only for the ε component.
I Cov(yi , yj) = 0, ∀i 6= j . Since the residuals are uncorrelated.
I yi ∼ N[(β0 + β1x1), σ2] Since also the residuals are normal inshape.
![Page 10: The linear regression model: Theory and Application](https://reader033.vdocuments.net/reader033/viewer/2022061207/5483fb995906b594158b46e6/html5/thumbnails/10.jpg)
Ordinary Least Squares (OLS) Estimation
The OLS is the estimation method used to estimate the vector β.The idea is to minimize the value of the residuals.
Since ei = yi − yi we are interested in minimize the componentyi − β0 − β1xi .
N.B. εi = β0 − β1xi , while ei = β0 − β1xi
The method consist in minimize the sum of the squaredifferences:∑n
i (yi − yi )2 =
∑ni e
2i = Min,
which is equal to solve this 2 equation system derived usingderivates.
![Page 11: The linear regression model: Theory and Application](https://reader033.vdocuments.net/reader033/viewer/2022061207/5483fb995906b594158b46e6/html5/thumbnails/11.jpg)
Ordinary Least Squares (OLS) Estimation - 2
δ/δβ0
n∑i
e2i = 0 (1)
δ/δβ1
n∑i
e2i = 0 (2)
After some arithmetics, we end up with this estimators for thevector β:
β0 = y − β1x (3)
β1 =
∑ni (yi − y)(xi − x)∑n
i (xi − x)2(4)
![Page 12: The linear regression model: Theory and Application](https://reader033.vdocuments.net/reader033/viewer/2022061207/5483fb995906b594158b46e6/html5/thumbnails/12.jpg)
OLS estimators
I OLS β0 and β1 are stochastic estimators (they have adistribution in a sample space of all the possible estimtorsdefine with different samples)
I β1: measure the estimated variation in Y determined by aunitary variation in X (δY /δX )
I The OLS estimators are correct (E (β1) = β1),
and they are BLUE (corrects and with the lowest variance)
![Page 13: The linear regression model: Theory and Application](https://reader033.vdocuments.net/reader033/viewer/2022061207/5483fb995906b594158b46e6/html5/thumbnails/13.jpg)
Linear dependency index (R2)
The R2 index is the most used index to measure the linear fittingof the model.
R2 is confined in the boundary [−1, 1], where, values near to 1 (or-1) means the explanatories are usefull to describe the changes inY .
Let define
SQT = SQR + SQE , or∑ni (yi − y)2 =
∑ni (yi − y)2 +
∑ni (yi − yi )
2
The R2 is defined as SQRSQT or 1− SQE
SQT . Or, equivalent:
R2 =∑n
i (yi−y)2∑ni (yi−y)2
![Page 14: The linear regression model: Theory and Application](https://reader033.vdocuments.net/reader033/viewer/2022061207/5483fb995906b594158b46e6/html5/thumbnails/14.jpg)
Hypotesis testing on β1
The estimated slope parameter β1 is stochastic. It distributes as agaussian:
β1 ∼ N[β1, σ2/SSx ]
We can make use of the hypotesis tests approach to investigate onthe causal relation between Y and X :
H0 : β1 = 0
H1 : β1 6= 0,
where, alternative hypotesis mean causal relation.
The test is:
z = β1−β1
sqrt(σ2/SSx)∼ N(0, 1).
When SSx is unknown, we estimate it as : SSx =∑n
i (xi − y)2,and we use t − test with n − 1 degrees of freedom
![Page 15: The linear regression model: Theory and Application](https://reader033.vdocuments.net/reader033/viewer/2022061207/5483fb995906b594158b46e6/html5/thumbnails/15.jpg)
Forecasting within the regresion model
The question we want to answer is the following: Which is theexpected value of Y (say yn+1), for a certain observation that isnot in the sample?.
Suppose we have, for that observation, the value for the variable X(say xn+1)
We make use of the estimated β to determine:
yn+1 = β0 + β1xn+1
![Page 16: The linear regression model: Theory and Application](https://reader033.vdocuments.net/reader033/viewer/2022061207/5483fb995906b594158b46e6/html5/thumbnails/16.jpg)
Model Checking
Several methods are used to test the robustness of the model,most of them based on the stochastic part of the the model: theestimated residuals.
I Graphical checks: Plot residuals versus fitted values
I qq-plot for the normality
I Shapiro wilk test for normality
I Durbin-Watson test for serial correlation
I Breusch-Pagan test for heteroschedasticity
Moreover, the leverage is used to evaluate th importance of eachobservation in determining the estimated coefficients β.
The Stepwise procedure is used to choice between different modelspecifications.
![Page 17: The linear regression model: Theory and Application](https://reader033.vdocuments.net/reader033/viewer/2022061207/5483fb995906b594158b46e6/html5/thumbnails/17.jpg)
Model Checking using estimated residuals - Linearity
An example of departure from the linearity assumption: we candraw a curve (not a horizontal line) to interpolate the points
Figure: residuals (Y) versus estimated (X) values
![Page 18: The linear regression model: Theory and Application](https://reader033.vdocuments.net/reader033/viewer/2022061207/5483fb995906b594158b46e6/html5/thumbnails/18.jpg)
Model Checking using estimated residuals -Omoscedasticity
An example of departure from the omoschedasticity assumption(the estimated residuals increases as the predicted valuesincrease)
Figure: residuals (Y) versus estimated (X) values
![Page 19: The linear regression model: Theory and Application](https://reader033.vdocuments.net/reader033/viewer/2022061207/5483fb995906b594158b46e6/html5/thumbnails/19.jpg)
Model Checking using estimated residuals - Normality
An example of departure from the normality assumption: theqq-points do not follow the qq-line
Figure: residuals (Y) versus estimated (X) values
![Page 20: The linear regression model: Theory and Application](https://reader033.vdocuments.net/reader033/viewer/2022061207/5483fb995906b594158b46e6/html5/thumbnails/20.jpg)
Model Checking using estimated residuals - Serialcorrelation
An example of departure from the serial incorrelation assumption:the residual at i depend on the value at i − 1
Figure: residuals (Y) versus estimated (X) values
![Page 21: The linear regression model: Theory and Application](https://reader033.vdocuments.net/reader033/viewer/2022061207/5483fb995906b594158b46e6/html5/thumbnails/21.jpg)
Homeworks
1. Using cement data (n = 13), determine the β0 and β1
coefficients manually, using OLS formula at page 11, of themodel y = β0 + β1x1
2. Using cement data, estimate the R2 index of the modely = β0 + β1x1, using formula at page 13.
![Page 22: The linear regression model: Theory and Application](https://reader033.vdocuments.net/reader033/viewer/2022061207/5483fb995906b594158b46e6/html5/thumbnails/22.jpg)
Charts - 1
Figure: Slope coefficient in the linear model
![Page 23: The linear regression model: Theory and Application](https://reader033.vdocuments.net/reader033/viewer/2022061207/5483fb995906b594158b46e6/html5/thumbnails/23.jpg)
Charts - 2
Figure: Fitted (line) versus real (points) values