introduction to correlation and regression analysis
TRANSCRIPT
Introduction to Correlation &
Regression Analysis
Farzad Javidanrad
November 2013
Some Basic Concepts:
o Variable: A letter (symbol) which represents the elements of
a specific set.
o Random Variable: A variable whose values are randomly
appear based on a probability distribution.
o Probability Distribution: A corresponding rule (function)
which corresponds a probability to the values of a random
variable (individually or to a set of them). E.g.:
๐ 0 1
๐(๐ฅ) 0.5 0.5In one trial ๐ป, ๐
In two trials ๐ป๐ป, ๐ป๐, ๐๐ป, ๐๐
Correlation:Is there any relation between:
fast food sale and different seasons?
specific crime and religion?
smoking cigarette and lung cancer?
maths score and overall score in exam?
temperature and earthquake?
cost of advertisement and number of sold items?
To answer each question two sets of corresponding data need to be randomly collected.
Let random variable "๐" represents the first group of
data and random variable "๐" represents the second.
Question: Is this true that students who have a better
overall result are good in maths?
Our aim is to find out whether there is any linear
association between ๐ and ๐. In statistics, technical
term for linear association is โcorrelationโ. So, we are
looking to see if there is any correlation between two
scores.
โLinear associationโ : variables are in relations at
their levels, i.e. ๐ with ๐ not with ๐๐, ๐๐, ๐
๐or even
โ๐.
Imagine we have a random sample of scores in a
school as following:
In our example, the correlation between ๐ and ๐
can be shown in a scatter diagram:
0
10
20
30
40
50
60
70
80
90
100
0 20 40 60 80 100
Y
X
Correlation between maths score and overall score The graph shows a
positive correlation between maths scores and overall scores, i.e. when ๐increases ๐increases too.
Different scatter diagrams show different types of
correlation:
โข Is this enough? Are we happy?Certainly not!! We think we know things better
when they are described by numbers!!!!
Although, scatter diagrams are informative but to find
the degree (strength) of a correlation between two
variables we need a numerical measurement.
Adopted from www.pdesas.org
Following the work of Francis Galton on regression
line, in 1896 Karl Pearson introduced a formula for
measuring correlation between two variables, called
Correlation Coefficient or Pearsonโs Correlation
Coefficient.
For a sample of size ๐, sample correlation coefficient
๐๐๐ can be calculated by:
๐๐๐ = ๐
๐(๐๐ โ ๐)(๐๐ โ ๐)
๐๐(๐๐ โ ๐)๐ . ๐
๐(๐๐ โ ๐)๐=
๐๐๐(๐, ๐)
๐บ๐ . ๐บ๐
Where ๐ and ๐ are the mean values of ๐ and ๐ in the
sample and ๐บ represents the biased version of
โstandard deviationโ*. The covariance between ๐ and ๐(๐๐๐ ๐, ๐ ) shows how much ๐ and ๐ change together.
Alternatively, if there is an opportunity to observe all
available data, the population correlation coefficient
(๐๐๐) can be obtained by:
๐๐๐ =๐ฌ ๐๐ โ ๐๐ . (๐๐ โ ๐๐)
๐ฌ ๐๐ โ ๐๐๐. ๐ฌ(๐๐ โ ๐๐)๐
=๐๐๐(๐, ๐)
๐๐ . ๐๐
Where ๐ฌ, ๐ and ๐ are expected value, mean and
standard deviation of the random variables,
respectively and ๐ต is the size of the population.
Question: Under what conditions can we use this
population correlation coefficient?
If ๐ = ๐๐ + ๐ ๐๐๐ = ๐
Maximum (perfect) positive correlation.
If ๐ = ๐๐ + ๐ ๐๐๐ = โ๐
Maximum (perfect) negative correlation.
If there is no linear association between ๐ and ๐then ๐๐๐ = ๐.
Note 1: If there is no linear association between two
random variables they might have non linear
association or no association at all.
For all ๐ , ๐ โ ๐นAnd ๐ > ๐
For all ๐ , ๐ โ ๐นAnd ๐ < ๐
In our example, the sample correlation coefficient is:๐๐ ๐๐ ๐๐ โ ๐ ๐๐ โ ๐ ๐๐ โ ๐ . (๐๐ โ ๐) (๐ฅ๐โ ๐ฅ )2 (๐ฆ๐โ ๐ฆ )2
70 73 12 13.9 166.8 144 193.21
85 90 27 30.9 834.3 729 954.81
22 31 -36 -28.1 1011.6 1296 789.61
66 50 8 -9.1 -72.8 64 82.81
15 31 -43 -28.1 1208.3 1849 789.61
58 50 0 -9.1 0 0 82.81
69 56 11 -3.1 -34.1 121 9.61
49 55 -9 -4.1 36.9 81 16.81
73 80 15 20.9 313.5 225 436.81
61 49 3 -10.1 -30.3 9 102.01
77 79 19 19.9 378.1 361 396.01
44 58 -14 -1.1 15.4 196 1.21
35 40 -23 -19.1 439.3 529 364.81
88 85 30 25.9 777 900 670.81
69 73 11 13.9 152.9 121 193.21
5196.9 6625 5084.15
๐๐๐ = ๐
๐(๐๐ โ ๐)(๐๐ โ ๐)
๐๐(๐๐ โ ๐)๐ . ๐
๐(๐๐ โ ๐)๐= ๐๐๐๐.๐
๐๐๐๐ร๐๐๐๐.๐๐=๐.๐๐๐
which shows an strong positive correlation between maths score and overall score.
Positive Linear Association
No Linear Association
Negative Linear Association
๐บ๐ > ๐บ๐ ๐บ๐ = ๐บ๐ ๐บ๐ < ๐บ๐
๐๐๐ = ๐
Adapted and modified from www.tice.agrocampus-ouest.fr
๐๐๐ โ ๐
๐ < ๐๐๐ < ๐
๐๐๐ = ๐
โ๐ < ๐๐๐< ๐
๐๐๐ โ โ๐
๐๐๐ = โ๐
Perfect
Weak
No Correlation
Weak
Strong
Perfect
Strong
Some properties of the correlation coefficient:
(Sample or population)
a. It lies between -1 and 1, i.e. โ๐ โค ๐๐๐ โค ๐.
b. It is symmetrical with respect to ๐ and ๐, i.e. ๐๐๐ =
๐๐๐ . This means the direction of calculation is not
important.
c. It is just a pure number and independent from the
unit of measurement of ๐ and ๐.
d. It is independent of the choice of origin and scale
of ๐ and ๐โs measurements, that is;
๐๐๐ = ๐ ๐๐+๐ ๐๐+๐ (๐, ๐ > ๐)
e. ๐ ๐, ๐ = ๐ ๐ . ๐(๐) ๐๐๐ = ๐
Important Note:Many researchers wrongly construct a theory just based on a
simple correlation test.
Correlation does not imply causation.
If there is a high correlation between number of smoked
cigarettes and the number of infected lungโs cells it does not
necessarily mean that smoking causes lung cancer. Causality
test (sometimes called Granger causality test) is different from
correlation test.
In causality test it is important to know about the direction of
causality (e.g. ๐ on ๐ and not vice versa) but in correlation we
are trying to find if two variables moving together (same or
opposite directions).
๐ and ๐ are statistically independent, where ๐(๐, ๐) is the joint Probability
Density Function (PDF)
Determination Coefficient and Correlation Coefficient:
๐๐๐ = ยฑ๐ perfect linear relationship between variables:
i.e. ๐ is the only factor which describes variations of ๐ at the level (linearly); ๐ = ๐ + ๐๐ .
๐๐๐ โ ยฑ๐ ๐ is not the only factor which describes
variations of ๐ but we can still imagine that a line represents this
relationship which passing through most of the points or having a
minimum vertical distance from them, in total. This line is called
the โline of best fitโ or known technically as โregression lineโ.
Adopted from www.ncetm.org.uk/public/files/195322/G3fb.jpg
The graph shows a line of best fit between age of a car and its price. Imagine the line has the equation of ๐ = ๐ + ๐๐
The criterion to choose a line among others is the
goodness of fit which can be calculated through
determination coefficient, ๐๐.
In the previous example, age of a car is only factor
among many other factors that explain the price of a
car. Can you find some other factors?
If ๐ and ๐ represent price and age of cars respectively,
the percentage of the variation of ๐ which is determined
(explained) by the variation of ๐ is called โdetermination
coefficientโ.
Determination coefficient can be understood better by
Venn-Euler diagrams:
y x
y x
y x
y=x
๐๐ = ๐ , none of variations of y can be determined by x (no linear association)
๐๐ โ ๐, small percentage of variation of y can be determined by x (weak linear association)
๐๐ โ ๐, large percentage of variation of y can be determined by x (strong linear association)
๐๐ = ๐, all variation of y can be determined by xand no other factors (complete linear association)
The shaded area shows the percentage of variation of
y which can be determined by x. it is easy to
understand that ๐ โค ๐๐ โค ๐.
Although, determination coefficient (๐๐) is different
conceptually from correlation coefficient (๐๐๐) but one
can be calculated from another; in fact:
๐๐๐ = ยฑ ๐๐
Or, alternatively
๐๐ = ๐๐ ๐
๐ ๐๐ โ ๐ ๐
๐๐ ๐๐ โ ๐ ๐
= ๐๐๐บ๐
๐
๐บ๐๐
Where ๐ is the slope coefficient in the regression
line ๐ = ๐ + ๐๐ .
Note: If ๐ = ๐ + ๐๐ shows the regression line (๐ ๐๐ ๐)
and ๐ = ๐ + ๐ ๐ shows another regression line (๐ ๐๐ ๐)then we have: ๐๐ = ๐. ๐
Summary of Correlation & Determination Coefficients:โข Correlation means a linear association between two random variables which
could be positive or negative or zero.
โข Linear association means that variables are in relations at their levels
(linearly).
โข Correlation coefficient measures the strength of linear association between
two variables. It could be calculated for a sample or for the whole population.
โข The value of correlation coefficient is between -1 and 1, which show the
strongest correlation (negative or positive) but moving towards zero it makes
correlation weaker.
โข Correlation does not imply causation.
โข Determination coefficient shows the percentage of variation of one variable
which can be described by another variable and it is a measure for the
goodness of fit for lines passing through plotted points.
โข The value of determination coefficient is between 0 and 1 and can be
obtained from correlation coefficient by squaring it.
โข Knowing two random variables are just linearly associated is
not much satisfactory. There are sometimes a strong idea
that the variation of one variable can solidly explain the
variation of another.
โข To test this idea (hypothesis) we need another analytical
approach, which is called โregression analysisโ.
โข In regression analysis we try to study or predict the mean
(average) value of a dependent variable ๐ based on the
knowledge we have about independent (explanatory)
variable(s) ๐ฟ๐, ๐ฟ๐,โฆ, ๐ฟ๐. This is familiar for those who know
the meaning of conditional probabilities; as we are going to
make a linear model such as, which is a deterministic part of
the model in regression analysis:
๐ธ(๐ ๐1, ๐2,โฆ, ๐๐) = ๐ฝ0 + ๐ฝ1๐1 + ๐ฝ2๐2 + โฏ + ๐ฝ๐๐๐
โข The deterministic part of the regression model does reflect the
structure of the relationship between ๐ and ๐ฟโฒ๐ in a
mathematical world but we live in a stochastic world.
โข Godโs knowledge (if the term is applicable) is deterministic but
our perception about everything in this world is always
stochastic and our model should be built in this way.
โข To understand the concept of stochastic model letโs have an
example:
If we make a model between monthly consumption expenditure
๐ช and monthly income ๐ฐ, the model cannot be deterministic
(mathematical) such that for every value of ๐ฐ there is one and
only one value of ๐ช (which is the concept of functional
relationship in maths). Why?
Although, the income is the main variable determining the amount of
consumption expenditure but many other factors such as the mood of
people, their wealth, interest rate and etc. are overlooked in a simple
mathematical model such as ๐ช = ๐(๐ฐ) but their influences can change the
value of ๐ช even at the same level of ๐ฐ. If we believe that the average impact
of all their omitted variables is random (sometimes positive and sometimes
negative). So, in order to make a realistic model we need to add a stochastic
(random) term ๐ to our mathematical model: ๐ช = ๐ ๐ฐ + ๐
ยฃ1000
ยฃ1400
โฎ
โฎ
ยฃ800ยฃ1000ยฃ750
ยฃ900ยฃ1200ยฃ1150
I C
The change in the consumption
expenditure comes from the change of
income (๐ผ) or change of some
random elements (๐ข), so, we can write
๐ช = ๐ ๐ฐ + ๐
โข The general stochastic model for our purpose would be as
following, which is called โLinear Regression Model**โ:
๐๐ = ๐ฌ(๐๐ ๐ฟ๐๐, โฆ , ๐ฟ๐๐) + ๐๐
Which can be written as:
๐๐ = ๐ท๐ + ๐ท๐๐ฟ๐๐ + ๐ท๐๐ฟ๐๐ + โฏ + ๐ท๐๐ฟ๐๐ + ๐๐
Where ๐ (๐ = 1,2, โฆ , ๐) shows time period (days, weeks, months,
years and etc.) and ๐๐ is an error (stochastic) term and also a
representative of all other influential variables which are not
considered in the model and ignored.
โข The deterministic part of the model
๐ฌ(๐๐ ๐ฟ๐๐, โฆ , ๐ฟ๐๐) =๐ท๐ + ๐ท๐๐ฟ๐๐ + ๐ท๐๐ฟ๐๐ + โฏ + ๐ท๐๐ฟ๐๐
is called Population Regression Function (PRF).
โข The general form of the Linear Regression Model with ๐explanatory variables and ๐ observations can be shown in
the matrix form as:
๐๐ร1 = ๐ฟ๐ร๐๐ท๐ร1 + ๐๐ร1
Or simply:
๐ = ๐ฟ๐ท + ๐Where
๐ =
๐1
๐2
โฎ๐๐
, ๐ฟ =
1 ๐11 ๐21
1โฎ
๐12
โฎ๐22
โฎ1 ๐1๐ ๐2๐
โฆ ๐๐1โฆโฑ
๐๐2
โฎโฆ ๐๐๐
, ๐ท =
๐ฝ0
๐ฝ1
โฎ๐ฝ๐
and ๐ =
๐ข1๐ข2
โฎ๐ข๐
๐ is also called regressand and ๐ฟ is a vector of regressors.
โข ๐ท๐ is the intercept but ๐ท๐โฒ๐ are slope coefficients which are also
called regression parameters. The value of each parameter
shows the magnitude of one unit change in the associated
regressor ๐ฟ๐ on the mean value of the regressand ๐๐. The idea
is to estimate the unknown value of the population
regression parameters based on estimators which use
sample data.
โข The sample counterpart of the regression line can be written in
the form of:
๐๐ = ๐๐ + ๐๐
or
๐๐ = ๐๐ + ๐๐๐ฟ๐๐ + ๐๐๐ฟ๐๐ + โฏ + ๐๐๐ฟ๐๐ + ๐๐
Where ๐๐ = ๐๐ + ๐๐๐ฟ๐๐ + ๐๐๐ฟ๐๐ + โฏ + ๐๐๐ฟ๐๐ is the deterministic
part of the sample model and is called โSample Regression
Function (SRF) โand ๐๐โฒ๐ are estimators of unknown parameters
๐ท๐โฒ๐ and ๐๐ = ๐๐ is a residual.
The following graph shows the important elements of PRF and
SRF:
๐๐ โ ๐ฌ(๐ ๐ฟ๐) = ๐๐
๐๐ โ ๐๐ = ๐๐ = ๐๐
observation
Estimation of ๐๐ based on SRF
Estimation of ๐๐ based on PRF
Adopted and altered fromhttp://marketingclassic.blogspot.co.uk/2011_12_01_archive.html
In PRF
In SRF
The PRF is a hypothetical line which we have no idea about that but try to estimate its parameters based on the data in sample
๐บ๐น๐ญ: ๐๐ = ๐๐ + ๐๐๐ฟ๐
๐ท๐น๐ญ: ๐ฌ(๐ ๐ฟ๐) = ๐ท๐ + ๐ท๐๐ฟ๐
โข Now the question is how to calculate ๐๐โฒ๐ based on the
sample observations and how to ensure that they are good
and unbiased estimators of ๐ท๐โฒ๐ in the population?
โข There are two main methods of calculating ๐๐โฒ๐ and constructing
SRF, called the โmethod of Ordinary Least Square (OLS)โ and
the โmethod of Maximum Likelihood (ML)โ. Here, we focus on
OLS method as it is used most comprehensively. Here, for
simplicity, we start with two-variable PRF (๐๐ = ๐ท๐ + ๐ท๐๐ฟ๐) and
its SRF counterpart (๐๐ = ๐๐ + ๐๐๐ฟ๐).
โข According to OLS method we try to minimise some of the
squared residuals in a hypothetical sample; i.e.
๐๐๐
= ๐๐๐ = ๐๐ โ ๐๐
๐
= ๐๐ โ ๐๐ โ ๐๐๐ฟ๐๐
โข It is obvious from previous equation that the sum of squared
residuals is a function of ๐๐ and ๐๐, i.e.
๐๐๐ = ๐(๐๐, ๐๐)
because if these two parameters (intercept and slope) change,
๐๐๐ will change (see the graph on the slide 25).
โข Differentiating A partially with respect to ๐๐ and ๐๐ and
following the first and necessary conditions for optimisation in
calculus we have:
๐ ๐๐๐
๐๐๐= โ๐ ๐๐ โ ๐๐ โ ๐๐๐ฟ๐ = โ๐ ๐๐ = ๐
๐ ๐๐๐
๐๐๐= โ๐ ๐ฟ๐ ๐๐ โ ๐๐ โ ๐๐๐ฟ๐ = โ๐ ๐ฟ๐๐๐ = ๐
A
B
After simplifications we reach to two equations with two
unknowns ๐๐ and ๐๐:
๐๐ = ๐๐๐ + ๐๐ ๐ฟ๐
๐๐๐ฟ๐ = ๐๐ ๐ฟ๐ + ๐๐ ๐ฟ๐๐
Where ๐ is the sample size. So;
๐๐ = ๐ฟ๐ โ ๐ฟ ๐๐ โ ๐
๐ฟ๐ โ ๐ฟ ๐=
๐๐๐๐
๐๐๐
=๐๐๐(๐, ๐)
๐บ๐๐
Where ๐บ๐ is the biased version of sample standard deviation,
i.e. we have ๐ instead of (๐ โ ๐) in denominator.
๐บ๐ = ๐ฟ๐ โ ๐ฟ ๐
๐
And
๐0 = ๐ โ ๐1 ๐
โข The ๐๐ and ๐๐ obtained from OLS method are the point
estimators of ๐ท๐ and ๐ท๐in the population but in order to test
some hypothesis about the population parameters we need to
have knowledge about the distributions of their estimators. For
that reason we need to make some assumptions about the
explanatory variables and the error term in PRF. (see the
equations in B to find the reason).
The Assumptions Underlying the OLS Method:
1. The regression model is linear in terms of its parameters (coefficients).*
2. The values of the explanatory variable(s) are fixed in repeated sampling.
This means that the nature of explanatory variables (๐ฟโฒ๐) is non-stochastic.
The only stochastic variables are error term (๐๐) and regressand (๐๐).
3. The disturbance (error) terms are normally distributed with zero mean and
equal variance; given the value of ๐ฟโฒ๐. That is: ๐๐~๐ต(๐, ๐๐)
4. There is no autocorrelation between error terms, i.e.
๐๐๐ ๐๐, ๐๐ = ๐
This means they are completely random and there is no association between
them or any pattern in their appearance.
5. There is no correlation between error terms and explanatory variables, i.e.
๐๐๐ ๐๐, ๐ฟ๐ = ๐
6. The number of observations (sample size) should be bigger than the
number of parameters in the model.
7. The model should be logically and correctly specified in terms of functional
form or even the type and the nature of variables enter into the model.
These assumptions are the assumptions of the Classical Linear
Regression Models (CLRM), which sometimes they are called
Gaussian assumptions on linear regression models.
โข Under these assumptions and also the central limit theorem
the OLS estimators in sampling distribution (repeated sampling)
,when ๐ โ โ, have a normal distribution:
๐๐~๐ต(๐ท๐, ๐ฟ๐
๐
๐ ๐๐๐
. ๐๐)
๐๐~๐ต(๐ท๐,๐๐
๐๐๐)
where ๐๐ is the variance of the error term (๐๐๐ ๐๐ = ๐๐) and it
can be estimated itself through ๐ estimator, where:
๐ = ๐๐
๐
๐ โ ๐๐๐
๐ = ๐๐
๐
๐ โ ๐๐คโ๐๐ ๐กโ๐๐๐ ๐๐ ๐ ๐๐๐๐๐๐๐ก๐๐ ๐๐ ๐กโ๐ ๐๐๐๐๐.
โข Based on the assumptions of the classical linear regression
model (CLRM), Gauss-Markov Theorem asserts that the least
square estimators, among unbiased estimators, have the
minimum variance. So they are the Best, Linear, Unbiased
Estimators (BLUE).
Interval Estimation For Population Parameters:
โข In order to construct a confidence interval for unknown
๐ทโฒ๐ (PRFโs parameters) we can either follow Z distribution (if
we have a prior knowledge about ๐) or t-distribution (if we use
๐ instead).
โข The confidence intervals for the slope parameter at any level of
significance ๐ถ would be*:
๐ท ๐๐ โ ๐ ๐ถ๐. ๐๐๐
โค ๐ท๐ โค ๐๐ + ๐ ๐ถ๐. ๐๐๐
= ๐ โ ๐ถ
Or
๐ท ๐๐ โ ๐ ๐ถ๐,(๐โ๐). ๐๐๐
โค ๐ท๐ โค ๐๐ + ๐ ๐ถ๐,(๐โ๐). ๐๐๐
= ๐ โ ๐ถ
Hypothesis Testing For Parameters:
โข The critical values (Z or t) in the confidence intervals, can be
used to find the rejection area(s) and test any hypothesis on
parameters.
โข For example, to test ๐ฏ๐: ๐ท๐ = ๐ against the alternative ๐ฏ๐: ๐ท๐ โ ๐, after finding the critical values t (which means we do not have prior knowledge of ๐ and use ๐ instead) at any
significance level ๐ถ, we will have two critical regions and if the
value of the test statistic
๐ =๐๐โ๐ท๐
๐
๐๐๐
be in the critical region ๐ฏ๐: ๐ท๐ = ๐ must be rejected.
โข In case we have more than one slope parameter the degree of
freedom for t-distribution will be the sample size ๐ minus the
number of estimated parameters including the intercept
parameters, i.e. for ๐ parameters ๐ ๐ = ๐ โ ๐ .
Determination Coefficient ๐๐ and Goodness of Fit:
โข In early slides we talked about determination coefficient and
its relationship with correlation coefficient. The coefficient of
determination ๐๐ come to our attention when there is no issue
about estimation of regression parameters.
โข It is a measure which shows how well the SRF fits the data.
โข to understand this measure properly letโs have a look at it
from different angle.
We know that
๐๐ = ๐๐ + ๐๐
And in the deviation form after
subtracting ๐ from both sides
๐๐ โ ๐ = ๐๐ โ ๐ + ๐๐
We know that ๐๐ = ๐๐ โ ๐๐
๐๐ Ad
op
ted
from
Basic Eco
no
me
trics Go
jaratiP7
6
๐
๐๐ โ ๐
So;๐๐ โ ๐ = ( ๐๐ โ ๐) + (๐๐ โ ๐๐)
Or in the deviation form๐๐ = ๐๐ + ๐๐
By squaring both sides and adding all over the sample we have:
๐๐๐ = ๐๐
๐ + ๐ ๐๐ ๐๐ + ๐๐๐
= ๐๐๐ + ๐๐
๐
Where ๐๐ ๐๐ = ๐ according to the OLSโs assumptions 3 and 5.
And if we change it to the non-deviated form:
๐๐ โ ๐ 2 = ๐๐ โ ๐2
+ ๐๐ โ ๐๐2
Total variation of the observed Y values around their mean =Total Sum of
Squares= TSS
Total explained variation of the estimated Y values around their
mean = Explained Sum of Squares (by explanatory
variables)= ESS
Total unexplained variation of the observed Y values around the regression line= Residual Sum of Squares (Explained by
error terms)= RSS
Dividing both sides by Total Sum of Squares (TSS) we have:
1 =๐ธ๐๐
๐๐๐+
๐ ๐๐
๐๐๐=
๐๐ โ ๐ 2
๐๐ โ ๐ 2+
๐๐ โ ๐๐2
๐๐ โ ๐ 2
Where ๐๐โ ๐ ๐
๐๐โ ๐ ๐=
๐ฌ๐บ๐บ
๐ป๐บ๐บis the percentage of the variation of the actual
(observed) ๐๐ which is explained by the explanatory variables (by
regression line).
โข A good reader knows that this is not a new concept; the
determination coefficient ๐๐ was described already as a
measure of the goodness of fit between different alternative
sample regression functions (SRFs).
๐ = ๐๐ +๐น๐บ๐บ
๐ป๐บ๐บโ ๐๐ = ๐ โ
๐น๐บ๐บ
๐ป๐บ๐บ
= ๐ โ ๐๐
๐
๐๐โ ๐ ๐
โข A good model must have a reasonable high ๐๐ but this does not
mean any model with a high ๐๐ is a good model. Extremely high
level of ๐๐ could be as a result of having a spurious regression
line due to the variety of reasons such as non-stationarity of
data, cointegration problem and etc.
โข In a regression model with two parameters, ๐๐ can be directly
calculated:
๐๐ = ๐๐โ ๐
๐
๐๐โ ๐ ๐ = ๐๐+๐๐๐ฟ๐โ๐๐โ๐๐๐ฟ
๐
๐๐โ ๐ ๐
=๐๐
๐ ๐ฟ๐โ๐ฟ๐
๐๐โ ๐ ๐ =๐๐
๐ ๐๐๐
๐๐๐ = ๐๐
๐ ๐บ๐ฟ๐
๐บ๐๐
Where ๐บ๐ฟ๐ and ๐บ๐
๐ are the standard deviations of ๐ฟ and ๐respectively.
Multiple Regression Analysis:
โข If there are more than two explanatory variables in the
regression line we need additional assumptions about the
independency of the explanatory variables and also having no
exact linear relationship between them.
โข The population and the sample regression models for three
variables model can be described as following:
In Population: ๐๐ = ๐ท๐ + ๐ท๐๐ฟ๐๐ + ๐ท๐๐ฟ๐๐ + ๐๐
In Sample: ๐๐ = ๐๐ + ๐๐๐ฟ๐๐ + ๐๐๐ฟ๐๐ + ๐๐
โข The OLS estimators can be obtained by minimising ๐๐๐. So,
the values of the SRF parameters in the deviation form are as
following:
๐๐ =( ๐๐๐๐๐)( ๐๐๐
๐) โ ( ๐๐๐๐๐)( ๐๐๐๐๐๐)
( ๐๐๐๐)( ๐๐๐
๐) โ ( ๐๐๐๐๐๐)๐
๐๐ =( ๐๐๐๐๐)( ๐๐๐
๐) โ ( ๐๐๐๐๐)( ๐๐๐๐๐๐)
( ๐๐๐๐)( ๐๐๐
๐) โ ( ๐๐๐๐๐๐)๐
And the intercept parameter will be calculated in the non-deviated
form as:
๐๐ = ๐ โ ๐๐๐ฟ๐ โ ๐๐๐ฟ๐
โข Under the classical assumptions and also the central limit
theorem the OLS estimators in sampling distribution (repeated
sampling),when ๐ โ โ, have a normal distribution:
๐๐~๐ต(๐ท๐,๐๐
๐. ๐๐๐๐
( ๐๐๐๐)( ๐๐๐
๐) โ ( ๐๐๐๐๐๐)๐)
๐๐~๐ต(๐ท๐,๐๐
๐. ๐๐๐๐
( ๐๐๐๐)( ๐๐๐
๐) โ ( ๐๐๐๐๐๐)๐)
โข The distribution of the intercept parameter ๐๐ is not of primary
concern as in many cases it has no practical importance.
โข If the variance of the disturbance (error) term (๐๐๐) is not known
the residual variance (sample variance) can be used ( ๐๐๐),
which is an unbiased estimator of the earlier:
๐๐๐ =
๐๐๐
๐ โ ๐
Where ๐ is the number of parameters in the model (including the
intercept ๐๐). Therefore, in a regression model with two slope
parameters and one intercept parameter the residual variance can
be calculated by:
๐๐๐ =
๐๐๐
๐ โ ๐
So, for a model with two slope parameters, the unbiased
estimates of the variance of these parameters are:
๐บ๐๐
๐ = ๐๐
๐
๐ โ ๐.
๐๐๐๐
( ๐๐๐๐)( ๐๐๐
๐) โ ( ๐๐๐๐๐๐)๐
= ๐๐
๐
๐๐๐๐ (๐ โ ๐๐
๐๐)
Where ๐๐๐๐ =
๐๐๐๐๐๐๐
๐๐๐๐ ๐๐๐
๐ .
and
๐บ๐๐
๐ = ๐๐
๐
๐ โ ๐.
๐๐๐๐
( ๐๐๐๐)( ๐๐๐
๐) โ ( ๐๐๐๐๐๐)๐
= ๐๐
๐
๐๐๐๐ (๐ โ ๐๐
๐๐)
๐๐๐
The Coefficient of Multiple Determination (๐น๐and ๐น๐ ):
The same concept of the coefficient of determination used for a
bivariate model can be extended for a multivariate model.
โข If ๐น๐ is denoted as the coefficient of multiple determination it
shows the proportion (percentage) of the total variation of ๐explained by the explanatory variables and it is calculated by:
๐ 2 =๐ธ๐๐
๐๐๐=
๐ฆ๐2
๐ฆ๐2 =
๐1 ๐ฆ๐๐ฅ1๐+๐2 ๐ฆ๐๐ฅ2๐
๐ฆ๐2
And we know that:
0 โค ๐ 2 โค 1
Note that ๐ 2 can also be calculated through RSS, i.e.
๐ 2 = 1 โ๐ ๐๐
๐๐๐= 1 โ
๐๐2
๐ฆ๐2
C
โข ๐น๐ is likely to increase by including an additional explanatory
variable (see ). Therefore, in case we have two alternative
models with the same dependent variable ๐ but different
number of explanatory variables we should not be misled by the
high ๐น๐of the model with more variables.
โข To solve this problem we need to bring the degrees of freedom
into our consideration as a reduction factor against adding
additional explanatory variables. So, the adjusted ๐น๐ which can
be shown by ๐น๐ is considered as an alternative coefficient of
determination and it is calculated as:
๐ 2 = 1 โ
๐๐2
๐ โ ๐ ๐ฆ๐
2
๐ โ 1
= 1 โ๐ โ 1
๐ โ ๐. ๐๐
2
๐ฆ๐2
= 1 โ๐โ1
๐โ๐(1 โ ๐ 2)
C
Partial Correlation Coefficients:
โข For a three-variable regression model such as
๐๐ = ๐๐ + ๐๐๐ฟ๐๐ + ๐๐๐ฟ๐๐ + ๐๐
We can talk about three linear association (correlation) between
๐ and ๐ฟ๐ ๐๐๐๐, between ๐ and ๐ฟ๐ (๐๐๐๐
) and finally between
๐ฟ๐ and ๐ฟ๐ (๐๐๐๐๐). These correlations are called simple (gross)
correlation coefficients but they do not reflect the true linear
association between two variables as the influence of the third
variable on the other two is not removed.
โข The net linear association between two variables can be
obtained through the partial correlation coefficient, where the
influence of the third variable is removed (the variable is hold
constant). Symbolically, ๐๐๐๐. ๐๐represents the partial
correlation coefficient between ๐ and ๐ฟ๐ holding ๐ฟ๐ constant.
โข Two partial correlation coefficients in our model can be
calculated as following:
๐๐๐๐. ๐๐=
๐๐๐๐โ ๐๐๐๐
๐๐๐๐๐
๐ โ ๐๐๐๐๐๐
. ๐ โ ๐๐๐๐๐
๐๐๐๐. ๐๐=
๐๐๐๐โ ๐๐๐๐
๐๐๐๐๐
๐ โ ๐๐๐๐๐๐
. ๐ โ ๐๐๐๐๐
โข The correlation coefficient ๐๐๐๐๐.๐ has no practical importance.
Specifically, when the direction of causality is from ๐ฟโฒ๐ to ๐ we
can simply use the simple correlation coefficient in this case:
๐ = ๐๐๐๐
๐๐๐ . ๐๐
๐
โข They can be used to find out which explanatory variable has
more linear association with the dependent variable.
Hypothesis Testing in Multiple Regression Models:
In a multiple regression model hypotheses are formed to test
different aspects of this type of regression models:
i. Testing hypothesis about an individual parameter of the
model. For example;
๐ฏ๐: ๐ท๐ = ๐ against ๐ฏ๐: ๐ท๐ โ ๐
If ๐ is unknown and is replaced by ๐ the test statistic
๐ =๐๐โ๐ท๐
๐๐(๐๐)=
๐๐
๐๐(๐๐)
follows the t-distribution with ๐ โ ๐ df (for a regression model with
three parameters, including intercept, ๐๐ = ๐ โ ๐)
ii. Testing hypothesis about the equality of two parameters
in the model. For example,
๐ฏ๐: ๐ท๐ = ๐ท๐ against ๐ฏ๐: ๐ท๐ โ ๐ท๐
Again, if ๐ is unknown and is replaced by ๐ the test statistic
๐ =๐๐ โ ๐๐ โ ๐ท๐ โ ๐ท๐
๐๐(๐๐ โ ๐๐)
=๐๐ โ ๐๐
๐๐๐ ๐๐ + ๐๐๐ ๐๐ โ ๐๐๐๐(๐๐, ๐๐)
follows the t-distribution with ๐ โ ๐ df.
โข If the value of test statistic ๐ > ๐๐ถ
๐,(๐โ๐) we must reject ๐ฏ๐,
otherwise there is not much evidence to reject that.
iii. Testing hypothesis about the overall significance of the
estimated model by checking if all the slope parameters
are simultaneously zero. For example, to test
๐ฏ๐: ๐ท๐ = ๐ (โ ๐) against ๐ฏ๐: โ๐ท๐ โ ๐
the analysis of variance (ANOVA) table can be used to find if the
mean sum of squares (MSS), due to the regression (or
explanatory variables) are very far from the MSS due to the
residuals. If this is true, it means the variation of explanatory
variables contribute more towards the variation of the dependent
variable than the variation of residuals, so, the ratio
๐ด๐บ๐บ ๐๐ข๐ ๐ก๐ ๐๐๐๐๐๐ ๐ ๐๐๐ (๐๐ฅ๐๐๐๐๐๐ก๐๐๐ฆ ๐ฃ๐๐๐๐๐๐๐๐ )
๐ด๐บ๐บ ๐๐ข๐ ๐ก๐ ๐๐๐ ๐๐๐ข๐๐๐ (๐๐๐๐๐๐ ๐๐๐๐๐๐๐ก๐ )
should be much higher than one.
โข The ANOVA table for the three-variable regression model can
be formed as following:
โข If we believe that the regression model is meaningless so we
cannot reject the null hypothesis that all slope coefficients are
simultaneously equal to zero, otherwise the test statistic
๐น =๐ธ๐๐/๐๐
๐ ๐๐/๐๐=
๐๐ ๐๐๐๐๐ + ๐๐
๐๐๐๐๐
๐ ๐๐
๐
๐ โ ๐
Which follows the F-distribution with 2 and ๐ โ ๐ df must be much
bigger than 1.
Source of variation Sum of Squares (SS) df Mean Sum of Squares (MSS)
Due to Explanatory Variables
๐๐ ๐๐๐๐๐ + ๐๐ ๐๐๐๐๐ 2
๐๐ ๐๐๐๐๐ + ๐๐ ๐๐๐๐๐
๐
Due to Residuals ๐๐
๐๐ โ ๐
๐๐ = ๐๐
๐
๐ โ ๐
Total ๐๐
๐๐ โ ๐
โข In general, to test the overall significance of the sample
regression for a multi-variable model (e.g with ๐ slope
parameters) the null and alternative hypotheses and the test
statistic are as following:
๐ฏ๐: ๐ท๐ = ๐ท๐ = โฏ = ๐ท๐ = ๐๐ฏ๐: ๐๐ ๐๐๐๐๐ ๐๐๐๐๐ ๐๐ ๐๐๐ ๐ท๐ โ ๐
๐ญ = ๐ฌ๐บ๐บ
๐โ๐
๐น๐บ๐บ๐โ๐
โข If ๐ญ > ๐ญ๐ถ, ๐โ๐, ๐โ๐ we reject ๐ฏ๐ at the significance level of ๐ถ,
otherwise there is no enough evidence to reject it.
โข It is sometimes easier to use the determination coefficient ๐น๐
to run the above test, because
๐น๐ =๐ฌ๐บ๐บ
๐ป๐บ๐บโ ๐ฌ๐บ๐บ = ๐น๐. ๐ป๐บ๐บ
and also
๐น๐บ๐บ = ๐ โ ๐น๐ . ๐ป๐บ๐บ
โข The ANOVA table can also be written as:
โข So, the test statistic F can be written as:
๐ญ = ๐น๐ ๐๐
๐
(๐ โ ๐)
(๐ โ ๐น๐) ๐๐๐
(๐ โ ๐)
=๐ โ ๐
๐ โ ๐.
๐น๐
๐ โ ๐น๐
Source of variation Sum of Squares (SS) df Mean Sum of Squares (MSS)
Due to Explanatory Variables
๐น๐ ๐๐๐
๐ โ ๐๐น๐ ๐๐
๐
๐ โ ๐
Due to Residuals(๐ โ ๐น๐) ๐๐
๐ ๐ โ ๐ ๐๐ =
(๐ โ ๐น๐) ๐๐๐
๐ โ ๐
Total ๐๐
๐๐ โ ๐
iv. Testing hypothesis about parameters when they satisfy
certain restrictions.*
e.g.๐ฏ๐: ๐ท๐ + ๐ท๐ = ๐ against ๐ฏ๐: ๐ท๐ + ๐ท๐ โ ๐
v. Testing hypothesis about the stability of the estimated
regression model in a specific time period or in two cross-
sectional unit.**
vi. Testing hypothesis about different functional forms of
regression models.***