chapter 7 multicollinearity. what is in this chapter? how do we detect this problem? what are the...

Chapter 7

Multicollinearity

What is in this Chapter?

• How do we detect this problem?

• What are the consequences?

• What are the solutions?

• An example by Gauss

Dependent Variable: Y

Method: Least Squares

Date: 12/25/09 Time: 15:22

Sample: 1 30

Included observations: 30

Variable Coefficient Std. Error t-Statistic Prob.

C 0.221179 0.234646 0.942607 0.3542

X1 1.559895 2.840030 0.549253 0.5873

X2 1.444632 2.912596 0.495995 0.6239

R-squared 0.993479 Mean dependent var 7.581333

Adjusted R-squared 0.992996 S.D. dependent var 9.949418

S.E. of regression 0.832655 Akaike info criterion 2.566244

Sum squared resid 18.71947 Schwarz criterion 2.706364

Log likelihood -35.49366 F-statistic 2056.801

Durbin-Watson stat 2.456606 Prob(F-statistic) 0.000000


• In Chapter 4 we stated that one of the assumptions in the basic regression model is that the explanatory variables are not exactly linearly related. If they are, then not all parameters are estimable

• What we are concerned with in this chapter is the case where the individual parameters are not estimable with sufficient precision (because of high standard errors)

• This often occurs if the explanatory variables are highly intercorrelated (although this condition is not necessary).

Dependent Variable: X1


Date: 12/25/09 Time: 15:24

Sample: 1 30



C -0.047905 0.012721 -3.765648 0.0008

X2 1.025415 0.003158 324.6592 0.0000



S.E. of regression 0.055407 Akaike info criterion -2.883889

Sum squared resid 0.085958 Schwarz criterion -2.790476

Log likelihood 45.25834 F-statistic 105403.6



• This chapter is very important, because multicollinearity is one of the most misunderstood problems in multiple regression

• There have been several measures for multicollinearity suggested in the literature (variance-inflation factors VIF, condition numbers CN, etc.)

• This chapter argues that all these are useless and misleading

• They all depend on the correlation structure of the explanatory variables only.


• It is argued here that this is only one of several factors determining high standard errors

• High intercorrelations among the explanatory variables are neither necessary nor sufficient to cause the multicollinearity problem

• The best indicators of the problem are the t-ratios of the individual coefficients.


• This chapter also discusses the solutions offered for the multicollinearity problem:– principal component regression– dropping of variables

• However, they are ad hoc and do not help

• The only solutions are to get more data or to seek prior information

7.1 Introduction

• Very often the data we use in multiple regression analysis cannot give decisive (significant) answers to the questions we pose.

• This is because the standard errors are very high or the t-ratios are very low.

• This sort of situation occurs when the explanatory variables display little variation and/or high intercorrelations.

7.1 Introduction

• The situation where the explanatory variables are highly intercorrelated is referred to as multicollinearity

• When the explanatory variables are highly intercorrelated, it becomes difficult to disentangle the separate effects of each of the explanatory variables on the explained variable

7.2 Some Illustrative Examples

• Thus only(β1 +2β2) would be estimable.

• We cannot get estimates of β1 and β2 separately.

• In this case we say that there is “perfect multicollinearity,” because x1 and x2 are perfectly correlated (with =1).

• In actual practice we encounter cases where r2 is not exactly 1 but close to 1.

212r


• As an illustration, consider the case where

so that the normal equations are

• The solution is .• Suppose that we drop an observation and the

new values are

1ˆand1ˆ21


• Now when we solve the equations

• We get


• Thus very small changes in the variances and covariances produce drastic changes in the estimates of regression parameters.

• It is easy to see that the correlation coefficient between the two explanatory variables is given by

which is very high.


• In practice, addition or deletion of observations would produce changes in the variances and covariances

• Thus one of the consequences of high correlation between x1 and x2 is that the parameter estimates would be very sensitive to the addition or deletion of observations

• This aspect of multicollinearity can be checked in practice by deleting or adding some observations and examining the sensitivity of the estimates to such perturbations

Dependent Variable: Y


Date: 12/25/09 Time: 15:22

Sample: 1 30



C 0.221179 0.234646 0.942607 0.3542

X1 1.559895 2.840030 0.549253 0.5873

X2 1.444632 2.912596 0.495995 0.6239



S.E. of regression 0.832655 Akaike info criterion 2.566244

Sum squared resid 18.71947 Schwarz criterion 2.706364

Log likelihood -35.49366 F-statistic 2056.801



Thus the variance of will be high if:

1. σ2 is high.

2. S11 is low.

3. is high.

1̂

212r


• Even if is high, if σ2 is low and S11 high, we will not have the problem of high standard errors.

• On the other hand, even if is low, the standard errors can be high if σ2 is high and S11 is low (i.e., there is not sufficient variation in x1).

• What this suggests is that high value of do not tell us anything whether we have a multicollinearity problem or not.

• When we have more than two explanatory variables, the simple correlations among them become all the more meaningless.

212r

212r

212r


• As an illustration, consider the following example with 20 observations on x1, x2, and x3:

x1 =(1, 1, 1, 1, 1, 0, 0, 0, 0, 0, and 10 zeros)

x2 =(0, 0, 0, 0, 0, 1, 1, 1, 1, 1, and 10 zeros)

x3 =(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, and 10 zeros)


• Obviously, x3=x1+x2 and we have perfect multicollinearity.

• But we can see that ,and thus the simple correlations are not high.

• In the case of more than two explanatory variables, what we have to consider are multiple correlations of each of the explanatory variables with the other explanatory variables.

59.03/1and3/1 231312 rrr


• Note that the standard error formulas corresponding to equations (7.1) and (7.2) are

where σ2 and Sii are defined as before in the case of two explanatory variables and represents the squared multiple correlation coefficient between xi and the other explanatory variables.

)4.7()1(

)ˆ(2

2

iiii RS

V

2iR

7.3 Some Measures of Multicollinearity

• It is important to be familiar with two measures that are often suggested in the discussion of multicollinearity : the variance inflation factor (VIF) and the condition number (CN).

• The VIF is defined as

where is the squared multiple correlation coefficient between xi and the other explanatory variables.

21

1)ˆ(

ii R

VIF

2iR


• A measure that considers the correlations of the explanatory variable with the explained variable is Theil’s measure, which is defined as

where R2 = squared multiple correlation from a regression of y on x1, x2,…..,xk

= squared multiple correlation from a regression of y on x1, x2,…..,xk with xi omitted

k

iiRRRm

1

222 )(

2iR


• The quantity is termed the “incremental contribution” to the squared multiple correlation by Theil.

• If x1, x2,…..,xk are mutually uncorrelated, then m wi

ll be 0 because the incremental contributions all add up to .

• In other cases m can be negative as well as highly positive.

2R

)( 21

2 RR

7.4 Problems with Measuring Multicollinearity

• Let us define

C= real consumption per capita

Y= real per capita current income

Yp= real per capita permanent income

YT= real per capita transitory income

Y=YT+Yp and Yp and YT are uncorrelated


• Suppose that we formulate the consumption function as

• All these equation are equivalent. However, the correlations between the explanatory variables will be different depending in which of the three equations is considered.

• In equation (7.5), since Y and Yp are often highly correlat

ed, we would say that there is high multicollinearity.

)7.7()(

)6.7()(

)5.7(

uYYC

uYYC

uYYC

T

pT

p


• In equation (7.6), since YT and Yp are uncorrelat

ed, we would say that there is no multicollinearity.

• However, the two equations are essentially the same.

• What we should be talking about is the precision with which α and β or (α+β ) are estimable.


Consider, for instance, the following data：


• For these data the estimation of equation (7.5) gives (figures in parentheses are standard errors)

• One reason for the imprecision in the estimates is that Y and Yp are highly correlated (the correlat

ion coefficient is 0.95).

1.0ˆ59.030.0 2

)33.0()32.0( upYYC


• For equation (7.6) the correlation between the explanatory variables is zero and for equation (7.7) it is 0.32.

• The least squares estimates of α and β are no more precise in equation (7.6) or (7.7).

• Let us consider the estimation of equation (7.6). We get

pT YYC)11.0()32.0(

89.030.0


• The estimate at (α+β) is thus 0.89 and the standard error is 0.11.

• Thus (α+β) is indeed more precisely estimated than either αorβ.

• As for α, it is not precisely estimated even though the explanatory variables in this equation are uncorrelated.

• The reason is that the variance of YT is very low

[ see formula (7.1)]


• In Table 7.1 we present data in C, Y, and L for the period from the first quarter of 1952 to the second quarter of 1961.

• C is consumption expenditures, Y is disposable income, and L is liquid assets at the end of the previous quarter.

• All figures are in billions of 1954 dollars.• Using the 38 observations we get the following

regression equations .


• Equation(7.10) shows that L and Y are very highly correlated.• In fact, substituting the value of L in terms Y from (7.10) into e

quation (7.9) and simplifying, we get equation (7.8) correct to four decimal place!

• However, looking at the t-ratios in equation (7.9) we might conclude that multicollinearity is not a problem.


• Are we justified in this conclusion?• Let us consider the stability of the coefficients

with deletion of some observations.• Using only the first 36 observations we get the

following results:


• Comparing equation (7.11) with (7.8) and equation (7.12) with (7.9) we see that the coefficients in the latter equation show far greater changes than in the former equation.

• Of course, if one applies the tests for stability discussed in Section 4.11, one might conclude that the results are not statistically significant at the 5% level.

• Note that the test for stability that we use us the “predictive” test for stability.


• Finally, we might consider predicting C for the first two quarters of 1961 using equations (7.11) and (7.12).

• The predictions are:


• Thus the prediction from the equation including L is further off from the true value than the predictions from the equations excluding L.

• Thus if prediction was the sole criterion, one might as well drop the variable L.


• The example above illustrates four different ways of looking at the multicollinearity problem:– 1. Correlation between the explanatory variabl

es L and Y, which is high. – 2. Standard errors or t-ratios for the estimated

coefficients• In this example the t-ratios are significant, suggesti

ng that multicollinearity might not be serious.


– 3. Stability of the estimated coefficients when some observations are deleted.

– 4. Examining the predictions from the model: • If multicollinearity is a serious problem, the predicti

ons from the model would be worse than those from a model that includes only a subset of the set of explanatory variables.


• The last criterion should be applied if prediction is the object of the analysis. Otherwise, it would be advisable to consider the second and third criteria.

• The first criterion is not useful, as we have so frequently emphasized.

7.6 Principal Component Regression

• Another solution that is often suggested for the multicollinearity problem is the principal component regression.

• Suppose that we k explanatory variables.• Then we can consider linear functions of these v

ariables

.......

......

22112

22111

etcxbxbxbz

xaxaxaz

kk

kk


• Suppose we choose the a’s so that the variance of z1 is maximized subject to the condition that

• This is called the normalization condition.

• Z1 is then said to be the first principal component.

• It is the linear function of the x’s that has the highest variance.

1...... 21

22

21 aaa


• The process of maximizing the variance of the linear function z subject to the condition that the sum of squares of the coefficients of the x’s is equal to 1, produces k solutions.

• Corresponding to these we construct k linear functions z1, z2,…..,zk. These are called

the principal components of the x’s.


• They can be ordered so that

var(z1)>var(z2)>…..>var(zk)

• z1, the one with the highest variance, is calle

d the first principal component

• z2, with the next highest variance, is called th

e second principal component, and so on• Unlike the x’s, which are correlated, the z’s ar

e orthogonal or uncorrelated.


• The data are presented in Table 7.3.• First let us estimate an import demand function.

• The regression of y on x1, x2, x3 gives the following results:


• The R2 is very high and the F-ratio is highly significant but the individual t-ratios are all insignificant.

• This is evidence of the multicollinearity problem.

• Chatterjee and Price argue that before any further analysis is made, we should look at the residuals from this equation.


• They find (we are omitting the residual plot here) a distin

ctive pattern-the residuals declining until 1960 and then ri

sing.

• Chatterjee and Price argue that the difficulty with the mo

del is that the European Common Market began operatio

ns in 1960, causing change in import- export relationship

s

• Hence they drop the years after 1959 and consider only t

he 11 years 1949-1959. The regression results are belo

w:


• The residual plot (not shown here) is now satisfa

ctory (there are no systematic patterns), so we c

an proceed.

• Even though the R2 is very high, the coefficient o

f x1 is not significant.

• There is thus a multicollinearity problem.


• To see what should be done about it, we first

look at the simple correlations among the

explanatory variables.

• These

are .

• We suspect that the high correlation between x1

and x3 could be the source of the trouble.

036.0and,99.0,026.0 223

213

212 rrr


• Does principal component analysis help us? First, the principal components (obtained from a principal components program) are

X1, X2, X3 are the normalized values of x1, x2 ,x3.


• That is,

,where m1, m2 ,m3 are

the means and σ1, σ2 , σ3 are the standard

deviations of x1, x2 ,x3 respectively.

• Hence var(X1)=var(X2)=var(X3)=1

• The variances of the principal components are

var(z1)=1.999 var(z2)=0.998 var(z3)=0.003

22221111 /)(,/)( mxXmxX 3333 /)(and, mxX


• Note that .

• The fact that var(z3)=0 identifies that linear function a

s the source of multicollinearity.

• In this example there is only one such linear function. In some examples there could be more.

• Since E(X1)=E(X2)=E(X3)=0 because of normalization,

the z’s have mean zero.

• Thus z3 has mean zero and its variance is also close

to zero. Thus we can say that .

3)var()var( ii Xz

03 z


• Looking at the coefficients of the X’s, we can say that (ignoring the coefficients that are very small)


• In terms of the original (nonnormalized) variables the regression of x3 on x1 is (figure in parentheses is standard error)

998.0686.0258.6 21

)0077.0(3 rxx


then substituting for x3 in terms of x1 we get

• This gives the linear functions of the β‘s that are estimable.

• They are (β2+6.258β3), (β1+0.686β3), and β2.

• The regression of y and x1 and x2 gave the following results:


• Of course, we can estimate a regression of x1

and x3.

• The regression coefficient is 1.451.

• We now substitute for x1 and estimate a

regression y on x1 and x3.

• The results we get are slightly better (we get a higher R2).


• The results are:

• The coefficient of x3 now is )451.1( 13


• Suppose that we consider regressing y on the

components z1 and z2 (z3 is omitted because it is

almost zero).

• We saw that .

• We have to transform these to the original

variables.

22311 and)(7.0 XzXXz


• We get


• Thus, using z2 as a regressor is equivalent to usi

ng x2, and using z1 is equivalent to using

.

• Thus the principal component regression amoun

ts to regressing y on .

• In our example .

))/(( 3311 xx

23311 and))/(( xxx

4536.1/ 31


• The results are

• This is the regression equation we would have estimated if we assumed that .

• Thus the principal component regression amounts, in this example, to the use of the prior information .

11313 4536.1)/(

13 4536.1

7.7 Dropping Variables

• Consider the model

• If our main interest is β1. Then we drop x2 and estimate the equation (7.16):

)15.7(2211 uxxy


• Then we drop x2 and estimate the equation

y= β1 x1+v (7.16)

• Let the estimator of β1 from the complete model

(7.15) be denoted by and the estimator of β1

from the omitted variable model be denoted by .

• is the OLS estimator and is the OV (omitted variable) estimator.

1̂

1̂

*1

*1


• As an estimator of β1, we use the conditional omitted variable (COV) estimator, defined as


• Also, instead if using ,depending on we can consider a linear combination of both, namely

• This is called the weighted (WTD) estimator and it has minimum mean-square error if .

• Again t2 is not known and we have to use its estimated value .

2t̂

2t̂

*11 orˆ

*11 )1(ˆ

)1/( 22

22 tt

7.8 Miscellaneous Other Solutions

• Using Ratios or First Differences • Getting More Data

chapter 7 multicollinearity. what is in this chapter? how do we detect this problem? what are the...

Documents