NOTES ON MULTIPLE REGRESSION USING MATRICES

Multiple Regression

Tony E. Smith

ESE 502: Spatial Data Analysis

Matrix Formulation of Regression

Applications to Regression Analysis

SIMPLE LINEAR MODEL

Data: ( , ) , 1,..,i iy x i n

Parameters: 2

0 1( , ) ,

Model:

0 1 , 1,..,i i iY x i n 2~ (0, ) , 1,..,i iid N i n

0 1( | ) , 1,..,i i iE Y x x i n

SIMPLE REGRESSION ESTIMATION

Data Points: ( , )i iy x

Predicted Value:

0 1ˆ ˆˆi iy x

Estimate Conditional Mean:

0 1( | )E Y x x

iyy

ˆiy

iyix

Line of Best Fit

where:

0 1

20 1 0 11( , )

ˆ ˆ( , ) min [ ( )]n

i iiy x

STANDARD LINEAR MODEL

Data: 1( , ,.., ) , 1,..,i i iky x x i n

Parameters: 2

0 1( , ,.., ) ,k

Model:

0 1, 1,..,

k

i j ij ijY x i n

2~ (0, ) , 1,..,i iid N i n

1 0 1( | ,.., )

k

i i ik j ijjE Y x x x

STANDARD LINEAR MODEL (k = 2)

Data: 1 2( , , ) , 1,..,i i iy x x i n

Parameters: 2

0 1 2( , , ) ,

Model:

0 1 1 2 2 , 1,..,i i i iY x x i n 2~ (0, ) , 1,..,i iid N i n

1 2 0 1 1 2 2( | , )i i i i iE Y x x x x

REGRESSION ESTIMATION (for k =2)

Plane of Best Fit

1 2( , )i ix x

iy

ˆiy

y

1x

2x

Data Points: 1 2( , , )i i iy x x

Predicted Value:

0 1 1 2 2ˆ ˆ ˆˆi i iy x x

where:

0 1 2

20 1 2 0 1 1 2 21( , , )

ˆ ˆ ˆ( , , ) min [ ( )]n

i i iiy x x

MATRIX REPRESENTATION OFTHE STANDARD LINEAR MODEL

Vectors and Matrices:

1

2 ,:n

YYY

Y

11 12

21 22

1 2

11 ,: : :1 n n

x xx xX

x x

0

1

2

,

1

2:n

Matrix Reformulation of the Model:

00where: 0

0

1 0 00 1and 00 0 1

nI

Y X 2~ (0, )nN I

LINEAR TRANSFORMATIONSIN ONE DIMENSION

Linear Function: ( )f x a x

(1) 1f a a

( ) (1)f x f x

Graphic Depiction:

0 1 a x

a x

LINEAR TRANSFORMATIONSIN TWO DIMENSIONS

Linear Transformation:

1 11 1 12 2

2 21 1 22 2( ) x a x a xf x f x a x a x

11 12

21 22

1 0,0 1a af fa a

11 2

2

1 00 1

xf f x f xx

Graphical Depiction of Linear Transformation:

01

10

1

2

xx x

01f

10f

110f x

201f x

( )f x

SOME MATRIX CONVENTIONS

11 1 11 1

( ) ( )

1 1

: : : :n k

k n n k

k kn n kn

a a a aA A A A

a a a a

1

( 1) (1 ) 1: ( ,., )k k k

k

aa a a a a a

a

Transposes of Vectors and Matrices:

Symmetric (Square) Matrices: A A

Important Example: ( )A A A A i symmes tric

Column Representation of Matrices:

11 1 111

1

1 1

: : : ,.., : ( ,.., )n n

k

k kn k kn

a a aaA a a

a a a a

Row Representation of Matrices:

11 1 11 1 1

1 1

( ,.., ): : : :

( ,.., )

n n

k kn k kn k

a a a a aA

a a a a a

1 1 1: , : :k n

k n k

a x a xA A x Ax

a x a x

Matrix Multiplication:

1 1

1: , :

n

i ii

n n

a xa x a x a x x a

a x

Inner Product of Vectors:

1 1 1

1

1

,.., : :m

n m m

k k m

a b a bB B b b AB

a b a b

( )AB B A Transposes:

MATRIX REPRESENTATIONS OF LINEAR TRANSFORMATIONS

11 12

21 22

1 0,0 1a af fa a

11 2

2

1 0( ) 0 1xf x f f x f xx

For any Two-Dimensional Linear Transformation :

with :

11 1 12 2 11 12 1

21 1 22 2 21 22 2( ) a x a x a a xf x Axa x a x a a x

Graphical Depiction of Matrix Representation:

01

10

1

2

xx x

12

22

aa

11

21

aa

111

21

a xa

122

22

a xa

Ax

Inversion of Square Matrices (as Linear Transformations):

11 1 111

1 1

1 0: : : ,.., : : ,.., :

0 1

n n

n

n nn n nn

a a aaAI

a a a a

1111

1

1 0: ,.., : : ,.., :

0 1

n

n nn

aaA

a a

1 1nA A I AA

DETERMINANTS OF SQUARE MATRICES

11 12

21 22

a aA a a

det( )A

11 22 21 21a a a a

| det( ) |A

Area of the of the unit square undimage er A

12

22

aa

11

21

aa

NONSINGULAR SQUARE MATRICES

12

22

aa

11

21

aa

1A exists

11 21

21 22

a aanda a are not colinear

det 0A

A is nonsingular

LEAST-SQUARES ESTIMATION

2 20 1 11 1

( ) [ ( )] ( )n n

i i k ik i ii iS y x x y x

2 2

1 1 12 ( )

n n n

i i i ii i iy y x x

General Sum-of-Squares:

General Regression Matrices:

11 11

2 21 2

1

11 ,: : : : :1

k

k

n n nk

x xxx x xX

x x x

0

1 ,:k

1

2 ,:n

yyy

y

1

1

,:i

i

ik

xx

x

1

2:n

xxX

x

( ) 2S y y y X X X

DIFFERENTIATION OF FUNCTIONS

General Derivative:

0

( ) ( )( ) limd

dxf x f x

f x

Example: 2( )f x x

2 2

0

( )( ) limd

dxx x

f x

2 2 2

0

( 2 )lim

x x x

0lim (2 )x

( ) 2o oddx f x x

( )of x

ox

( )of x

PARTIAL DERIVATIVES

z

1 2( , )z f x x

2ox

1 2( , )o ox x

1

1 2 1 21 2 0

( , ) ( , )( , ) lim

o o o oo o

xf x x f x x

f x x

VECTOR DERIVATIVES

1( ) ( ,.., )nf x f x x

11( )( )

( ) : :( ) ( )

n

x

n

x

x

f xf xf x

f x f x

Derivative Notation for:

1( ) ( ,.., ) , 1,..,ii nxf x f x x i n

Gradient Vector:

TWO IMPORTANT EXAMPLES

1( )

n

i iif x a x a x

Linear Functions:

( ) , 1,..,i if x a i n ( )x f x a

Quadratic Functions:

1

1( ) ( ,.., ) :n

n

a xf x x Ax x x

a x

1 1

n n

i ij ji jx a x

1( )

n

i iix a x

Quadratic Derivatives:

1 1( )

n n

k kh hk hf x x Ax x a x

1 1( )

n n

i ih h ki kh kf x a x a x

i ia x a x

1 1( ) : :x

n n

a x a xf x Ax A x

a x a x

( ) ( ) 2xA A x Ax Ax

Symmetric Case:

MINIMIZATION OF FUNCTIONS

First-Order Condition:

Example:

( *) 0ddx f x

2( ) 2f x a bx x

( ) 2 2ddx f x b x

0 ( *) 2 2 * *ddx f x b x x b

( )f x

*x

1ox

1x

2ox

z

TWO-DIMENSIONAL MINIMIZATION

1 2( , )z g x x

1 1 2( , ) 0o ox g x x

2 1 2( , ) 0o ox g x x

( ) 0ox g x

LEAST SQUARES ESTIMATION

Solution for: 0 1

ˆ ˆ ˆ ˆ( , ,.., )k

min ( ) 2( )S y y y X X X

ˆ ˆ0 ( ) 2 2S X y X X

ˆX X X y

1ˆ ( )X X X y det( ) 0if X X

NON-MATRIX VERSION (k = 2)

1 2 1 2( , , ) , 1,.., , ( , , )i i iy x x i n y x x sample means Data:

Beta Estimates:

21 2 2 1 21 1 1 1

1 2 21 2 1 21 1 1

ˆn n n n

i i i i i i ii i i i

n n n

i i i ii i i

y x x y x x x

x x x x

1 2 1 1 2 2( , , ) ( , , )i i i i i iy x x y y x x x x deviation form

21 1 1 1 21 1 1 1

2 2 21 2 1 21 1 1

ˆn n n n

i i i i i i ii i i i

n n n

i i i ii i i

y x x y x x x

x x x x

0 1 1 2 2ˆ ˆ ˆ , :y x x where

EXPECTED VALUES OF RANDOM MATRICES

Random Vectors and Matrices

11 1

1

: : :k

n k

k kn

Y YY Y

Y Y

1

1 : ,n

n

YY Y

Y

Expected Values:

11 1

1

( ) ( )( ) : : :

( ) ( )

k

k kn

E Y E YE Y

E Y E Y

1( )( ) : ,

( )n

E YE Y

E Y

EXPECTATIONS OF LINEARFUNCTIONS OF RANDOM VECTORS

Linear Combinations

1 1( ) ( ) ( )

n n

i i i ii ia Y aY E a Y a E Y a E Y

Linear Transformations

1 1 ( ): ( ) : ( )

( )n n

a Y a E YAY E AY AE Y

a Y a E Y

EXPECTATIONS OF LINEARFUNCTIONS OF RANDOM MATRICES

Left Multiplication

Right Multiplication (by symmetry of inner products):

1 1 1

1

: : :k

h n n k

h h k

a Y a YAY A Y

a Y a Y

1 1 1

1

( ) ( )( ) : : : ( )

( ) ( )

k

h h k

a E Y a E YE AY AE Y

a E Y a E Y

( ) ( )k n n hYB Y B E YB E Y B

COVARIANCE OF RANDOM VECTORS

Random Variables :

Random Vectors:

( ) , 1,..,i iE Y i n

cov( , ) [( )( )] , 1,..,i j ij i i j jY Y E Y Y i n

1( ) ( ,.., ) ,nE Y

11 1 1 1 1 1 1 1 1 1

1 1 1

[( )( )] [( )( )]cov( ) : : : : : :

[( )( )] [( )( )]

n

n nn n n n n n n

E Y Y E Y YY

E Y Y E Y Y

1 1 1 1 1 1 1 1 1 1 1 1

1 1

( )( ) ( )( ): : : : :

( )( ) ( )( )n n n n n n n n n n

Y Y Y Y Y YE E

Y Y Y Y Y Y

cov( ) [( )( ) ]Y E Y Y

COVARIANCE OF LINEARFUNCTIONS OF RANDOM VECTORS

Linear Combinations:

Linear Transformations:

cov( ) [( )( ) ]AY E AY A AY A

( )E Y

[ ( )( ) ]E A Y Y A

[( )( ) ]AE Y Y A

[( )( ) ]AE Y Y A ( Right Mult )

( Left Mult )

cov( ) cov( )AY A Y A

cov( ) cov( )a Y a Y a

TRANSLATIONS OF RANDOM VECTORS

Translation: Y b Y

cov( ) [( { })( { }) ]b Y E b Y b b Y b

Means: ( ) ( ) ( ) ( )E b Y E b E Y b E Y

Covariances:

( ) ( )E b AY b AE Y

( )E Y

[( )( ) ] cov( )E Y Y Y

cov( ) cov( )b AY A Y A

RESIDUAL VECTOR IN THE STANDARD LINEAR MODEL

Linear Model Assumption: 2~ (0, ) , 1,..,i iid N i n

Residual Means: ( ) 0 , 1,.., ( ) 0iE i n E

Residual Covariances:

2 2var( ) ( ) , cov( , ) ( ) 0 ,i i i j i jE E j i

cov( ) [( 0)( 0) ] ( )E E

21 1 1 1 1 1

21 1

( ) ( ) 0: : : : : :

( ) ( ) 0

n n

n n n n n n

E EE

E E

2cov( ) nI

MOMENTS OF BETA ESTIMATES

Linear Model: 2, ~ (0, )nY X N I

1 1ˆ ( ) ( ) ( )X X X Y X X X X 1 1( ) ( )X X X X X X X 1( )X X X

Mean of Beta Estimates:

1ˆ ˆ( ) ( ) ( ) ( )E X X X E E (Unbiased Estimator)

Covariance of Beta Estimates:

1 1ˆcov( ) cov[ ( ) ] cov[( ) ]V X X X X X X 1 1 2 1 1( ) cov( ) ( ) ( ) ( )X X X X X X X X X X X X

2 1ˆcov( ) ( )V X X

ESTIMATION OF RESIDUALVARIANCE

Residual Variance: 2 2var( ) ( ) , 1,..,i iE i n

Residual Estimates: ˆ ˆ , 1,..,i i iy y i n

Natural Estimate of Variance:

2 211

1 1ˆ ˆ ˆ ˆ ˆ ˆˆ , ( ,.., )n

i nin n where

Bias-Correct Estimate of Variance:

2 1( 1) ˆ ˆn ks (Compensates for Least Squares)

ˆ ˆ ˆˆ ˆ( ) ( ) ( )S y y y y

ESTIMATION OF BETA COVARIANCE

Beta Covariance Matrix:

11 12 1

1

ˆcov( ) ( ) : :n

n nn

v vV X X

v v

Beta Covariance Estimates:

11 12 1

1

ˆ ˆˆˆ cov ( ) ( ) : :

ˆ ˆ

n

est

n nn

v vV s X X

v v

ˆ ˆvar ( )est j iiv

ˆ ˆstd-err( )j iiv