biol 582 lecture set 19 matrices, matrix calculations, linear models using linear algebra
TRANSCRIPT
BIOL 582
Lecture Set 19
Matrices, Matrix calculations,
Linear models using linear algebra
• Compact method of expressing mathematical operations (including statistics)
• Makes linear models easier to compute
BIOL 582 Matrix operations
Scalar: a numberVector: an ordered list (array) of scalars (nrows x 1cols)
Matrix: a rectangular array of scalars (nrows x pcols)
3b
Nomenclature: elements (or variables) are italicized, matrices are bold. Lowercase = vector; Capital = matrixMany variants in scientific literature
•Reverse rows and columns•Represent by At or A′•Vector transpose works identically
a d
b e
c f
A ' a b c
d e f
tA = A
BIOL 582 Matrix operations: transpose
• Matrices must have same dimensions• Add/subtract element-wise• Vector addition/subtraction works identically
2 4 6 8
1 3 5 9
8 12A B =
6 12
2 4 6 8
1 3 5 9
-4 -4A B =
-4 -6
Addition
Subtraction
BIOL 582 Matrix operations: addition and subtraction
inner
• Scalar multiplication: Multiply scalar by each element in matrix or vector
• Matrix/vector multiplication is a summed multiplication
• Inner dimensions allow multiplication• Outer dimensions determine size of result• Order of matrices makes a difference: AB ≠ BA
AB
n1 × p1 * n2 × p2
BIOL 582 Matrix operations: multiplication
Inner dimension must agree or multiplication cannot take place
Scalar multiplication:
Matrix multiplication: 1 1 1 2
1 1
2 1 2 21 1
n n
i i i ii i
n n
i i i ii i
a b a b
a b a b
AB =
2 11 2 3 2 6 12 1 6 6
3 34 5 6 8 15 24 4 15 12
4 2
20 13AB =
47 31
BIOL 582 Matrix operations
• Inner (scalar) product: vector multiplication resulting in a scalar (weighted linear combination)
• Outer (matrix) product: vector multiplication resulting in a matrix
6
1 2 3 5 28
4
ta b =
1 6 5 4
2 6 5 4 12 10 8
3 18 15 12
tab =
Inner Product
Outer Product
Inner dimensions MUST AGREE!!!
BIOL 582 Matrix operations
1 0 0
0 1 0
0 0 1
I
1 1
1 1
1
1 2 4
2 5 6
4 6 3
T
4 0 0
0 1 0
0 0 2
D
0 0
0 0
0
2 1 1 0 2 1:
3 4 0 1 3 4
AI A
BIOL 582 Special matrices
np pnX X
• I: Identity matrix (equivalent to ‘1’ for matrices)• 1: A matrix of all ones• 0: A matrix of all zeros• Diagonal: diagonal contains non-zero elements • Square: n = p• Symmetric: off-diagonal elements same:
• Orthogonal: square matrix with property: • VERY useful for statistics and other fields (e.g,
morphometrics)
t AA I
.7071 .7071 .7071 .7071 1 0:
.7071 .7071 .7071 .7071 0 1t
AA IOrthonormal Example:
BIOL 582 Special Matrices
• Cannot divide matrices,
• so calculate the inverse (reciprocal) of denominator and multiply
• Inverses have property that:
• Inverses are tedious to calculate, so in practice we use a computer
• Only works for square matrices whose determinant ≠ 0 (singular)
• Determinant: combination of diagonal and off-diagonal elements
A
B
det( )a ad bc Aa b
c d
A
BIOL 582 Matrix operations: division
• For the 2 x 2 case:
Example:
Confirm:
a b
c d
A 1 1
d b
d b
c a c a
A AA
A
A A
2 3
1 4
A
1
4 34 3 0.8 0.61 5 51 2 1 2 0.2 0.44*2 3*1
5 5
A
1 2 3 .8 .6 1 0
1 4 .2 .4 0 1
AA
BIOL 582 Matrix operations: invserse
• The linear equation
• Can be written in matrix form as
• where
BIOL 582 Linear Model using matrix operations
• Why is it so simple? Consider just this part
• for a simple example of four subjects and two independent variables:
BIOL 582 Linear Model using matrix operations
• The linear model is:
• The estimated coefficients (parameter estimates) are solved as:
• How/why?
• Try to solve for
• Cannot divide both sides by X• Cannot multiply by inverse of X, unless X is square-
symmetric
BIOL 582 Linear Model using matrix operations
• Making X symmetric:
• This matrix can be inverted:
• So, multiplying both sides of by will assist inverting the necessary part
• Note the dimensions so far: (k x n)(n x 1) = (k x n)(n x k)(k x 1) (k x 1) = (k x 1)
• Now multiply both sides by inverse above
• Which has dimensions: (k x n)(n x k) (k x 1) = (k x n)(n x k) (k x 1)
(k x 1) = (k x 1)
BIOL 582 Linear Model using matrix operations
• The equation
• Simplifies to
• And the dimensions of each side remain (k x 1)
• One problem is that the predicted values of the response are unknown without knowing the parameter estimates. However, the best estimates of the response values are the values themselves, so the equation is written as
• What this means is that one does not have to calculate SS for x and y and solve each coefficient independently!
BIOL 582 Linear Model using matrix operations
• Done for a simple linear model of head size as a function of log SVL
BIOL 582 Example in R using Snake data
> snake<-read.csv("snake.data.csv")> attach(snake)> # number of responses> n<-length(HS)> X<-matrix(c(rep(1,n),log(SVL)), nrow=n, ncol=2)> X[1:10,]
[,1] [,2] [1,] 1 3.532226 [2,] 1 4.062166 [3,] 1 4.075841 [4,] 1 4.359270 [5,] 1 4.387014 [6,] 1 4.432007 [7,] 1 4.437934 [8,] 1 4.443827 [9,] 1 4.480740[10,] 1 4.488636
> dim(X)[1] 40 2
• Done for a simple linear model of head size as a function of log SVL
BIOL 582 Example in R using Snake data
> y<-matrix(HS, nrow=n, ncol=1)> y[1:10,];dim(y)
[,1] [1,] 11.40 [2,] 15.30 [3,] 7.16 [4,] 10.50 [5,] 10.30 [6,] 8.25 [7,] 9.74 [8,] 13.10 [9,] 15.10[10,] 14.10
[1] 40 1
• Done for a simple linear model of head size as a function of log SVL
BIOL 582 Example in R using Snake data
> B<-solve(t(X)%*%X)%*%t(X)%*%y> B [,1][1,] -14.377543[2,] 5.695249> > # compare to canned function> lm.snake<-lm(HS~log(SVL),x=T)> lm.snake
Call:lm(formula = HS ~ log(SVL), x = T)
Coefficients:(Intercept) log(SVL) -14.378 5.695
• Done for a simple linear model of head size as a function of log SVL
BIOL 582 Example in R using Snake data
> # Predictions (fitted values)> y.hat<-X%*%B> y.hat[1:7,][1] 5.739363 8.757504 8.835389 10.449585 10.607597 10.863840 10.897599> > # Residuals> e<-y-y.hat> e[1:7,][1] 5.66063702 6.54249646 -1.67538851 0.05041518 -0.30759682 -2.61383971 -1.15759944> > # Compare to> predict(lm.snake)[1:7] 1 2 3 4 5 6 7 5.739363 8.757504 8.835389 10.449585 10.607597 10.863840 10.897599
> resid(lm.snake)[1:7] 1 2 3 4 5 6 7 5.66063702 6.54249646 -1.67538851 0.05041518 -0.30759682 -2.61383971 -1.15759944 >
• After solving
• How does one determine if any or all coefficients are significant?• Do the same thing for a reduced model and compare SSE• First, how does one find SSE?
• First:
• Then
• Thus
BIOL 582 Analysis of variance using matrix operations
• How is ?
• Using the snake example…
BIOL 582 Analysis of variance using matrix operations
> SSE.f<-t(e)%*%e> SSE.f [,1][1,] 236.9725
• ANOVA step by step for the snake data
BIOL 582 Analysis of variance using matrix operations
> # ANOVA by hand, with matrix operations> > X.f<-matrix(c(rep(1,n),log(SVL)),nrow=n,ncol=2)> X.r<-matrix(rep(1,n),nrow=n,ncol=1)> y<-matrix(HS, nrow=n, ncol=1)> B.f<-solve(t(X.f)%*%X.f)%*%t(X.f)%*%y> B.r<-solve(t(X.r)%*%X.r)%*%t(X.r)%*%y> e.f<-y-X.f%*%B.f> e.r<-y-X.r%*%B.r> SSE.f<-t(e.f)%*%e.f> SSE.r<-t(e.r)%*%e.r> > SSE.f [,1][1,] 236.9725> SSE.r [,1][1,] 525.9513
> k.f<-ncol(X.f);k.r<-ncol(X.r)> F.snake<-((SSE.r-SSE.f)/(k.f-k.r))/(SSE.f/(n-k.f))> F.snake [,1][1,] 46.33955
> P.value<-1-pf(F.snake,(k.f-k.r),(n-k.f))> P.value [,1][1,] 4.487631e-08
> R2<-(SSE.r-SSE.f)/(SSE.r) # only because X.r includes only an intercept> R2 [,1][1,] 0.5494403
• ANOVA for the snake data, this time relying on lm functions
BIOL 582 Analysis of variance using matrix operations
> # ANOVA first using lm, then matrix operations> > lm.f<-lm(HS~log(SVL),x=T)> lm.r<-lm(HS~1,x=T)> e.f<-resid(lm.f)> e.r<-resid(lm.r)> SSE.f<-t(e.f)%*%e.f> SSE.r<-t(e.r)%*%e.r> > SSE.f [,1][1,] 236.9725> SSE.r [,1][1,] 525.9513> > k.f<-ncol(X.f);k.r<-ncol(X.r)> F.snake<-((SSE.r-SSE.f)/(k.f-k.r))/(SSE.f/(n-k.f))> F.snake [,1][1,] 46.33955> P.value<-1-pf(F.snake,(k.f-k.r),(n-k.f))> P.value [,1][1,] 4.487631e-08> > R2<-(SSE.r-SSE.f)/(SSE.r)> R2 [,1][1,] 0.5494403
• ANOVA for the snake data, what R does should be clear now
BIOL 582 Analysis of variance using matrix operations
> # ANOVA via model comparison method> > lm.f<-lm(HS~log(SVL),x=T)> lm.r<-lm(HS~1,x=T)> > anova(lm.r,lm.f)Analysis of Variance Table
Model 1: HS ~ 1Model 2: HS ~ log(SVL) Res.Df RSS Df Sum of Sq F Pr(>F) 1 39 525.95 2 38 236.97 1 288.98 46.34 4.488e-08 ***---Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 >
• ANOVA for the snake data, what R does should be clear now
BIOL 582 Analysis of variance using matrix operations
> # or just a model summary> summary(lm.f)
Call:lm(formula = HS ~ log(SVL), x = T)
Residuals: Min 1Q Median 3Q Max -4.4953 -1.6932 -0.3986 1.1925 6.5425
Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -14.3775 3.5088 -4.098 0.000211 ***log(SVL) 5.6952 0.8366 6.807 4.49e-08 ***---Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 2.497 on 38 degrees of freedomMultiple R-squared: 0.5494, Adjusted R-squared: 0.5376 F-statistic: 46.34 on 1 and 38 DF, p-value: 4.488e-08
• It is worth looking at design matrices…
BIOL 582 Analysis of variance using matrix operations
> X.f [,1] [,2] [1,] 1 3.532226 [2,] 1 4.062166 [3,] 1 4.075841 [4,] 1 4.359270 [5,] 1 4.387014 [6,] 1 4.432007 [7,] 1 4.437934 [8,] 1 4.443827 [9,] 1 4.480740[10,] 1 4.488636[11,] 1 4.509760[12,] 1 4.514151[13,] 1 3.918005[14,] 1 4.146304[15,] 1 4.309456[16,] 1 4.423648[17,] 1 4.479607[18,] 1 4.490881[19,] 1 4.499810[20,] 1 4.567468[21,] 1 4.603168[22,] 1 4.614130[23,] 1 4.668145[24,] 1 4.700480[25,] 1 4.720283[26,] 1 3.303217[27,] 1 3.411148[28,] 1 3.540959[29,] 1 4.326778[30,] 1 4.398146
> lm.f$x (Intercept) log(SVL)1 1 3.5322262 1 4.0621663 1 4.0758414 1 4.3592705 1 4.3870146 1 4.4320077 1 4.4379348 1 4.4438279 1 4.48074010 1 4.48863611 1 4.50976012 1 4.51415113 1 3.91800514 1 4.14630415 1 4.30945616 1 4.42364817 1 4.47960718 1 4.49088119 1 4.49981020 1 4.56746821 1 4.60316822 1 4.61413023 1 4.66814524 1 4.70048025 1 4.72028326 1 3.30321727 1 3.41114828 1 3.54095929 1 4.32677830 1 4.398146
> X.r [,1] [1,] 1 [2,] 1 [3,] 1 [4,] 1 [5,] 1 [6,] 1 [7,] 1 [8,] 1 [9,] 1[10,] 1[11,] 1[12,] 1[13,] 1[14,] 1[15,] 1[16,] 1[17,] 1[18,] 1[19,] 1[20,] 1[21,] 1[22,] 1[23,] 1[24,] 1[25,] 1[26,] 1[27,] 1[28,] 1[29,] 1[30,] 1
> lm.r$x (Intercept)1 12 13 14 15 16 17 18 19 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 1
• Now for an example for a single factor ANOVA
BIOL 582 Analysis of variance using matrix operations
> # Single factor Anova example, relying more so on lm commands> lm.f<-lm(HS~Sex,x=T)> lm.f$x (Intercept) SexM1 1 02 1 03 1 04 1 05 1 06 1 07 1 08 1 09 1 010 1 011 1 012 1 013 1 114 1 115 1 116 1 117 1 118 1 119 1 120 1 121 1 122 1 123 1 124 1 125 1 126 1 027 1 028 1 029 1 030 1 0
> lm.f<-lm(HS~Sex,x=T)> X.f<-lm.f$x> lm.r<-lm(HS~1,x=T)> X.r<-lm.r$x> y<-HS> B.f<-solve(t(X.f)%*%X.f)%*%t(X.f)%*%y> B.r<-solve(t(X.r)%*%X.r)%*%t(X.r)%*%y> e.f<-y-X.f%*%B.f> e.r<-y-X.r%*%B.r> SSE.f<-t(e.f)%*%e.f> SSE.r<-t(e.r)%*%e.r> > SSE.f [,1][1,] 522.5028> SSE.r [,1][1,] 525.9513
> k.f<-ncol(X.f);k.r<-ncol(X.r)> F.snake<-((SSE.r-SSE.f)/(k.f-k.r))/(SSE.f/(n-k.f))> F.snake [,1][1,] 0.2508023> P.value<-1-pf(F.snake,(k.f-k.r),(n-k.f))> P.value [,1][1,] 0.6193992> > R2<-(SSE.r-SSE.f)/(SSE.r)> R2 [,1][1,] 0.006556786
• Now for an example for a single factor ANOVA
BIOL 582 Analysis of variance using matrix operations
> summary(lm.f)
Call:lm(formula = HS ~ Sex, x = T)
Residuals: Min 1Q Median 3Q Max -5.911 -2.356 0.069 3.337 5.619
Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 9.6811 0.8740 11.077 1.85e-13 ***SexM -0.5902 1.1785 -0.501 0.619 ---Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 3.708 on 38 degrees of freedomMultiple R-squared: 0.006557, Adjusted R-squared: -0.01959 F-statistic: 0.2508 on 1 and 38 DF, p-value: 0.6194
> B.f [,1](Intercept) 9.681111SexM -0.590202>