techniques for studying correlation and covariance structure principal components analysis (pca)...
Post on 23-Dec-2015
235 Views
Preview:
TRANSCRIPT
Techniques for studying correlation and covariance structure
Principal Components Analysis (PCA)
Factor Analysis
Principal Component Analysis
Let x
and covariance matrix .
Definition:
1 1 1 p pC a x a x a x
have a p-variate Normal distribution
with mean vector
The linear combination
is called the first principal component if
1, , pa a a
is chosen to maximize
1Var C Var a x a a
subject to2 21 1pa a a a
Let
, 1 1g a V a a a a a a
Consider maximizing
subject to 2 21 1pa a a a
V Var a x a a
Using the Lagrange multiplier technique
Now
,1 0 if 1
g aa a a a
and
,2 2 0 if
g aa a a a
a
Thus is an eigenvector of and is the eigenvalue
associated with .
a
a
Also Var a x a a a a a a
Hence is maximized if is the largest
eigenvalue of .
Var a x
Summary
1 1 1 p pC a x a x a x
is the first principal component if 1
p
a
a
a
2 21i.e. 1pa a a a
is the eigenvector (length 1)of associated with the largest eigenvalue 1 of .
Let x
and covariance matrix .
Definition:
1 11 1 1 1
1 1
p p
p p pp p p
C a x a x a x
C a x a x a x
have a p-variate Normal distribution
with mean vector
The set of linear combinations
are called the principal components of
1, ,i i ipa a a
2 21 1i i i ipa a a a
The complete set of Principal components
x
if are chosen such that
Note: we have already shown that
1 11 1, , pa a a
is the eigenvector of associated with the largest eigenvalue, 1 ,of the covariance matrix and
1 1 1 1 1Var C Var a x a a
and
1. Var(C1) is maximized.
2. Var(Ci) is maximized subject to Ci being independent of C1, …, Ci-1 (the previous i -1 principle components)
We will now show that
1, ,i i ipa a a
is the eigenvector of associated with the ith largest eigenvalue, i of the covariance matrix and
i i i i iVar C Var a x a a
Proof (by induction – Assume true for i -1, then prove true for i)
1 1 1
i
i i i
C a x a
x A x
C a x a
Now
has covariance matrix
1
1i i i
i
a
A A a a
a
1 1 1 1 1 1 1
1 1 1 1 1 1 1
1 1 1 1
0
0
i i i
i i i i i i i i
i i i i i i i i i i
a a a a a a a a
a a a a a a a a
a a a a a a a a a a a a
Hence Ci is independent of C1, …, Ci-1 if
1 1 1, , ,i i i i ig a a a a a
We want to maximize i i iVar C a a
subject to
1 11. 0i i ia a a a
2. 1i ia a
Let
1 1
1 1 1
1 1
0
0
0
i i i
i i i i
i i i i
a a a a
a a a a
a a a a
1 1 1i i i i i ia a a a
1 1, , , ,0 for 1, 1i i i
j ij
g aa a j i
Now
and
1 1, , , ,1 0i i i
i ii
g aa a
Now
1, , ,i i
i
g a
a
1 1 1 12 2
0i i i i ia a a a
1 1 1 1
1
2i i i i ia a a a
1 1 1 1
1
2j i i j i j i j ia a a a a a a a
1
0 02 j j ja a
hence
Also for j < i
Hence j = 0 for j < I and equation (1) becomes
(1)
i i ia a
1, , pa a
i p
the principal component thi iC i a x
are the eignevectors of associated with the eigenvalues
Thus
and
1. Var(C1) is maximized.
2. Var(Ci) is maximized subject to Ci being independent of C1, …, Ci-1 (the previous i -1 principal components)
where
1, , pa a
0i p
Recall any positive matrix,
where are eigenvectors of of length 1 and
1 1
1
0
, ,
0p
p p
a
a a PDP
a
are eigenvalues of
1, , is an orthogonal matrix.
( )
pP a a
P P PP I
Example
In this example wildlife (moose) population density was measured over time (once a year) in three areas.
Year Area 1 Area 2 Area 3 Year Area 1 Area 2 Area 3
1 11.3 14.1 6.9 13 6.1 9.9 6.82 10.4 14 11.2 14 9.7 13.2 6.63 9.9 13 8.7 15 8.1 9.4 44 8.2 11.4 3.3 16 11.3 11.8 4.95 10.1 11.9 8.7 17 8.8 11.5 8.86 10.7 13.8 12.5 18 9.4 11.6 5.77 11 14.9 8.9 19 7.5 11.4 4.98 7.1 8.5 3.7 20 8.8 10.7 7.29 14.7 14.5 12.1 21 7.5 11.1 7
10 5.4 9 4.1 22 9.1 13.2 8.911 7.3 7.6 5.6 23 6.8 9.8 7.612 10.2 10.9 7.3
picture
Area 3
Area 1
Area 2
The Sample Statistics
4.297 3.307 3.295
3.527 3.527
6.566
S
9.10
11.62
7.19
x
1 .796 .620
1 .687
1
R
The mean vector
The covariance matrix
The correlation matrix
Principal component Analysis
1 2 3
.522 .582 .624
.523 , .359 , .733
.674 .730 .117
a a a
1 2 311.85974, 2.204232, 0.814249 The eigenvalues of S
The eigenvectors of S
1 1 2 3
2 1 2 3
3 1 2 3
.522 .523 .674
.582 .359 .730
.624 .733 .117
C x x x
C x x x
C x x x
The principal components
Area 3
Area 1
Area 2
1 1 2 3.522 .523 .674C x x x
Area 3
Area 1
Area 2
2 1 2 3.582 .359 .730C x x x
Area 3
Area 1
Area 2
3 1 2 3.624 .733 .117C x x x
Graphical Picture of Principal Components
Multivariate Normal data falls in an ellipsoidal pattern.
The shape and orientation of the ellipsoid is determined by the covariance matrix
The eignevectors of are vectors giving the directions of the axes of the ellopsoidThe eigenvalues give the length of these axes.
Recall that if is a positive definite matrix
1 1 1 p p pa a a a
1 1
1 1
0
, ,
0 p p
a
a a
a
PDPwhere P is an orthogonal matrix (P’P = PP’ = I) with the columns equal to the eigenvectors of .and D is a diagonal matrix with diagonal elements equal to the eigenvalues of .
The vector of Principal components
1 1 1
p p p
C a x a
C x P x
C a x a
c P P P PDP P P P D P P
has covariance matrix
1 0
0 p
D
An orthogonal matrix rotates vectors, thus
C P x
ctr tr P P tr PP tr
Also
1 1
p p
i iii i
1 1
var var Total Variance of p p
i ii i
C x x
rotates the vector x
into the vector of Principal components C
tr(D) =
The ratio
1 1
var
Total Variance of ii i
p p
j jjj j
C
x
denotes the proportion of variance explained by the ith principal component Ci.
The Example
i i % variance
1 11.8597 79.71%2 2.20423 14.82%3 0.81425 5.47%
Total 14.8782 100%
cov , cov ,i j i j i jC x a x e x a e
Also
1 1 1i p p p ja a a a a e
i i j i ija e a
0,0, ,0,1,0, ,0jj
e
where
cov ,corr ,
i j
i j
i j
C xC x
Var C Var x
i ij iij
jji jj
aa
Comment:If instead of the covariance matrix, , The correlation matrix , is used to extract the Principal components then the Principal components are defined in terms of the standard scores of the observations:
i ii
ii
xz
*and corr i i ijC a
The correlation matrix is the covariance matrix of the standard scores of the observations:
More Examples
Example 2: Bone Lengths of White Leghorn Fowl: The correlation matrix of the complete set of six fowl bone
measurements had the following form: Skull Length 1.000 0.584 0.615 0.601 0.570 0.600 Skull Breadth 1.000 0.576 0.530 0.526 0.555 Humerus 1.000 0.940 0.875 0.878 Ulna 1.000 0.877 0.886 Femur 1.000 0.924 Tibia 1.000
Table: Principal Components Component Dimension 1 2 3 4 5 6
Skull: Length 0.74 0.45 0.49 -0.02 0.01 0.00 Breadth 0.70 0.59 -0.41 0.00 0.00 -0.01 Wing: Humerus 0.95 -0.16 -0.03 0.22 0.05 0.16 Ulna 0.94 -0.21 0.01 0.20 -0.04 -0.17 Leg: Femur 0.93 -0.24 -0.04 -0.21 0.18 -0.03 Tibia 0.94 -0.19 -0.03 -0.20 -0.19 0.04 Eigenvalue 4.568 0.714 0.412 0.173 0.076 0.057 % of Total 76.1 11.9 6.9 2.9 1.3 0.9 Variance
Identification of Components: component Description 1 General average of all bone dimensions (Size) 2 Comparison of skull sizewith Wing and Leg lengths 3 Comparison of skull length and breadth (Skull Shape) 4 Comparison of Wing and Leg lengths 5 Comparison of femur and tibia 6 Comparison of humerus and ulna
Example 3: Weschler Adult Intelligence Scale Subtest Scores Table: Principal Components Component 1 2 3 4 WAIS subtest: Information 0.83 0.33 -0.04 -0.01 Comprehension 0.75 0.31 0.07 -0.17 Arihtmetic 0.72 0.25 -0.08 0.35 Similarities 0.78 0.14 0.00 -0.21 Digit Span 0.62 0.00 -0.38 0.58 Vocabulary 0.83 0.38 -0.03 -0.16 Digit Symbol 0.72 -0.36 -0.26 -0.01 Picture Completion 0.78 -0.10 -0.25 -0.01 Block Design 0.72 -0.26 0.36 0.18 Picture Arrangement 0.72 -0.23 0.04 -0.05 Object assembly 0.65 -0.30 0.47 0.13 Age: -0.34 0.80 0.26 0.18 Years of Education: 0.75 0.01 -0.30 -0.23
Eigenvalue 6.69 1.42 0.80 0.71
% of Total Variance 51.47 10.90 6.15 5.48 Cum % of Variance 51.47 62.37 68.52 74.01
Identification of Components: component Description 1 General intellectual Performance 2 Experiential or age factor - bipolar dimension comparing verbal
or informational skills known to increase with advancing age to subtests measuring spatial-perceptual qualities and other cognitive abilities known to decrease with age
3 Spatial imagery or perception dimension 4 Numerical facility
Computation of the eigenvalues and eigenvectors of
1 1 1 p p pa a a a
21 1 1 p p pa a a a
1 1 1 p p pa a a a
2 21 1 1 p p pa a a a
Recall:
continuing we see that:
1 1 1n n n
p p pa a a a
21 1 1
1 1
n n
pnp p p pa a a a a a
1 1 1n a a
For large values of n
1 1n a a
The algorithm for computing the eigenvector 1a
1. Compute
2 4 8 16, , , , etc
rescaling so that the elements do not become to large in value.
i.e. rescale so that the largest element is 1.
2. Compute
1 1 1 1 for large and 1n ka a n a a 1a
using the fact that:
3. Compute 1 using
1 1 1a a
4. Repeat using the matrix
21 1 1 2 2 2 p p pa a a a a a
5. Continue with i = 2 , … , p – 1 using the matrix
11 1 1
i ii i i i i i p p pa a a a a a
Example – Using Excel - Eigen
top related