1

Summary of Matrix Algebra

2

Outline

1. Basic Definitions

2. Matrix Operations and Features

3. Moments and Distributions of Random Vectors

3

Basic Definitions

Matrix

A matrix is a rectangular array of numbers.

𝐴 = 𝑎𝑖𝑗 =

𝑎11 ⋯ 𝑎1𝑛⋮ ⋱ ⋮𝑎𝑚1 ⋯ 𝑎𝑚𝑛

Square Matrix

A square matrix has the same number of rows and columns.

4

Basic Definitions

Vectors

A 1 × 𝑚 matrix is called a row vector and can be written as 𝑥 ≡ (𝑥1, 𝑥2, … , 𝑥𝑚).

An 𝑛 × 1 matrix is called a column vector and can be written as

𝑦 ≡

𝑦1𝑦2⋮𝑦𝑛

5

Basic Definitions

Diagonal Matrix

A square matrix 𝐴 is a diagonal matrix when all of its off-diagonal elements are zero.

𝐴 =

𝑎11 0 0 … 00 𝑎22 0 … 0

⋮0 0 0 … 𝑎𝑛𝑛

6

Basic Definitions

Identity Matrix

The 𝑛 × 𝑛 identity matrix is the diagonal matrix with unity(one) in each diagonal position, and zero elsewhere.

𝐼 ≡ 𝐼𝑛 ≡

1 0 0 … 00 1 0 … 0⋮0 0 0 … 1

Zero Matrix

The 𝑚 × 𝑛 zero matrix is the matrix with zero for all entries.

7

Outline

1. Basic Definitions

2. Matrix Operations and Features

3. Moments and Distributions of Random Vectors

8

Matrix Operations

Matrix Addition

𝐴 + 𝐵 = 𝑎𝑖𝑗 + 𝑏𝑖𝑗

Scalar Multiplication

𝛾𝐴 = 𝛾𝑎𝑖𝑗

Matrix Multiplication

𝐴𝑚×𝑛𝐵𝑛×𝑝 = 𝑎𝑖𝑘𝑏𝑘𝑗

𝑛

𝑘=1 𝑚×𝑝

9

Matrix Operations

Transpose

Let 𝐴 = 𝑎𝑖𝑗 be an 𝑚 × 𝑛 matrix. The transpose of

𝐴, denoted 𝐴′, is the 𝑛 ×𝑚 matrix obtained by interchanging the rows and columns of 𝐴.

𝐴′ = 𝑎𝑗𝑖

Symmetric Matrix ?

A square matrix 𝐴 is a symmetric matrix if and

only if 𝐴′ = 𝐴

10

Matrix Operations

Trace?

For any 𝑛 × 𝑛 matrix 𝐴, the trace of 𝐴, denoted 𝑡𝑟(𝐴), is the sum of its diagonal element.

𝑡𝑟 𝐴 = 𝑎𝑖𝑖

𝑛

𝑖=1

Inverse?

An 𝑛 × 𝑛 matrix 𝐴 has an inverse, denoted 𝐴−1, provided that 𝐴−1𝐴 = 𝐴𝐴−1 = 𝐼𝑛.

In this case, 𝐴 is said to be invertible or nonsingular.

11

Linear Independence. Rank of a Matrix

Let {𝑥1, 𝑥2, … , 𝑥𝑟} be a set of 𝑛 × 1 vectors. Linear Independence ?

These are linearly independent vectors if and only if

𝛼1𝑥1 + 𝛼2𝑥2 +⋯+ 𝛼𝑟𝑥𝑟 = 0

implies that 𝛼1 = 𝛼2 = ⋯ = 𝛼𝑟 = 0.

12

Linear Independence. Rank of a Matrix

Let 𝐴 be an 𝑛 ×𝑚 matrix. Rank of matrix 𝐴 ?

The rank of a matrix 𝐴, denoted 𝑟𝑎𝑛𝑘(𝐴), is the maximum number of linearly independent columns of 𝐴.

If 𝐴 is 𝑛 × 𝑚 and 𝑟𝑎𝑛𝑘 𝐴 = 𝑚, then 𝐴 has full column rank.

13

Quadratic Forms

Let 𝐴 be an 𝑛 × 𝑛 symmetric matrix. The quadratic form associated with the matrix 𝐴 is the real-valued function defined for all 𝑛 × 1 vectors 𝑥:

𝑓 𝑥 = 𝑥′𝐴𝑥 = 𝑎𝑖𝑖𝑥𝑖2

𝑛

𝑖=1

+ 𝑎𝑖𝑗𝑥𝑖𝑥𝑗𝑗≠𝑖

𝑛

𝑖=1

14

Positive Definite & Positive Semi-Definite

Positive Definite Matrix?

A symmetric matrix 𝐴 said to be positive definite (p.d.) if

𝑥′𝐴𝑥 > 0 𝑓𝑜𝑟 𝑎𝑙𝑙 𝑛 × 1 𝑣𝑒𝑐𝑡𝑜𝑟𝑠 𝑥 𝑒𝑥𝑐𝑒𝑝𝑡 𝑥 = 𝟎.

Positive Semi-Definite Matrix?

A symmetric matrix 𝐴 said to be positive semi-definite (p.s.d.) if

𝑥′𝐴𝑥 ≥ 0 𝑓𝑜𝑟 𝑎𝑙𝑙 𝑛 × 1 𝑣𝑒𝑐𝑡𝑜𝑟𝑠.

15

Properties of p.d. and p.s.d. Matrices

A p.d. matrix has diagonal elements that are strictly positive, while a p.s.d. matrix has nonnegative diagonal elements.

If 𝐴 is p.d., then 𝐴−1 exists and is p.d.

If 𝑋 is 𝑛 × 𝑘, then 𝑋′𝑋 and 𝑋𝑋′ are p.s.d.

If 𝑋 is 𝑛 × 𝑘 and 𝑟𝑎𝑛𝑘 𝑋 = 𝑘, then 𝑋′𝑋 is p.d. (and therefore nonsingular).

16

Idempotent Matrices

Idempotent Matrix?

Let 𝐴 be an 𝑛 × 𝑛 symmetric matrix. Then 𝐴 is said to be an idempotent matrix if and only if

𝐴𝐴 = 𝐴.

Example?

𝐴 =

1

2

1

21

2

1

2

17

Properties of Idempotent Matrices

Let 𝐴 be an 𝑛 × 𝑛 idempotent matrix.

𝑟𝑎𝑛𝑘 𝐴 = 𝑡𝑟 𝐴 ;

𝐴 is positive semi-definite.

Example: Let 𝑋 be an 𝑛 × 𝑘 matrix with 𝑟𝑎𝑛𝑘 𝑋 = 𝑘.

𝑃 ≡ 𝑋 𝑋′𝑋 −1𝑋′

𝑀 ≡ 𝐼𝑛 − 𝑃 = 𝐼𝑛 − 𝑋 𝑋′𝑋 −1𝑋′

P and M are symmetric, idempotent matrices with 𝑟𝑎𝑛𝑘 𝑃 = 𝑘 & 𝑟𝑎𝑛𝑘 𝑀 = 𝑛 − 𝑘.

18

Differentiation of Linear and Quadratic Forms

For a given 𝑛 × 1 vector 𝑎, define

𝑓 𝑥 = 𝑎′𝑥,

for all 𝑛 × 1 vector 𝑥. So:

𝜕𝑓(𝑥)

𝜕𝑥= 𝑎′.

For an 𝑛 × 𝑛 symmetric matrix 𝐴, define

𝑔 𝑥 = 𝑥′𝐴𝑥

So:

𝜕𝑔(𝑥)

𝜕𝑥= 2𝑥′𝐴.

19

Outline

1. Basic Definitions

2. Matrix Operations and Features

3. Moments and Distributions of Random Vectors

20

Expected Value

If 𝑍 is an 𝑛 × 𝑚 random matrix, 𝐸(𝑍) is the 𝑛 × 𝑚 matrix of expected values:

𝐸 𝑍 = 𝐸(𝑧𝑖𝑗)

Properties

𝐴:𝑚 × 𝑛 matrix and 𝑏: 𝑛 × 1 vector, where both are nonrandom, then

E Ay + b = AE y + b

𝐴: 𝑝 × 𝑛 and 𝐵:𝑚 × 𝑘, where both are nonrandom, then

𝐸 𝐴𝑍𝐵 = 𝐴𝐸 𝑍 𝐵

21

Variance-Covariance Matrix

If 𝑦 is an 𝑛 × 1 random vector, its variance-covariance matrix, denoted 𝑉𝑎𝑟(𝑦), is defined as

𝑉𝑎𝑟 𝑦 =

𝜎12 𝜎12 … 𝜎1𝑛𝜎21 𝜎2

2 … 𝜎2𝑛⋮𝜎𝑛1 𝜎𝑛2 … 𝜎𝑛

2

where 𝜎𝑗2 = 𝑉𝑎𝑟(𝑦𝑗) and 𝜎𝑖𝑗 = 𝐶𝑜𝑣(𝑦𝑖 , 𝑦𝑗).

𝜎𝑖𝑗 = 𝜎𝑗𝑖 , So variance-covariance matrix is

symmetric.

22

Properties of Variance-Covariance Matrix

If 𝑎 is an 𝑛 × 1 nonrandom vector, then

𝑉𝑎𝑟 𝑎′𝑦 = 𝑎′[𝑉𝑎𝑟 𝑦 ]𝑎 ≥ 0

If 𝑉𝑎𝑟 𝑎′𝑦 > 0 for all 𝑎 ≠ 0, 𝑉𝑎𝑟 𝑦 is positive definite.

If 𝜇 = 𝐸(𝑦), then

𝑉𝑎𝑟 𝑦 = 𝐸[(𝑦 − 𝜇)(𝑦 − 𝜇)′]

If 𝐴:𝑚 × 𝑛 matrix and 𝑏: 𝑛 × 1 vector, where both are nonrandom, then

𝑉𝑎𝑟 𝐴𝑦 + 𝑏 = 𝐴[𝑉𝑎𝑟(𝑦)]𝐴′

23

Multivariate Normal Distribution

If 𝑦 is an 𝑛 × 1 multivariate normal random vector with mean 𝜇 and variance-covariance matrix Σ, we write 𝑦~𝑁𝑜𝑟𝑚𝑎𝑙(𝜇, Σ) (multivariate normal distribution).

Properties

If 𝑦~𝑁𝑜𝑟𝑚𝑎𝑙(𝜇, Σ), then each element of 𝑦 is normally distributed.

If 𝑦~𝑁𝑜𝑟𝑚𝑎𝑙(𝜇, Σ), then 𝑦𝑖 and 𝑦𝑗, any two

elements of 𝑦, are independent if and only if they are uncorrelated, 𝜎𝑖𝑗 = 0.

24

Properties of Multivariate Normal Distribution

If 𝑦~𝑁𝑜𝑟𝑚𝑎𝑙(𝜇, Σ), then

𝐴𝑦 + 𝑏~𝑁𝑜𝑟𝑚𝑎𝑙 𝐴𝜇 + 𝑏, 𝐴Σ𝐴′

where A and b are nonrandom.

If 𝑦~𝑁𝑜𝑟𝑚𝑎𝑙(𝜇, Σ), then, for nonrandom matrices 𝐴 and 𝐵, 𝐴𝑦 and 𝐵𝑦 are independent if and only if 𝑨𝜮𝑩′ = 𝟎.

In particular, if Σ = 𝜎2𝐼𝑛, then 𝐴𝐵′ = 0 is necessary and sufficient for independence of 𝐴𝑦 and 𝐵𝑦.

25

Properties of Multivariate Normal Distribution

If 𝑦~𝑁𝑜𝑟𝑚𝑎𝑙(0, 𝜎2𝐼𝑛), 𝐴 is a 𝑘 × 𝑛 nonrandom matrix, and 𝐵 is an 𝑛 × 𝑛 symmetric, idempotent matrix, then 𝐴𝑦 and 𝑦′𝐵𝑦 are independent if and only if 𝐴𝐵 = 0.

If 𝑦~𝑁𝑜𝑟𝑚𝑎𝑙 0, 𝜎2𝐼𝑛 and 𝐴 and 𝐵 are nonrandom symmetric, idempotent matrices, then 𝑦′𝐴𝑦 and 𝑦′𝐵𝑦 are independent if and only if 𝐴𝐵 = 0.

26

Chi-Square Distribution

If 𝑢~𝑁𝑜𝑟𝑚𝑎𝑙(0, 𝐼𝑛), then 𝑢′𝑢 ~𝜒𝑛2.

Properties

If 𝑢~𝑁𝑜𝑟𝑚𝑎𝑙(0, 𝐼𝑛) and 𝐴 is an 𝑛 × 𝑛 symmetric, idempotent matrix with 𝑟𝑎𝑛𝑘 𝐴 = 𝑞, then

𝑢′𝐴𝑢~𝜒𝑞2

If 𝑢~𝑁𝑜𝑟𝑚𝑎𝑙(0, 𝐼𝑛) and 𝐴 and 𝐵 are 𝑛 × 𝑛 symmetric, idempotent matrices such that 𝐴𝐵 = 0, then 𝑢′𝐴𝑢 and 𝑢′𝐵𝑢 are independent, chi-square random variables.

27

t Distribution and F Distribution

If 𝑢~𝑁𝑜𝑟𝑚𝑎𝑙(0, 𝐼𝑛), 𝑐 is an 𝑛 × 1 nonrandom vector, 𝐴 is a nonrandom 𝑛 × 𝑛 symmetric, idempotent matrix with rank q, and 𝐴𝑐 = 0, then

𝑐′𝑢 𝑐′𝑐 1 2

𝑢′𝐴𝑢 1 2 ~𝑡𝑞

If 𝑢~𝑁𝑜𝑟𝑚𝑎𝑙(0, 𝐼𝑛) and 𝐴 and 𝐵 are 𝑛 × 𝑛 nonrandom symmetric, idempotent matrices with 𝑟𝑎𝑛𝑘 𝐴 = 𝑘1, 𝑟𝑎𝑛𝑘 𝐵 = 𝑘2, and 𝐴𝐵 = 0, then

𝑢′𝐴𝑢 𝑘1

𝑢′𝐵𝑢 𝑘2 ~𝐹𝑘1,𝑘2