Matrix Algebra for Applied Economics

Shayle R. Searle Departments of Biometrics and of Statistical Science Cornell University

LoisSchertzWillett Food and Resource Economics Department University of Florida

A Wiley-Interscience Publication JOHN WILEY & SONS, INC.

Contents

LIST OF CHARTERS v

PREFACE xix

I B A S I C S 1

1 INTRODUCTION 3 1.1 The Scope of Matrix Algebra 3 1.2 Using Computers for Matrix Arithmetic 4 1.3 A Matrix is an Array 5 1.4 Subscript Notation 6 1.5 Summation Notation 7 1.6 Dot Notation 11 1.7 Definition of a Matrix 11 1.8 Some Basic Special Forms 12

a. Square matrices 12 b. Diagonal matrices 13 c. Identity matrices 13 d. Triangulär matrices 13 e. Null matrices 13 f. Equal matrices 14 g. Vectors 14 h. Sealars 14

1.9 Description by Elements 15 1.10 Notation 15 1.11 Examples and Illustrations 16 1.12 Exercises 16

vu

Vlll CONTENTS

2 BASIC MATRIX OPERATIONS 21 2.1 Transposing a Matrix 21

a. A reflexive Operation 22 b. Vectors 22 c. Symmetrie matrices 22 d. Notation: iff and NSC 24

2.2 Partitioned Matrices 24 a. An example 24 b. General speeification 26 c. Transposing a partitioned matrix 26 d. Partitioning into vectors 27

2.3 Trace of a Square Matrix 27 2.4 Matrix Addition 28

a. Transposing a sum 29 b. Trace of a sum 30 c. Scalar multiplication 30

2.5 Matrix Subtraction 30 2.6 Multiplication 33

a. Inner produet of two vectors 33 b. A matrix-vector produet 34 c. A matrix-matrix produet 36 d. Existence of matrix produets 39 e. Products with vectors 40 f. Products with scalars 43 g. Products with null matrices 43 h. Products with diagonal matrices 43 i. Products with identity matrices 44 j . Transpose of a produet 44 k. Trace of a produet 45 1. Powers of a matrix 46 m. Multiplying partitioned matrices 48 n. Hadamard produets 49

2.7 The Laws of Algebra 50 a. Associative laws 50 b. Distributive law 51 c. Commutative laws 51

2.8 Contrasts with Scalar Algebra 52 2.9 Exercises 53

3 SPECIAL MATRICES 63 3.1 Linear Transformations 63

CONTENTS ix

3.2 Symmetry: Some Basic Outcomes 64 a. Verifying symmetry 64 b. Product of Symmetrie matrices 64 c. Properties of AA' and A'A 65 d. Products of vectors 66 e. Sums of outer produets 67 f. Elementary vectors 67 g. Skew-symmetric matrices 68

3.3 Summing Vectors and Their Products 68 3.4 Idempotent Matrices 71 3.5 Orthogonal Matrices 72 3.6 Quadratic Forms 73

a. Definition 73 b. The matrix A is always Symmetrie in x'Ax . . . 74 c. Numerical example 75 d. Explicit expansion 76 e. Bilinear form 76 f. Positive (and negative) definite matrices 76

- i. Positive definiteness 77 - ii. Negative definiteness 78

3.7 Variance-Covariance Matrices 78 3.8 Correlation Matrices 79 3.9 LDU Decomposition 80 3.10 Exercises 81

4 DETERMINANTS 87 4.1 Introduction 87 4.2 Expansion by Minors 90 4.3 Elementary Expansions 93

a. Determinant of a transpose 93 b. Two rows the same 94 c. Adding to a row a multiple of another row . . . . 94 d. Adding a row to a multiple of a row 97 e. Products 97

- i. Determinant of a triangulär matrix . . 98 - ii. Reducing a determinant to triangulär

form 98 - iii. Two partitioned determinants 99 - iv. Verification of the product rule 99 - v. Extensions to rectangular matrices . . . 100

x CONTENTS

- vi. Useful corollaries 101 f. Cramer's rule for linear equations 101

- i. Illustration: Input-output equations . . 102 - ii. Illustration: Supply and demand . . . . 102

4.4 Elementary Row Operations 103 a. Definitions 103 b. Eactorization 105 c. One row being a multiple of another 105 d. Row (column) of zeros 106 e. Interchanging rows (columns) 106

4.5 Diagonal Expansion 106 a. Notation for minors 106 b. Determinant of A + D 107 c. Principal minors 107 d. Special case: A + AI 108

4.6 Laplace Expansion 108 4.7 Sums and Differences of Determinants 110 4.8 Exercises 111

5 INVERSE MATRICES 117 5.1 Introduction 117 5.2 Existence and Uniqueness of an Inverse 118

a. Existence 118 b. Uniqueness 119

5.3 Rectangular Matrices 119 5.4 Cofactors 120 5.5 Deriving the Inverse 121

a. For a matrix of order 3 121 b. General case 123 c. Adjugate matrix 123

5.6 Properties of the Inverse 124 5.7 Some Simple Special Cases 124

a. Inverses of order 2 125 b. Diagonal matrices 125 c. Orthogonal matrices 125

5.8 Solving Linear Equations 126 5.9 Algebraic Simplifications 127 5.10 Partitioned Matrices 129 5.11 Left and Right Inverses 130 5.12 Using Computing Software 131

CONTENTS XI

a. Problems of accuracy 131 b. Algorithmic error 131 c. Rounding error 132

- i. Addition 132 - ii. Inverting a matrix 132 - iii. Solving linear equations 134 - iv. Rounding at data input 136

d. Speed 136 5.13 Exercises 137

II N E C E S S A R Y T H E O R Y 145

6 LINEARLY ( IN)DEPENDENT VECTORS 147 6.1 Linear Combinations of Vectors 147 6.2 Linear Dependence and Independence 149

a. Definitions 149 - i. Linearly dependent vectors 149 - ii. Linearly independent vectors 150

b. General properties 151 - i. Sets of vectors 151 - ii. Some coefficients zero 151 - iii. Existence and non-uniqueness of non­

zero coefficients 152 - iv. Null vectors 152

6.3 Linearly Dependent Vectors 152 a. At least two coefficients are non-zero 152 b. Vectors are linear combinations of others . . . . 153 c. Zero determinants 153 d. Inverse matrices 154 e. Testing for dependence (simple cases) 154

6.4 Linearly Independent (LIN) Vectors 156 a. Nonzero determinants and inverse matrices . . . 156 b. Linear combinations of LIN vectors 156 c. Maximum number of LIN vectors 157

6.5 Exercises 159

7 R A N K 161 7.1 Number of LIN Rows and Columns 161 7.2 Rank of a Matrix 163

xii CONTENTS

7.3 Rank and Inverse Matrices 164 7.4 Elementary Operators 165

a. Row Operations 166 b. Transposes 166 c. Column Operations 166 d. Inverses 167

7.5 Rank and the Elementary Operators 168 a. Rank 168 b. Products of elementary Operators 168 c. Equivalence 168

7.6 Calculating the Rank of a Matrix 169 a. Some special LIN vectors 169 b. Calculating rank 170

7.7 Permutation Matrices 171 7.8 Matrix Factorization 172

a. Matrices with linearly dependent columns . . . . 172 b. Full-rank factorization 173 c. Using permutation matrices 174

7.9 Results on Rank 175 7.10 Vector Spaces 175

a. Euclidean space 175 b. Vector Spaces 176 c. Spanning sets and bases 177 d. Many spaces of order n 177 e. Sub-spaces 178 f. Range and null space of a matrix 178

7.11 Exercises 179

8 CANONICAL FORMS 185 8.1 Introduction 185 8.2 Equivalent Canonical Form 185

a. Equivalent matrices 185 b. Reduction to equivalent canonical form 186

- i. Example 186 - ii. The general case 188

c. Non-uniqueness of P and Q 189 d. Existence and non-singularity of P and Q . . . . 189 e. Füll rank factorization 190

8.3 Congruent Reduction 191 a. Symmetrie matrices 191

CONTENTS xiii

b. Example 193 8.4 Quadratic Forms 194 8.5 Füll Rank Factoring a Symmetrie Matrix . . . . . . . . 195 8.6 Non-negative Definite Matrices 196

a. Diagonal elements and principal minors 196 b. Congruent canonical form 197 c. Full-rank factorization 197 d. Useful producta 197

8.7 Proofs of Results on Rank 198 8.8 Exercises 200

9 GENERALIZED INVERSES 203 9.1 Moore-Penrose Inverse 203 9.2 Generalized Inverses 204 9.3 Deriving a Generalized Inverse 205

a. Using the diagonal form 205 b. Inverting a sub-matrix 206 c. A more general inversion 207

9.4 Arbitrariness in a Generalized Inverse 207 9.5 Symmetrie Matrices 208

a. Two general results 208 b. Non-negative definite matrices 209 c. The matrix X 'X 209 d. Moore-Penrose inverses 210

9.6 Exercises 210

10 SOLVING LINEAR EQUATIONS 213 10.1 Equations Having Many Solutions 213 10.2 Consistent Equations 215

a. Definition 215 b. Existence of Solutions 215 c. Testing for consistency 216

10.3 Equations with Only One Solution 217 10.4 Solutions Using a Generalized Inverse 218

a. Obtaining one Solution 218 b. Obtaining many Solutions 219

10.5 Complete Example 221 10.6 Exercises 223

XIV CONTENTS

11 EIGENROOTSANDEIGENVECTORS 227 11.1 Basic Equation 227

a. Non-null vectors 227 b. Deriving roots 228 c. Definitions 228 d. Cayley-Hamilton theorem 229

11.2 Elementary Properties of Eigenroots 230 a. Eigenroots of powers of a matrix 230 b. Eigenroots of a scalar multiple of a matrix . . . . 230 c. Eigenroots of polynomials 231 d. Sum and product of eigenroots 231 e. Eigenroots of particular matrices 231

- i. Transposed 231 - ii. Orthogonal 231 - iii. Idempotent 231

11.3 Calculating Eigenvectors 232 11.4 Similar Canonical Form 233 11.5 Asymmetrie Matrices, Multiple Roots 235

a. Multiple roots 235 b. Vectors for multiple roots 236 c. Diagonability theorem 236

11.6 Symmetrie Matrices 237 a. Eigenroots all real 237 b. Symmetrie matrices are diagonable 238 c. Eigenvectors are orthogonal 238 d. Canonical form under orthogonal similarity . . . 238 e. Rank equals number of non-zero eigenroots . . . 239 f. Spectral decomposition 240 g. Non-negative definite (n.n.d.) matrices 240

11.7 Dominant Eigenroots 241 11.8 Singular-Value Decomposition 242 11.9 Exercises 243

12 MISCELLANEA 249 12.1 Orthogonal Matrices: A Summary 249 12.2 Idempotent Matrices: A Summary 250 12.3 Matrix al + 6J: A Summary 250 12.4 Non-negative Definite Matrices 251 12.5 Canonical Forms and Other Decompositions 252

a. Equivalent canonical form 252 b. Congruent canonical form 252

CONTENTS xv

c. Similar canonical form 252 d. Orthogonal similar canonical form 253 e. Singular-value decomposition 253 f. Spectral decomposition 253 g. The LDU and LU decompositions . 253

12.6 Matrix Functions 254 a. Functions of matrices 254 b. Matrices of functions 254

12.7 Direct Sums and Products 254 a. Direct sums 255 b. Direct products 255

12.8 Vec and Vech Operators 256 12.9 Differential Calculus 258

a. Sealars 258 b. Vectors 259 c. Quadratic forms 260 d. Inverses 260 e. Traces 261 f. Determinants 261 g. Jacobians 263 h. Hessians 265

12.10 Exercises 266

III W O R K I N G W I T H M A T R I C E S 269

13 APPLYING LINEAR EQUATIONS 271 13.1 Cost Minimization in a Firm 271 13.2 Consumer's Utility and Expenditure 274 13.3 Input-Output Analysis 276 13.4 Exercises 279

14 REGRESSION ANALYSIS 285 14.1 Simple Regression Model 285

a. Model speeification 285 b. Data 286 c. Estimation 287

14.2 Multiple Linear Regression 289 a. Themodel 289 b. The meaning of linear 290

xvi CONTENTS

14.3 Estimation 290 a. The general result 290 b. Using deviations from means 292

14.4 Statistical Model 296 14.5 Unbiasedness and Variances 296 14.6 Estimating the Variance 297 14.7 Partitioning the Total Sum of Squares 299 14.8 Multiple Correlation 302 14.9 Testing Linear Hypotheses 303

a. Stating a hypothesis 303 b. The F-statistic 304 c. Equivalent Statements of a hypothesis 305 d. Hypotheses not involving the intercept 306 e. Special cases 307

14.10 Predicting and Forecasting 308 a. The traditional predictor 308 b. Forecasting 309

14.11 Analysis of Variance 309 14.12 Confidence Intervals 312 14.13 The No-Intercept Model 312 14.14 Simultaneous Equations 314

a. Model 314 b. Identification 316 c. Methods of estimation 317

14.15 Appendix 318 14.16 Exercises 319

15 LINEAR STATISTICAL MODELS 323 15.1 General Description 323

a. Illustration (Linear model) 323 b. Model 324

15.2 Normal Equations 326 a. General form 326 b. Many Solutions 328

15.3 Solving the Normal Equations 328 a. Generalized inverses of X 'X 328 b. Solutions 329

15.4 Expected Values and Variances 330 15.5 Predicted y-Values 331 15.6 Estimating the Residual Variance 331

CONTENTS xvii

a. Error sum of Squares 331 b. Solutions obtained without using a G 333 c. Expected value 333 d. Estimation 334

15.7 Paxtitioning the Total Sum of Squares 334 15.8 Coefficient of Determination 335 15.9 Analysis of Variance 336

15.10 Estimable Functions 339 15.11 Testing Linear Hypotheses 341 15.12 Confidence Intervals 345 15.13 The Illustration Generalized 345 15.14 Appendix: Results on Quadratic Forms 348

a. Expected value 348 b. Chi-square distributions . 348 c. Independence 348 d. Independence with linear forms 348 e. J-distributions 348

15.15 Exercises 349

16 LINEAR PROGRAMMING 351 16.1 The Maximization Problem 351

a. Illustration (Profit maximization) 351 b. Matrix formulation 353 c. Graphical Solution 354 d. Extreme points 356 e. Slack variables 357 f. Basic Solution 358

16.2 The Minimization Problem 360 a. Matrix formulation 360 b. Illustration 360 c. Surplus variables 361

16.3 Simplex Method 362 16.4 Related Topics 363

a. Changing a minimization problem to a maximization problem 363

b. Mixed constraints 363 c. Changing the direction of an inequality 363 d. Unconstrained variables 364 e. The dual problem 364

16.5 Applications of Linear Programming 367

xvm CONTENTS

16.6 Exercises 370

17 MARKOV CHAIN MODELS 373 17.1 Illustration (Market Share) 373 17.2 Steady-State Probabilities 375 17.3 Tiransient States 378 17.4 Periodic or Cycling Behavior 378 17.5 Recurrent Sets 379 17.6 Existence of Steady-State Probabilities 380 17.7 Rewards in Markov Chains 381 17.8 Additional Applications of Markov Chains 384 17.9 Exercises 385

REFERENCES 389

INDEX 393