indr 262 introduction to optimization methods linear algebra indr 262 metin türkay 1
TRANSCRIPT
INDR 262INTRODUCTION TO
OPTIMIZATION METHODS
LINEAR ALGEBRA
INDR 262 Metin Türkay 1
Matrices
• m and n are positive integers• Order of matrix: mxn• The number in the ith row and jth column of A is
called the ijth element of A and is written aij.
mnmm
n
n
aaa
aaa
aaa
A
..
........
..
..
21
22221
11211
INDR 262 Metin Türkay 2
Example
7
6
1
987
654
321
31
23
11
a
a
a
A
INDR 262 Metin Türkay 3
Equal Matrices
• Two matrices A and B are equal if and only if A and B are of the same order and for all i and j, aij=bij.
• If A=B, then x=1, y=2, w=3, z=4
1 2 B
3 4
x yA
w z
INDR 262 Metin Türkay 4
Vectors
• A list of n real numbers, say (a1, a2, …, an) is called an n-dimensional vector. An n-dimensional vector also maybe displayed as a 1 by n matrix.
321 ,2
1
INDR 262 Metin Türkay 5
Scalar Product of Two Vectors
• The scalar product of vectors
u = [u1 u2 … un] and
is u1v1+u2v2+…+unvn
nv
v
v
v..2
1
INDR 262 Metin Türkay 6
Example
• uv = 1x2+2x1+3x2 = 10
2
1
2
321 vu
INDR 262 Metin Türkay 7
Notes
• If u = [1 2 3] and , then uv is
not defined because the vectors are of different dimensions.
• Two vectors are perpendicular to each other if and only if their scalar product is equal to 0.
• E.g., u = [1 -1] and .
4
3v
1
1v
INDR 262 Metin Türkay 8
Scalar Multiple of a Matrix
• Given any matrix A and any scalar c, the scalar multiple of matrix A, cA, is obtained from the matrix A by multiplying each element of A by c.
03
633
01
21AA
INDR 262 Metin Türkay 9
Addition of two Matrices
• Let A=[aij] and B=[bij] be two matrices with the same order (say mxn). Then, the matrix C=A+B is defined to be the mxn matrix whose ijth element is aij+bij.
002
000
111120
332211
112
321
110
321
BAC
BA
INDR 262 Metin Türkay 10
The Transpose of a Matrix
• Given any mxn matrix
the transpose of A (written AT) is the nxm matrix
mnmm
n
n
aaa
aaa
aaa
A
..
........
..
..
21
22221
11211
nmnn
n
n
T
aaa
aaa
aaa
A
..
........
..
..
21
22212
12111
INDR 262 Metin Türkay 11
Example
• For any matrix A, (AT)T=A.
654
321
63
52
41
654
321 T TTAAA
INDR 262 Metin Türkay 12
Matrix Multiplication
• Given two matrices A and B, the matrix product of A and B is defined if and only if the number of columns in A is equal to the number of rows in B. The matrix product C=AB is determined as follows:
cij = scalar product of (row i of A and column j of B)
INDR 262 Metin Türkay 13
Properties of Matrix Multiplication
1. Matrix multiplication is associative, i.e., A(BC)=(AB)C.
2. Matrix multiplication is distributive, i.e., A(B+C)=AB+AC.
INDR 262 Metin Türkay 14
Example
53
1 12
42
1 12
43
1 11
32
1 11
22
21
12
11
c
c
c
c
ABC
BA
32
11
12
11
54
43ABC
INDR 262 Metin Türkay 15
Matrices and Systems of Linear Equations
• Consider a system of linear equations given by
• x1, x2, …, xn are referred to as variables• aij’s and bi’s are constants• A set of equations like above is called a linear system of m
equations in n variables
nnmnmm
nn
nn
bxaxaxa
bxaxaxa
bxaxaxa
...
...........................
...
...
2211
22222121
11212111
INDR 262 Metin Türkay 16
Solution
• A solution to a linear system of m equations in n unknowns is a set of values for the unknowns that satisfy each of the systems m equations.
INDR 262 Metin Türkay 17
Example
x1 + 2x2 = 5
2x1 - x2 = 0
2
1x
1
3x
00 0 ? 2)1(2
55 5 ? )2(21
05 0 ? 1)3(2
55 5 ? )1(23
Solution
Not a solution
INDR 262 Metin Türkay 18
Matrix Representation of Systems of Linear Equations
Ax = b
nnmnmm
n
n
b
b
b
b
x
x
x
x
aaa
aaa
aaa
A...
,...
,
...
............
...
...
2
1
2
1
21
22221
11211
mmnmm
n
n
b
b
b
aaa
aaa
aaa
bA...
...
............
...
...
2
1
21
22221
11211
Augmented matrix
INDR 262 Metin Türkay 19
The Gauss-Jordan Method for Solving Systems of Linear Equations
• Gauss-Jordan method is used to find solution(s) to systems of linear equations. A system of linear equations must satisfy one of the following cases:
• Case 1: The system has no solution• Case 2: The system has a unique solution• Case 3: The system has an infinite number of solutions
INDR 262 Metin Türkay 20
Elementary Row Operations• An ero transforms a given matrix A into a new matrix A’ via one
of the following operations.• Type 1 ero: A’ is obtained by multiplying any row of A by a
nonzero scalar.• Type 2 ero: Begin by multiplying any row of A (say, row i) by a
nonzero scalar c. For some ji, let row j of A’ = c(row i of A) + row j of A, and let the other rows of A’ be the same as the rows of A.
• Type 3 ero: Interchange any two rows of A.
INDR 262 Metin Türkay 21
Facts
• If the matrix A’ is obtained from A via an ero, A’ and A are equivalent.
• If the augmented matrix [A’b’] is obtained from [Ab] via an ero, the systems Ax=b and A’x=b’ are equivalent.
• Any sequence of ero’s performed on the augmented matrix [Ab] corresponding to the system Ax=b will yield an equivalent linear system.
INDR 262 Metin Türkay 22
Gauss-Jordan Method
• The Gauss-Jordan method solves a linear system of equations by utilizing ero’s in a systematic fashion.
Step 1 To solve Ax=b, write down the augmented matrix [Ab].
Step 2 At any stage, define a current row, current column, and a current entry. Begin with row 1 as the current row, column 1 as the current column, and a11 as the current entry.
INDR 262 Metin Türkay 23
Gauss-Jordan Method
a If a11 (the current entry) is nonzero, use ero’s to transform column 1 (the current column) to [1 0 … 0]T. Then, obtain the new current row, column, and entry by moving down one row and one column to the right, and go to Step 3.
b If a11 (the current entry) equals 0, then do a Type 3 ero involving the current row and any row that contains a nonzero entry in the current column. Use ero’s to transform column 1 (the current column) to [1 0 … 0]T. Then, obtain the new current row, column, and entry by moving down one row and one column to the right, and go to Step 3.
c If there are no nonzero numbers in the first column, obtain a new current column and entry by moving one column to the right. Then go to Step 3.
INDR 262 Metin Türkay 24
Gauss-Jordan Method
Step 3a If the new current entry is nonzero, use ero’s to transform it to 1 and the
rest of the current column’s entries to 0. When finished, obtain a new current row, column, and entry. If this is impossible, stop. Otherwise, repeat Step 3.
b If the current entry is 0, do a Type 3 ero with the current row and any row that contains a nonzero entry in the current column. Then, use ero’s to transform column entry to 1 and the rest of the current column’s entries to 0. When finished, obtain the new current row, column, and entry. If this is impossible, stop. Otherwise, repeat Step 3.
c If the current column has no nonzero numbers below the current row, obtain a new current column and entry and repeat Step 3. If it is impossible, stop.
INDR 262 Metin Türkay 25
Gauss-Jordan Method
Step 4 Write down the system of equations A’x=b’ that corresponds to the matrix [A’b’] obtained when Step 3 is completed. Then, A’x=b’ will have the same set of solutions as Ax=b.
• The Gauss–Jordan method converts the augmented matrix [Ab] into [A’b’] such that
'
'2
'1
...1...00
............
0...10
0...01
''
mb
b
b
bA''
22'11 ..., , , mn bxbxbx
INDR 262 Metin Türkay 26
Example
• Solve the following system of linear equations using the Gauss-Jordan method.
2x1 + 2x2 + x3 = 9
2x1 - x2 + 2x3 = 6
x1 - x2 + 2x3 = 5
INDR 262 Metin Türkay 27
Solution
5
6
9
211
212
122
bA
5
62
9
211
2122
111
11 bA
5
32
9
211
1302
111
22 bA
21
32
9
2320
1302
111
33 bA
INDR 262 Metin Türkay 28
Solution
2112
9
2320
31102
111
44 bA
2112
7
2320
31106
501
55 bA
2512
7
6500
31106
501
66 bA
3
12
7
1003
1106
501
77 bA
INDR 262 Metin Türkay 29
Solution
3
2
1
100
010
001
99 bA
3
1
1
1003
110
001
88 bA
x1 = 1 x2 = 2 x3 = 3
INDR 262 Metin Türkay 30
Analysis of the Solutions to Systems of Linear Equations
• For any linear system, a variable that appears with a coefficient of 1 in a single equation and a coefficient of 0 in all other equations is called a basic variable. Any variable that is not a basic variable is called a nonbasic variable.
x set of all of the variables in the system Ax=bxB set of all of the basic variables in the system Ax=bxN set of all of the nonbasic variables in the system
Ax=bx = xB xN
INDR 262 Metin Türkay 31
Analysis of the Solutions to Systems of Linear Equations
• The solution to A’x=b’ can be categorized in one of the three cases:
Case 1: A’x=b’ has at least one row of the form [0 0 … 0c] and c0. Then, Ax=b has no solution.
Case 2: When case 1 does not apply and xN=, then Ax=b has a unique solution.
Case 3: When case 1 does not apply and xN, then Ax=b has infinite number of solutions.
INDR 262 Metin Türkay 32
Example
• Case 1 does not apply since there are no rows of the form [0 0 … 0c] and c0.
• Case 2 does not apply since,
xB={x1, x2, x3}
xN={x4, x5}• There are infinite number of solutions.
0
1
2
3
00000
10100
02010
11001
'' bA
INDR 262 Metin Türkay 33
Example
• Assign arbitrary values to the variables in xN; x4=c, x5=k.• Write down the equations in [A’b’],
x1 + c + k = 3 x1 = 3 - c – kx2 + 2c = 2 x2 = 2 - 2cx3 + k = 1 x3 = 1 - k
• It is easy to see that there are infinite number of values of c and k that will satisfy this system of equations.
0
1
2
3
00000
10100
02010
11001
'' bA
INDR 262 Metin Türkay 34
Linear Combination
• A linear combination of the vectors in V is any vector of the form c1v1+c2v2+…+ckvk, where c1, c2, …, ck are arbitrary scalars.
• Example: V={[1,2], [2,1]}
2v1-v2 = 2([1 2]) – [2 1] = [0 3]
0v1-3v2 = 0([1 2]) – 3([2 1]) = [6 3]
INDR 262 Metin Türkay 35
Linear Independence & Linear Dependence
• A set V of m-dimensional vectors is linearly independent if the only linear combination of vectors in V that equals 0 is the trivial linear combination.
• A set V of m-dimensional vectors is linearly dependent if there is a nontrivial linear combination of the vectors in V that adds up to 0.
INDR 262 Metin Türkay 36
Example 1
• V={[1,0], [0,1]} Try to find a linear combination of vectors in V that yields 0.
• c1([1 0]) + c2([0 1]) = [0 0]
• In order to satisfy this, [c1 c2] = [0 0] c1=c2=0
• The only linear combination of vectors in V that yields 0 is the trivial linear combination. Therefore, V is a linearly independent set of vectors.
INDR 262 Metin Türkay 37
Example 2• V={[1,2], [2,4]} Try to find a linear combination of
vectors in V that yields 0.• c1([1 2]) + c2([2 4]) = [0 0]
[c1 2c1] + [2c2 4c2] = [0 0]c1 + 2c2 = 0 c1 = -2c2
2c1 + 4c2 = 0 2c1 = -4c2
• So, c1 = 2 c2 = -1 is one of the possible solutions.• There exists a nontrivial linear combination of vectors in
V that yields 0. Therefore, V is a linearly dependent set of vectors.
INDR 262 Metin Türkay 38
The Rank of a Matrix
• Let A be any mxn matrix, and denote the rows of A by r1, r2, …, rm. Also define R={ r1, r2, …, rm}.
• The rank of A is the number of vectors in the largest linearly independent subset of R.
• If for a matrix A with m rows, rank A=m; then the matrix is a collection of linearly independent set of vectors. If rank A<m; then the matrix contains a linearly dependent set of vectors.
INDR 262 Metin Türkay 39
Examples
3202
110
001
320
120
001
A
1002
110
001
100
010
001
Rank A = 3
011
010
001
rank B Rank B = 2
INDR 262 Metin Türkay 40
The Inverse of a Matrix• A single linear equation in a single variable can be solved
by multiplying both sides of the equation by multiplicative inverse of the variable coefficient.
• Example:4x=3 4-1(4x) = (4-1)3 x=3/4
• We can generalize this approach to square systems of linear equations (i.e., number of equations = number of unknowns).
INDR 262 Metin Türkay 41
Square and Identity Matrix
• A square matrix is any matrix that has an equal number of rows and columns.
• The diagonal elements of a square matrix are those elements aij such that i=j.
• A square matrix for which all diagonal elements are equal to 1 and all non-diagonal elements are equal to 0 is called an identity matrix.
100
010
001
, 10
0132 II
INDR 262 Metin Türkay 42
Inverse of A
• For a given mxm matrix A, the mxm matrix B is the inverse of A if
BA = AB = Im
100
010
001
201
715
101
101
213
102
AA-1=I
INDR 262 Metin Türkay 43
Finding the Inverse of a Matrix with the Gauss-Jordan Method
Step 1 Write down the mx2m matrix [AIm].
Step 2 Use ero’s to transform [AIm] into [ImB]. This will only be possible if rank A=m; in this case, B=A-1. If rank A<m, then A has no inverse.
INDR 262 Metin Türkay 44
Example
31
52A
10
01
31
522IA
10
021
312
5112IA
12
1
021
210
251
22IA
21
021
102
5132IA
21
53
10
0142IA
21
531A
INDR 262 Metin Türkay 45
Using Matrix Inverse to Solve Linear Systems of Equations
• Given a linear system of equations,
Ax=b• Multiply both sides by A-1
AA-1x = A-1b x = A-1b
INDR 262 Metin Türkay 46
Example
2x1 + 5x2 = 7
x1 + 3x2 = 4
4
7
31
52
2
1
x
x
21
53
31
52 1-AA
1
1
4
7
21
53
2
1
2
1
x
x
x
x
INDR 262 Metin Türkay 47
Determinant of a Matrix
• Any square matrix A has a number called the determinant of A (shown by set A or A).
122122112221
1211 det aaaaAaa
aaA
INDR 262 Metin Türkay 48
Minor of A
• If A is an mxm matrix; then for any values of i and j, the ijth minor of A (Aij) is the (m-1)x(m-1) submatrix of A obtained by deleting row i and column j of A.
12
11 ,
20
12
210
112
101
3212 AAA
INDR 262 Metin Türkay 49
Determinant of a Matrix
• Let A be any mxm matrix,
• det A = (-1)i+1ai1(det Ai1) + (-1)i+2ai2(det Ai2) + … +(-1)i+maim(det Aim)
mmmm
m
m
aaa
aaa
aaa
A
...
............
...
...
21
22221
11211
INDR 262 Metin Türkay 50
Example 1
210
112
101
A
1 1 1 2 1 31 1 2 1 2 1det ( 1) 1 det ( 1) 0 det ( 1) 1 det
1 2 0 2 0 1
det 1x1x 3 (-1)x0x4 1x1x2
det 5
A
A
A
det A = (-1)1+1 1 det A11 + (-1)1+2 0 det A12 + (-1)i+3 1 det Ai3
INDR 262 Metin Türkay 51
Example 2
5121
2122
1113
1201
A
det A = -48
INDR 262 Metin Türkay 52