Communication Signal Design Lab

Lecture Notes on Advanced Linear Algebra

Lecture #1

Simultaneous Linear Equations andGauss Elimination

Hong-Yeop Song

School of Electrical and Electronic Engineering

hysong@Yonsei.ac.kr

Topics

Introduction to this class “Linear Algebra” by Professor Hong-Yeop Song

Very basics of Chapter 1:

Simultaneous Linear Equations and Gauss Elimination

LDU decomposition/factorization

HW#1 Due next class (10 Problems)

Problems are in this lecture note

2

INTRODUCTION TO THE CLASS

Time 1 of Week 1

Course Basic

4

Room 공B004 Time/Place

Professor송홍엽

Song, Hong-YeopURL http://coding.yonsei.ac.kr/~hysong

Office B615 (제2공학관) Cell Phone 010-7661-4861

Email,

office-hourhysong@yonsei.ac.kr office-hour :

일반대학원 EEE7131-01 (고급 선형대수 및 응용)

• Textbook: Linear Algebra and its applications (G. Strang) 4th edition 학생회관 구내서점 구입가능. 반드시 구입요망. 이미 가지고 있다면 버전확인 필수 (HW 문제 번호 일치 여부 때문에).

• 선수과목: 아래과목을이수하지않았다면본과목수강에심각한지장있음 학부 “공업수학”의 선형대수 부분, 혹은, 학부수준의 “선형대수학” 위 조건을 만족하지 않고 수강하려면 이 시간 이후 즉시 나와 면담하기 바람

통신/신호처리/네트워크/제어 그리고 컴퓨터공학 전분야에서 필수 도구로 사용함. 위의 전공과목을 잘 생각해보면 거의 대부분이 선형대수학의 기본 이론을 사용하여 진행됨. 예를 들자면,

모든 비선형 방정식은 선형화를 거쳐서 “선형연립방정식”의 형태로 모델링

모든 실계수 미분방정식은 “선형연립방정식”으로 모델링

모든 (통신/압축/제어/랜덤) 신호처리 과정은 “선형변환”으로 모델링

관련된 “행렬”의 성질을 파악하고 이를 “변형(대각화)”하여 처리

수학적 사고방식 (증명 및 구조체 분석)배우기에 매우 적합한 초보적인 내용이며, 이러한 내용은 전공분야 연구활동 (논문 읽기, 논문 쓰기)에 필수적.

“제대로” 연구 해보겠다는 각오의 결과로 본 과목을 선택했으니, “제대로” 이 과목을 공부하겠다는 결심을 하기 바람

예습/복습(HW문제풀이)는 물론이고, 교재와 참고문헌 적어도 한 권씩 동시에 공부하기를추천함

선형대수를 왜 공부해야 하는가?

5

선형대수를 왜 공부해야 하는가?

송홍엽 교수의 강의로

완벽한 수업계획서 추진 중

YES ^^ I will try...

well..... we have other opinion on this...

other opinion !!

선형대수를 왜 공부해야 하는가?

7

송홍엽 교수의 강의로

???

• 독자적인 해석방법 및 증명

• 특히 증명의 중요성과 증명하는 다양한 방법

주당 4시간

주당 3시간

강의시간을 줄이면서강의방법을 판서에서

PPT로 바꾸었기때문에

학업량은오히려증가됨ㅠ.ㅠ

개선점? 좋은점?

(송홍엽의) 선형대수 과목의 특성 기본적인 초급 선형대수 개념을 모두 포함하며, 항상 example 위주의 매우 쉬운

설명으로부터 시작하여 수학적 감각을 느낄 수 있는 일반화와 완벽한 증명까지

도출함.

단순 “예”만 설명하는 것도 아니며, 딱딱한 “증명”만 강조하지도 않으려고 노력.

수강생은 수업 중 설명에 집중한다면 이 두 마리의 토끼를 모두 잡을 수 있음.

많은 경우에 scalar field를 finite field로 사용하여 대개의 example에서 다양한

구조(structure)의 모양을 손에 잡히게 보여줌. 또한, 정수론(number theory),

조합론(combinatorial mathematics) 등의 이산수학(discrete mathematics)의

개념과 현대대수 혹은 추상대수학(abstract algebra)의 일부 내용까지도 사용하

여 선형대수의 기본이론을 폭 넓게 그러나 구체적으로 설명함.

finite field, number theory, combinatorial mathematics, abstract algebra 등의 일부 개념이

강의 내용에 자주 등장함

초급 선형대수학과 다양한 응용수학 분야를 폭넓게 다루므로, 때때로 참고문헌 찾기가 어렵지만,

수업내용에 집중한다면 모두 해결되도록 설명함.

공학적 문제풀이에 자주 등장하는 행렬의 특성과 분해(decomposition,

factorization)에 초점이 맞추어짐.

다양한 공학적 응용 분야를 소개함. (ambitiously ??)

8

Communication Signal Design Lab

Week

number

Calendar

(lecture is on Friday)

Lecture

numberTopic (2015.8.17 버전)

HW

(due: 1 week)Semester Calender 보강 계획

01 1,2,3,4,5,6,7 1 Chapter 1 Matrices and Gauss Elimination HW #01(9.1)개강(9.2-9.4) 수강신청확인변경

02 8,9,10,11,12,13,14 2 Chapter 2 Vector Spaces – part 1 HW #02

03 15,16,17,18,19,20,21 3 Chapter 2 Vector Spaces – part 2 HW #03 (9.18-9.19) 정기연고전(9.14-9.19) 해외출장.

보강예정 (9.12토 ??)

04 22,23,24,25,26,27,28 4 Theory of permutations and Determinant HW #04 (9.26-9.28) 추석연휴

05 29,30,1,2,3,4,5 5 Chapter 3 Orthogonality – part 1 HW #05 (9.29) 대체공휴일

06 6,7,8,9,10,11,12 - discussions on HW problems(10.6-10.8) 수강철회(10.8) 학기 1/3선

(10.9) 한글날.

보강예정(10.10 토 ??)

07 13,14,15,16,17,18,19 - Midterm Exam (3 hours) (10.16-10.22) 중간시험(10.12-10.16) 출장.

중간시험기간중 3시간

08 20,21,22,23,24,25,26 - discussions on midterm exam (10.16-10.22) 중간시험

09 27,28,29,30,31,1,2 6 Chapter 3 Orthogonality – part 2 HW #06

10 3,4,5,6,7,8,9 7 Chapter 5 Eigenvalues and Eigenvectors – part 1 HW #07

11 10,11,12,13,14,15,16 8 Chapter 5 Eigenvalues and Eigenvectors – part 2 HW #08 (11.16) 학기 2/3 선

12 17,18,19,20,21,22,23 9 Chapter 6 Positive Definite Matrices and SVDProject

Announcement

HW #09

13 24,25,26,27,28,29,30 10 Some Extra Topics HW #10

14 1,2,3,4,5,6,7 - discussions on HW problems

15 8,9,10,11,12,13,14 - Self Study Week - Final Exam (3 hours) (12.8-12.21) 자율학습(12.11) 기말시험

자율학습기간중 3시간

16 15,16,17,18,19,20,21 - no class Project Report Due (12.8-12.21) 기말시험

The Plan (plan-A)

• Week starts from Tuesday to the next Monday.• we may have plan-B in the middle of the semester, according to your performance.

10

Lecture

numberTopic (2015.8.17 버전) The Plan (plan-A): Details HW

(due: 1 week)

1 Chapter 1 Matrices and Gauss Elimination Introduction to the class Simultaneous Linear Equations and Gauss Elimination LDU decomposition/factorization

HW #01

2 Chapter 2 Vector Spaces – part 1 Introduction to (scalar) fields Introduction to vector spaces over a field Four fundamental subspaces of an m x n matrix

HW #02

3 Chapter 2 Vector Spaces – part 2 Introduction to Linear Transformations Linear Transformations and Matrices Quotient Space and First Isomorphism Theorem

HW #03

4 Theory of permutations and Determinant Some theory of permutations Some theory of determinant

HW #04

5 Chapter 3 Orthogonality – part 1 Dual Space and Expansion Formula Inner Product Space and Projection Projection and Least Square

HW #05

- discussions on HW problems

- Midterm Exam (3 hours) Midterm Exam (3 hours)

- discussions on midterm exam

6 Chapter 3 Orthogonality – part 2 Orthonormal Basis and Gram-Schmidt Process, QR decomposition Orthogonal Subspaces

HW #06

7 Chapter 5 Eigenvalues and Eigenvectors–part 1 Introduction to eigenvectors and eigenvalues Some properties of eigenvectors and eigenvalues Linear difference equations, Markov Matrices and Spectral Theorem

HW #07

8 Chapter 5 Eigenvalues and Eigenvectors–part 2 matrix exponentials and linear differential equations Complex matrices and Similar transformation Schur’s Lemma on Triangulization and Normal Matrices

HW #08

9 Chapter 6 Positive Definite Matrices and SVD Positive definite matrices Congruence transformation SVD

HW #09

10 Some Extra Topics Jordan Canonical Form Circulant matrices cyclic-type Hadamard matrices

HW #10

- discussions on HW problems

- Self Study Week - Final Exam (3 hours) Self Study Week - Final Exam (3 hours)

- no class Project Report Due

출석확인점수(최대 100점): 강의13주x3시간 = 39회, 각각 3점씩, 총점 117점. 매주 금요일 3시간 수업에서 3회의 출석을 확인함

Midterm exam: 200점 (중간시험 기간)

Final exam: 200점 (기말시험 기간)

Project Report: 100점 (13주차공지 16주차 마감 )

HW 풀이 제출: 300점=30점x10회 (회당 대략 10문제 수준) 대략 매주 공지되며, 공지 후 제출마감은 다음수업시간시작직전.

마감 이후 1일 후부터 1주일 이내 제출시 40% 삭감. 그 이후는 받지 않습니다.

지각으로 인하여 당일 제출이지만 마감시간이 지나는 경우 10% 삭감.

총점: 900점. 절대평가.

650점 이상 (72%이상): A+,A,A-

450점 이상 (50%이상): B+,B,B-

450점 미만 (50%미만): C or F

성적과 관계없는 F학점 조건

무단결석 2회 –사유가 있는 결석은 무단결석이 아님. 사유서 제출 필수.

수업중핸드폰 혹은 컴퓨터 조작

시험 중부정행위

보고서(HW문제풀이) 작성 중부정행위

Evaluation Mechanics

11

수강생 유의사항(중요)

12

모든문의사항은이메일 hysong@yonsei.ac.kr 혹은핸드폰 010-7661-4861 문자로보내세요.

• 이메일을 보낼 때는 제목에 반드시 " (과목명:선형대수)홍길동 -문의내용제목"을 사용하고 메시지의 마지막 부분에자신의이름과날

짜를 기록하기 바랍니다. 문자를 보낼 때는과목명과이름을 반드시 기록하세요.

OFFICE-HOUR란수강생여러분을위한시간입니다. 이시간에는내가방문을열어놓고여러분을기다리니 (제2공학관 615호) 그저

노크하고들어오면됩니다.

• 따로 방문예약이나 이메일 보내지 말고 그냥 오세요. 이 시간에는 항상 환영합니다. 수업내용이나 기타 모든 면에서 나와 이야기 하고

싶다면 이 시간을 이용하세요. 특정한 질문이 없이 그냥 방문해도 환영합니다. 내 방에서 차 한잔 같이 해요.

• OFFICE-HOUR 이외의시간에는가급적나를찾지마세요^^ 나를 만나고 싶다면 반드시 예약하기 바랍니다.

무단결석 2회면 F학점입니다.

• 무단결석을 피하려면 반드시 전날까지 이메일 혹은 문자메시지로 (010-7661-4861) 결석함을 통지하기 바랍니다. 긴급상황을 제외하

고 당일 연락하는 결석계는 인정하지 않습니다.

• 결석 후 처음 출석하는 수업시간에 결석계를 자필 서명하여 제출하세요. 결석계에는 결석 사유를 자세히 설명하기 바랍니다. 입원기록,

진료기록,예비군훈련통보, 입사를위한면접통지등의공식서류가있다면첨부하세요. 사적인 사유(피곤함/늦잠 등)으로 결석하게 되면

그 사유를 잘 설명하세요.

연세대학교는부정행위에대해매우심각한학칙을적용하고있습니다.

• 시험시간 중 일체의 부정행위를 인정하지 않습니다. 별도의 확인절차 없이 부정행위 가담자 전원 F학점 처리합니다.

• 제출 전 HW 문제에 관한 토론은 금지이며, 동일한 풀이가 확인되면 관련자 전원 F학점 처리합니다. 다만, 제출 이후 토론은 적극 권

장합니다.

수강생 유의사항(추천)

13

담당교수의 OFFICE HOUR를최대한활용하세요. 비싼등록금을지불하는여러분의권리입니다.^^

• 강의내용에대한질문과해결이나이미제출한 HW 문제에대한질문과해결을위해서담당교수를이용한다면가장현명한공부방법이될것입니다.

• 그이외의어떠한이야기도가능합니다. 편하게방문하기바랍니다.

모든수업은처음 5분이매우중요합니다. 이부분을놓치지마세요.^^

• 수업시간에 지각하지 마세요. 적어도 10분전까지 도착하겠다는 계획을 세우세요.

• 지각하는경우 HW 제출이불가능하여감점대상이됩니다.

수업시간중일체의질문을환영합니다.조금이라도궁금하다면언제고맘편하게질문하세요. 한사람의 (멋진혹은엉뚱한^^) 질문이전체수강생의수업효과를높입니다.

• 수업시간에는 핸드폰이나 컴퓨터의 사용을 금지합니다. 반드시 turn-off 하기 바랍니다. 소리가 울리거나 컴퓨터를 조작하다가 지적 받으면 곧바로 F학점입니다.

매주수행해야할숙제에최선을다하세요. 이모든과정과노력이실질적인시험준비가됩니다.

• 제출 전 HW에 관한 토론은 금지이지만 제출 후 토론은 적극 권장합니다. 제출후에는각자해결하지못했던문제에대해서서로의견나누기바랍니다.

• HW문제는 강의내용을 복습하기 위한 목적이며,시험문제는얼마나많이연습했는지를측정합니다.

본수업은 G. Strang 교수의MIT Open Course Ware와연동하여진행합니다. 인터넷에서검색하여참조하기바랍니다.

• 해당 주제의 내용을 다루는 Strang 교수님의동영상강의를미리듣고본수업에참여한다면수업효과를 극대화 할 수 있습니다.

Initial HW – due: week 02 class

14

대학원생: JPG file of your face + 자기 소개의 글 + 지도교수님 성함

“선형대수”에 대한 자신의 수준을 간략히 설명

학부생: JPG file of your face + 자기 소개의 글

“선형대수”에 대한 자신의 수준을 간략히 설명

email: hysong@yonsei.ac.kr

subject: 선형대수 수강생인사 – 홍길동 제목 줄 형식 주의: 이를 지키지 않으면 내가 이메일박스에서 찾지 못합니다.

Use A4 size paper and only one-side. (이면지 사용가능)

Staple once at the top-left corner.

Put your name and Student id number at the top of the first page.

Write down in your own handwriting the following:

이보고서를작성하는과정에어떤부정행위도하지않았습니다. (서명) 날짜.

I promise that this work was done only by myself. and SIGN, and put DATE

위 형식을 지키지 않으면 10% 삭감합니다.

Due time is always the BEGINNING OF CLASS of the due day.

Late submission rule: Due time 이후 당일 제출, 수업 이후 2시간 이내 (지각 혹은 망각 등의 사유) – 10% 삭감

Due time 이후 그 다음날부터 1주일 이내 – 40%삭감

그 이후는 받지 않습니다.

Preparation of all your HW report

15

HW#1 Problem#0 (DO NOT SUBMIT)

Given a 2x2 matrix 𝐴 = 𝑎 𝑏𝑐 𝑑

, answer the following questions.

Assume that 𝑎 = 2, 𝑏 = 1, 𝑐 = 1, 𝑑 = 2 are reals.

1. Calculate 𝐴2.

2. Find the determinant and rank of 𝐴.

3. Find all the solutions 𝑥 =𝑥𝑦 to the equation 𝐴𝑥 =

10

.

4. Find its eigenvectors and corresponding eigenvalues

5. Find ANY factorization of 𝐴 into the product of two matrices 𝐵𝐶, where none of 𝐵 and 𝐶 is the identity matrix

6. Is it positive definite? Verify your answer.

7. Is it normal? Verify your answer.

Now, assume that 𝑎, 𝑏, 𝑐, 𝑑 are integers mod 3, denoted by {0,1,2}.

8. Calculate 𝐴2 .

9. Find the determinant and rank of 𝐴.

10. Find all the solutions 𝑥 =𝑥𝑦 to the equation 𝐴𝑥 =

10

.

16

SIMULTANEOUS LINEAR EQUATIONS AND

GAUSS ELIMINATION

Time 2 of week 1

17

a Matrix of size 𝑚 × 𝑛 over 𝑭

18

components are members of 𝑭

field or ring or ...

1 2 3 ....... j ..... ............ n23...i...

m

𝑎𝑖,𝑗= A = (𝑎𝑖,𝑗)

𝑎𝑖,𝑗 = (𝑖, 𝑗)-component of A

𝑹 – real numbers

𝑄 – rational numbers

𝐶 – complex numbers

𝑍 – integers (not field)

𝑍𝑛 - integers mod 𝑛

𝑍𝑝 - integers mod 𝑝

𝐹2 - binary numbers

etc...

Matrix addition

Let A and B be two matrices over the same F

A+B is possible if both A and B have the same size.

A+B has the same size as A and B.

We say

A+B=C = (cij)

where cij=aij+bij for all i and j

19

addition of matrices A and B

addition defined on F

Matrix multiplication

Let A and B be two matrices over the same F

The multiplication AB (or A·B) is possible if the number of columns of A is the same as the number of rows of B.

We say

AB=C = (cij)

where cij=ai1b1j+ai2b2j+...+ainbnj

or

𝑐𝑖𝑗 =

𝑙=1

𝑛

𝑎𝑖𝑙𝑏𝑙𝑗

when n = number of columns of A

= number of rows of B

20

multiplication of matrices A and B

addition defined on F

multiplication defined on F

Communication Signal Design Lab

1 2 3 ....... ... ..... ............ n23...i...

m

1 2 3 ....... j ..... ........ k23...

...

n

A B

=

1 2 3 ....... j ... ..... k23...i...

m

C

𝑚 × 𝒏 𝒏 × 𝑘 𝑚 × 𝑘

cij

cij = ai1b1j + ai2b2j + ... + ainbnj

= “dot product” of i-th row of A and j-th column of B

Multiplication by a constant

22

1 2 3 ....... j ..... ............ n23...i...

m

𝑎𝑖,𝑗A = (𝑎𝑖,𝑗) =

1 2 3 ....... j ..... ............ n23...i...

m

𝑐𝑎𝑖,𝑗cA = c(𝑎𝑖,𝑗) = (c𝑎𝑖,𝑗) =

Simultaneous Linear Equations2 x 2 example

Consider

2x - y = 1

x + y = 5

3x = 6

or x=2

substitute back to ②: y=3

23

①

②+

yes. x=2 and y=3 is a solution

Row view

Column view

Two views of S.L.E.

Row view Rows are lines

Solutions (if any) are the intersections of these lines(rows)

Column view Columns are weighted and summed

Solutions (if any) are the appropriate weights of the columns

24

Row View

25

Rows are lines

Solutions (if any) are the intersections of these lines(rows)

2x - y = 1

x + y = 5

y = 2x-1

y = -x+5

5

-1

solution (x=2, y=3)

2

3

0

x

yy = 2x-1

y = -x+5

in general,(1) no solution

when they are parallel(2) many solutions

when they coincide(3) unique solution

in all other cases

THEOREM 1.

Column View

Columns are weighted and summed

Solutions (if any) are the appropriate weights of the columns

2x - y = 1

x + y = 5

⇔ x 21

+ y −11

= 15

26

0 x

y

-1 2

121

−11

221

3−11

15

2 21

+ 3 −11

= 15

solution !!

ax + by = c

column vectors

• no solution if c is not a linear combination of a and b

• at least one solution if c belongs to SPAN of a and b

many solution ↔ a, b and c are on the same line

unique solution↔ otherwise

Communication Signal Design Lab

ax + by = c

column vectors

• no solution if c is not a linear combination of a and b

• at least one solution if c belongs to SPAN of a and b

many solution ↔ a, b and c are on the same line

unique solution↔ otherwise

0 x

y

-1 2

121

=b

11/2

=a

15

=c is NOT a linear combination of

a and b

0 x

y

-1 2

121

=b

11/2

=a

42

=c

↔ a and b are on the same line, but c is not on this line

THEOREM 2.

Gauss Elimination

Known best algorithm of solving an SLE.

Complexity is in the order of n3 where n is the number of equations (or unknowns)

will go through ONLY an example of a square matrix of size n x n

28

Example of GE - HSA approach2u + v + w = 5 ---- ①4u - 6v = -2 ----②

-2u + 7v + 2w = 9 ----③

29

copy ① : 2u + v + w = 5 ---- ①’② + ①×(-2): - 8v -2w = -12 ----②’

③ + ①×(+1): 8v +3w = 14 ----③’

copy ①’ : 2u + v + w = 5 ---- ①’’copy ②’ : - 8v -2w = -12 ----②’’

③’ + ②’×(+1): +w = 2 ----③’’

Substitute w=2 back to ②’’: -8 v - 4 = -12 or v = 1

Substitute w=2 and v=1 back to ①’’: 2u +1 +2 = 5 or u = 1

solution !!

Augmented Matrix Approach

30

2u + v + w = 5 ---- ①4u - 6v = -2 ----②

-2u + 7v + 2w = 9 ----③

2 1 1 54 -6 0 -2-2 7 2 9

2 1 1 50 -8 -2 -120 8 3 14

2 1 0 30 -8 0 -80 0 1 2

2 1 0 30 1 0 10 0 1 2

2 0 0 20 1 0 10 0 1 2

1 0 0 10 1 0 10 0 1 2

Gau

ss E

lim

inat

ion

Gau

ss-J

ordan

Elim

inat

ion

solution!

2 1 1 50 -8 -2 -120 0 1 2

Computational Complexity

ignore the operations on RHS

ignore permutations of rows/columns

only count “multiply-add”

31

n

n

n(n-1)(n-1)(n-2)

...

2·1

~ n3

Matrix Notation of S.L.E.

32

2u + v + w = 5

4u - 6v = -2

-2u + 7v + 2w = 9

2 1 14 −6 0−2 7 2

𝑢𝑣𝑤=5−29

𝐴𝑥 = 𝑏

row view

column view

24−2𝑢 +1−67𝑣 +102𝑤 =5−20

Matrix Notation

Inverse of a square matrix

Definition Let 𝐴 be an 𝑛 × 𝑛matrix over a field 𝑭

An 𝑛 × 𝑛matrix 𝐵 over 𝑭 is called the inverse of 𝐴 if

𝐵𝐴 = 𝐼 = 𝐴𝐵and it is denoted by 𝐴−1

When the matrix A has an inverse, it is called “non-singular.”

when 𝑛 = 4 𝐼 =

1 00 1

0 00 0

0 00 0

1 00 1

33

identity matrix

additive identity of 𝐹

multiplicative identity of 𝐹

will study the formulafor the inverse of a non-singular matrix in Lecture #4

Remark on GJ Elimination

THEOREM 3: Given SLE Ax=b, we form (A b) of size n x (n+1). Apply GJE on the part A and arrive at (I c).

Then x=c is the solution of the SLE Ax=b.

THEOREM 4: Given A, we form (A I) of size n x 2n. Apply GJE on the part A and arrive at (I B).

Then B is the inverse of A.

Remark: The matrix A will be singular if A does not have an inverse, or equivalently, (I B) can not be reached by GJE from (A I).

GJE is in general not recommended since it is more computationally loaded than GE with back-substitution.

34

A=LU by Gauss Elimination (next hour class)

Ax=b → LUx=b

→ Ux=L-1b=c and back-substitute

HW#1 Problems #1,#2,#3(#1) Find all the solutions 𝐴𝑥 = 𝑏 when

𝐴 =1 4 04 1 40 4 1

and 𝑏 =110

using

(1) Gauss elimination and back-substitution

(2) Gauss-Jordan elimination

and interpret the situation in row-view

as well as in column view

(#2) Repeat above using the following:

𝐴 =2 1 01 2 00 0 1

and 𝑏 =122

(#3) Repeat above using the following:

𝐴 =2 4 01 2 00 0 1

and 𝑏 =122

35

HW#1 Problems #4

Apply GJ elimination to 3 x 6 augmented matrix of the following 3x3 matrices to find its inverse if it exists, and verify your result.

𝐴 =2 4 14 8 00 4 0

hint: Apply GJ elimination to the following 3 x 6 matrix2 4 14 8 00 4 0

1 0 00 1 00 0 1

and obtain the following result1 0 00 1 00 0 1

Then 𝐵 is the solution.

36

𝐵

summary Field and Ring

Matrix over a field F

Operations of Matrices addition and multiplication

distinguish those over the field F

Simultaneous Linear Equations row view and column view

span, linear combination

number of solutions

matrix notation

Gauss Elimination via Augmented matrix computational complexity

Gauss-Jordan Elimination

37

LDU DECOMPOSITION/FACTORIZATION

Time 3 of week 1

38

LDU Factorization

can be done for ANY matrix of size m x n

another view of Gauss Elimination One step of G.E. can be performed by multiplying an

appropriate matrix to the left of both sides of Ax=b

All the steps of G.E. can also be viewed as the same as multiplying an appropriate matrix to the left of both sides of Ax=b

First step is to find LU decomposition, where L=lower triangle matrix, and U=upper triangle matrix

So far, we have seen the following ways of solving the S.L.E.

Gauss Elimination and Back-Substitution

Augmented matrix approach of G.E.

Gauss-Jordan Elimination

LDU decomposition(factorization) – will study now.

39

SAMEprocesses

One step of G.E. can be performed by multiplying an appropriate matrix to the left of A

40

2u + v + w = 5 ---- ①4u - 6v = -2 ----②

-2u + 7v + 2w = 9 ----③

copy ① : 2u + v + w = 5 ---- ①’② + ①×(-2): - 8v -2w = -12 ----②’

③ + ①×(+1): 8v +3w = 14 ----③’

2 1 14 −6 0−2 7 2

𝑢𝑣𝑤=5−29

𝐸21𝐴 =1 0 0−𝟐 1 00 0 1

2 1 14 −6 0−2 7 2

=2 1 10 −8 −2−2 7 2

𝐸31𝐸21𝐴 =1 0 00 1 0+𝟏 0 1

2 1 10 −8 −2−2 7 2

=2 1 10 −8 −20 8 3

Can we do this in one step?

yes we can.

Communication Signal Design Lab

1 0 0−𝟐 1 00 0 1

2 1 14 −6 0−2 7 2

=2 1 10 −8 −2−2 7 2

1 0 00 1 0+𝟏 0 1

2 1 10 −8 −2−2 7 2

=2 1 10 −8 −20 8 3

1 0 0−𝟐 1 0+1 0 1

2 1 14 −6 0−2 7 2

=2 1 10 −8 −20 8 3

in fact,

𝐸31𝐸21 =1 0 00 1 0+𝟏 0 1

1 0 0−𝟐 1 00 0 1

=1 0 0−𝟐 1 0+1 0 1

≜ 𝐸1

Communication Signal Design Lab

copy ① : 2u + v + w = 5 ---- ①’② + ①×(-2): - 8v -2w = -12 ----②’

③ + ①×(+1): 8v +3w = 14 ----③’

copy ①’ : 2u + v + w = 5 ---- ①’’copy ②’ : - 8v -2w = -12 ----②’’

③’ + ②’×(+1): +1w = 2 ----③’’

𝐸32 𝐸31𝐸21ܣ =1 0 0

0 1 0

0 +𝟏 1

2 1 1

0 −8 −2

0 8 3

=

2 1 1

0 −8 −2

0 0 1

≜ 𝑈

upper triangle

Next step

Communication Signal Design Lab

so far we have,

1 0 00 1 00 +𝟏 1

1 0 0−𝟐 1 0+1 0 1

2 1 14 −6 0−2 7 2

=1 0 0−𝟐 1 0−𝟏 +𝟏 1

2 1 14 −6 0−2 7 2

=2 1 10 −8 −20 0 1

= 𝑈

or,

𝐸32𝐸31𝐸21ܣ = 𝑈

Two questions:

1. How about ܣ = 𝐸32𝐸31𝐸21

−1 𝑈 = 𝐿𝑈 ?

looks VERY good!

2. How to find𝐸32𝐸31𝐸21

−1 = 𝐿in simple form ?

???

lower triangle?

upper triangle

Inverse of product of A and B

44

If a square matrix A has an inverse,

it is called “non-singular”

Theorem 5.If 𝐴 and 𝐵 are non-singular,

then

𝐴𝐵 −1 = 𝐵−1𝐴−1

Proof𝐵−1𝐴−1𝐴𝐵= 𝐵−1 𝐴−1𝐴 𝐵= 𝐵−1 𝐼 𝐵= 𝐵−1𝐵= 𝐼

𝐿 = 𝐸32𝐸31𝐸21−1 = 𝐸21

−1𝐸31−1𝐸32

−1

45

𝐸𝑖𝑗 =𝑖

𝑗

11

1

1

a

𝑖 > 𝑗 and 𝑎 ≠ 0

𝐸𝑖𝑗−1 =?

try

𝐸21 =1 0 0−𝟐 1 00 0 1

and 𝐸21−1 =

1 0 0+𝟐 1 00 0 1

check:

1 0 0−𝟐 1 00 0 1

1 0 0+𝟐 1 00 0 1

=1 0 0𝟎 1 00 0 1

a

1

Theorem 6

46

𝐸𝑖𝑗 =𝑖

𝑗

11

1

1

a

𝑖 > 𝑗 and 𝑎 ≠ 0

𝐸𝑖𝑗−1 =

𝑖

𝑗

11

1

1

- a

Proof

Consider the multiplication 𝐸𝑖𝑗𝐸𝑖𝑗−1

Every row of 𝐸𝑖𝑗−1 will not be changed except for i-th row

i-th row will become:

0 0 0 0 0 0 0 1 0 01 2 3 ................. j-1 j j+1 i-1 i i+1 n

=a-a only here!!

?

𝐿 = 𝐸32𝐸31𝐸21−1

= 𝐸21−1𝐸31

−1𝐸32−1

=1 0 0−𝟐 1 00 0 1

−1 1 0 00 1 0+𝟏 0 1

−1 1 0 00 1 00 +𝟏 1

−1

=1 0 0𝟐 1 00 0 1

1 0 00 1 0−𝟏 0 1

1 0 00 1 00 −𝟏 1

=1 0 02 1 00 0 1

1 0 00 1 0−1 −1 1

=1 0 0𝟐 1 0−𝟏 −𝟏 1

47

Finally

in general

1 0 0𝟐 1 00 0 1

1 0 00 1 0−𝟏 0 1

1 0 00 1 00 −𝟏 1

=1 0 0𝟐 1 0−𝟏 −𝟏 1

𝑘

𝐸𝑖𝑘𝑗𝑘−1 =

48

?

11

1

1

𝑏𝑘 ’s

entry 𝑏𝑘 at (𝑖𝑘 , 𝑗𝑘) position for all k

sign changed value of those in 𝐸𝑖𝑘𝑗𝑘

so far we have 𝐴 = 𝐿𝑈

49

𝐴 = 𝐿𝑈 =1 0 0𝟐 1 0−𝟏 −𝟏 1

𝟐 𝟏 𝟏0 −𝟖 −𝟐0 0 𝟏

=1 0 0𝟐 1 0−𝟏 −𝟏 1

𝟐 𝟎 𝟎𝟎 −𝟖 𝟎𝟎 𝟎 𝟏

1 1/2 1/20 1 1/40 0 1

= 𝑳𝑫𝑼

inverse of GE operation result of GE on A

THEOREM 7. in general, we have

50

d1

d2

d3

dn

0’s

𝑎𝑖𝑗’s

di ≠ 0 for all j

=

d1

d2

d3

dn

0’s

0’s

11

1

1

0’s

𝑎𝑖𝑗di

(main) Theorem 8when we do not have to permute the rows of A in the process of G.E:

• Given an 𝑚 × 𝑛 matrix A, there exist an 𝑚 ×𝑚 matrix L and an 𝑚 × 𝑛matrix U with

𝐴 = 𝐿𝑈where L is a lower triangle matrix with 1’s on the diagonal and 0’s above the diagonal, and

U is an upper triangle matrix with non-zero entries on the diagonal and 0’s below the diagonal.

• U can further be decomposed into

𝑈 = 𝐷𝑈’ so that 𝐴 = 𝐿𝐷𝑈′where D is an 𝑚 ×𝑚 diagonal matrix with di on the diagonal and

U’ (of the same size as U) has 1’s on the diagonal and all other non-zero entries above the diagonal are the scaled version of those in U by di in the same row of D.

• When A=AT we have A=LDLT , that is, U’=LT

51

Remark on inverse of products We have seen that

𝐿 = 𝐸32𝐸31𝐸21−1 =

1 0 0−𝟐 1 00 0 1

−1 1 0 00 1 0+𝟏 0 1

−1 1 0 00 1 00 +𝟏 1

−1

=1 0 0𝟐 1 0−𝟏 −𝟏 1

Will it be true that

𝐿−1 = 𝐸32𝐸31𝐸21 =1 0 0−𝟐 1 00 0 1

1 0 00 1 0+𝟏 0 1

1 0 00 1 00 +𝟏 1

=1 0 0−𝟐 1 0+𝟏 +𝟏 1

?

No. Correct calculation gives

𝐿−1 = 𝐸32𝐸31𝐸21 =1 0 0−𝟐 1 00 0 1

1 0 00 1 0+𝟏 0 1

1 0 00 1 00 +𝟏 1

=1 0 0−𝟐 1 0−𝟏 +𝟏 1

Think about the reason by yourself why it does not work....

52

HW#1 Problems #5,#6(#5) Find A=LU=LDU' for the following matrices

(5-1) 𝐴 =1 4 04 12 40 4 0

(5-2) 𝐴 =2 4 80 3 90 0 7

(#6) Solve the equation Ax=b using the above factorization A=LU

when

𝑏 =101

using both A's above.

53

pivots

Pivots and Echelon matrix

54

1 3 3 22 6 9 7

-1 -3 3 4Consider A=

① →①② →②+①x(-2)③ →③+ ①x(+1)

1 3 3 20 0 3 30 0 6 6

① →①② →②③ →③+ ②x(-2)

1 3 3 20 0 3 30 0 0 0

i → i + j x(a) implies (i,j) position of L must be –a

Therefore, A=LU where1 3 3 22 6 9 7

-1 -3 3 4=

1 3 3 20 0 3 30 0 0 0

1 0 02 1 0

-1 2 1

echelon matrix=U

First and third columns are called pivot-column,

second and fourth columns are called non-pivot-column

rank of a matrixThe number of pivots in the Gauss elimination process of A is called the rank of A, and is denoted by r(A) or r.

Ex. The matrix A in the previous page has rank 2.

column rank of A

= max number of linearly independent columns of A

≤ number of columns of A

row rank of A

= max number of linearly independent rows of A

≤ number of rows of A

THEOREM 9: column rank = row rank = rankproof: next week.

55

permutation matrix Any row-permutation can be done by multiplying a matrix (called, a

permutation matrix) to the left.

For example, consider the permutation 𝒇 = (𝟏𝟐𝟑𝟒) that sends 𝟏 → 𝟐 → 𝟑 → 𝟒

The permutation of the rows of a 4 x n matrix A according to f can be performed by multiplying the following to the left of A:

𝑃𝑓 =

0 10 0

0 01 0

0 01 0

0 10 0

Note

𝑃𝑓𝐴 =

0 10 0

0 01 0

0 01 0

0 10 0

𝒂𝒃𝒄𝒅

=

𝒃𝒄𝒅𝒂

56

𝑃𝑓(𝑖, 𝑗) = 1 ⇔ 𝑓(𝑖) = 𝑗

Another example: Consider 𝑔 = (𝑎𝑏)(𝑐𝑑).

It moves as follows:

𝒂 𝒃 𝒄 𝒅

The permutation of the rows of a 4 x n matrix A according to f can be performed by multiplying the following to the left of A:

𝑃𝑔 =

0 11 0

0 00 0

0 00 0

0 11 0

Note

𝑃𝑔𝐴 =

0 11 0

0 00 0

0 00 0

0 11 0

𝒂𝒃𝒄𝒅

=

𝒃𝒂𝒅𝒄

57

will study “permutations” in general in Lecture #4

Permuting the rows in GE In general, in the process of

GE of A, we need to permute the rows of A in order to find a non-zero “pivot”

Permuting the rows can also be implemented by multiplying some matrix to the left of A.

Theorem 10. PA=LDU in general, for some permutation matrix P

In the example on the right, the swapping can be done by

58

2 1 1 54 2 3 -2

-2 7 0 9

2 1 1 50 0 1 -120 8 1 14

Gau

ss E

lim

inat

ion

2 1 1 50 8 1 140 0 1 -12

pivots

2 1 1 50 8 1 140 0 1 -12

swap

𝑃 =1 0 00 0 10 1 0

𝒓𝟏 𝒓𝟐 𝒓𝟑

HW#1 Problem #7

Find PA=LU and solve the equation Ax = b using this factorization, when

𝐴 =1 1 1−1 −1 21 0 3

and 𝑏 =101

59

HW#1 Problems #8,#9,#10

#8 Review Exercise Chapter 1 (text page 66): #1.13

#9 (This is first two parts of #1.26 in Page 67) True or false, with reason if true and

counterexample if false:

(a) If L1U1 = L2U2 then L1 = L2 and U1 = U2 .

(b) If A2+A = I then A-1 = A+I

#10 Review Exercise Chapter 1 (text page 67): #1.27

60

END OF LECTURE #1

Please, please, please, ... (thousands times more)

read Chapter 2 of the text before the next class....

Next lecture will be much more complicated than this....

61