f t luk - computer scienceflaherje/pdf/lin1.pdf · computational linear algebra course ma th csci...

15

Upload: hoangnhu

Post on 22-Nov-2018

238 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: F T Luk - Computer Scienceflaherje/pdf/lin1.pdf · Computational Linear Algebra Course MA TH CSCI Semester F all Instructors Joseph E Flahert y aherje cs rpi edu F ranklin T Luk luk

Computational Linear Algebra

�Course� �MATH� ����� CSCI� �����

� Semester� Fall ����

� Instructors�

� Joseph E Flaherty� aherje�csrpiedu

� Franklin T Luk� luk�csrpiedu

�Wesley Turner� turnerw�csrpiedu

Department of Computer ScienceRensselaer Polytechnic Institute

Troy� NY �����

Page 2: F T Luk - Computer Scienceflaherje/pdf/lin1.pdf · Computational Linear Algebra Course MA TH CSCI Semester F all Instructors Joseph E Flahert y aherje cs rpi edu F ranklin T Luk luk

OUTLINE

�� Introduction

��� Notation

��� Special matrices

�� Gaussian Elimination

��� Vector and matrix norms

��� Finite precision arithmetic

��� Factorizations

��� Pivoting

��� Accuracy estimation

�� Special Linear Systems

��� Symmetric positive de�nite systems

��� Banded and pro�le systems

��� Block tridiagonal systems

��� Symmetric inde�nite systems

��� Pro�le methods

��� Sparse Systems

�� Orthogonalization and Least Squares

��� Orthogonality and the singular value decomposition

��� Rotations and re�ections

��� QR factorizations

��� Least squares problems

Page 3: F T Luk - Computer Scienceflaherje/pdf/lin1.pdf · Computational Linear Algebra Course MA TH CSCI Semester F all Instructors Joseph E Flahert y aherje cs rpi edu F ranklin T Luk luk

OUTLINE

�� The Algebraic Eigenvalue Problem

��� Introduction

��� Power methods

��� Hessenberg and Schur forms� The QR algorithm

��� Symmetric problems

��� Jacobi iteration

�� Iterative Methods

��� Fixedpoint methods

��� The basic conjugate gradient method

��� Preconditioning

��� Krylov and generalizedresidual methods

� Introduction to Parallel Computing

�� Shared and distributedmemory computation

�� Matrix multiplication

�� Gaussian elimination

Page 4: F T Luk - Computer Scienceflaherje/pdf/lin1.pdf · Computational Linear Algebra Course MA TH CSCI Semester F all Instructors Joseph E Flahert y aherje cs rpi edu F ranklin T Luk luk

References

�� O� Axelsson ������� Iterative Solution Methods�Cambridge� Cambridge�

�� R� Barrett� M� Berry� T� Chan� J� Demmel� J� Do�nato� J� Dongarra� V� Eijkhout� V� Pozo� C� Romine�and H� van der Vorst ������� Templates for the

Solution of Linear Systems� Building Blocks for

Iterative Methods� SIAM� Philadelphia�

� A� Bj�orck ������� Numerical Methods for Least

Squares Problems� SIAM� Philadelphia�

�� W� Briggs ��� ��� A Multigrid Tutorial� SIAM�Philadelphia�

�� J�W� Demmel ������� Applied Numerical Linear

Algebra� SIAM� philadelphia�

�� A� George and J� Liu ��� ��� Computer Solution of

Large Sparse Positive De�nite Systems� Prentice�Hall� Englewood Cli�s�

�� G�H� Golub and C�F� Van Loan ������ Matrix

Computations� Third Edition� Johns Hopkins� Bal�timore�

� A� Greenbaum ������� Iterative Methods for Solv�

ing Linear Systems� SIAM Philadelphia�

Page 5: F T Luk - Computer Scienceflaherje/pdf/lin1.pdf · Computational Linear Algebra Course MA TH CSCI Semester F all Instructors Joseph E Flahert y aherje cs rpi edu F ranklin T Luk luk

�� W� Hackbusch ������� Iterative Solution of Large

Sparse Linear Systems of Equations� Springer�Verlag� Berlin�

��� N� Higham ������� Accuracy and Stability of Nu�

merical Algoritms� SIAM Philadelphia�

��� B� Parlett ��� ��� The Symmetric Eigenvalue Prob�

lem� Prentice�Hall� Englewood Cli�s�

��� Y� Saad ������� Iterative Methods for Sparse Lin�

ear Systems� PWS� Boston�

�� B� Smith� P� Bjorstad� and W� Gropp ������� Do�main Decomposition� Parallel Multilevel Meth�

ods for Elliptic Partial Di�erential Equations�Cambridge� Cambridge�

��� G�W� Stewart ������ Introduction to Matrix Com�

putations� Academic Press� New York�

��� G�W� Stewart and J��G� Sun ������ Matrix Per�

turbation Theory� Academic Press� New York�

��� L�N� Trefethen and D� Bau� III ������� Numerical

Linear Algebra� SIAM� Philadelphia� �

��� P� Wesseling ������� An Introduction to Multigrid

Methods� Wiley� Chichester��Text

Page 6: F T Luk - Computer Scienceflaherje/pdf/lin1.pdf · Computational Linear Algebra Course MA TH CSCI Semester F all Instructors Joseph E Flahert y aherje cs rpi edu F ranklin T Luk luk

� � D� Young ������� Iterative Solution of Large Lin�

ear Systems� Academic Press� New York�

Page 7: F T Luk - Computer Scienceflaherje/pdf/lin1.pdf · Computational Linear Algebra Course MA TH CSCI Semester F all Instructors Joseph E Flahert y aherje cs rpi edu F ranklin T Luk luk

�� Introduction

���� Notation

� Motivation�

� Linear algebra is involved in approximately �� of scien

ti�c computation

� Circuit and network analysis

� Potential theory

� Signal processing

� Acoustics and vibrations

� Optimization

� Finite di�erence and �nite element methods

� Tomography

� Optimal control

� Problems�

� Solve linear systems

� Solve least squares problems

� Solve eigenvalueeigenvector problems

Page 8: F T Luk - Computer Scienceflaherje/pdf/lin1.pdf · Computational Linear Algebra Course MA TH CSCI Semester F all Instructors Joseph E Flahert y aherje cs rpi edu F ranklin T Luk luk

Notation

� Reading Trefethen and Bau ������ Lecture �

� An m� n matrix

A �

�����

a�� a�� � � � a�na�� a�� � � � a�n��� ��� � � � ���

am� am� � � � amn

����� ���

� Bold letters denote vectors and matrices

� Italic letters denote scalars

� Lower case letters are elements of a matrix

� aij � �� i � �� �� ����m� j � �� �� ���� n� unless stated

� A � �m�n

� �m�n a vector space of m� n real matrices

� A vector x � �m�� � �m

x �

�����

x�x����

xm

����� � xT � �x�� x�� � � � � xm�

Page 9: F T Luk - Computer Scienceflaherje/pdf/lin1.pdf · Computational Linear Algebra Course MA TH CSCI Semester F all Instructors Joseph E Flahert y aherje cs rpi edu F ranklin T Luk luk

MATLAB

� Specify Algorithms using MATLAB

� Vector dot �scalar� product

c � xTy �

nXi��

xiyi ���

function c � dot�x� y�

� dot compute a dot product of the nvectors x and y

n � length�x�

c � �

for i � � n

c � c � x�i��y�i�

end

� Subscripts are enclosed in parentheses

� The MATLAB function length computes the dimension

of a vector

� The indicates a range

� In the for loop� i ranges over integers from � to n

� dot�x� y� is a library function in MATLAB

Page 10: F T Luk - Computer Scienceflaherje/pdf/lin1.pdf · Computational Linear Algebra Course MA TH CSCI Semester F all Instructors Joseph E Flahert y aherje cs rpi edu F ranklin T Luk luk

A SAXPY

� A saxpy ��scalar a x plus y�� is the vector

z � ax � y ���

function z � saxpy�a� x� y�

� saxpy compute the vector z � a � x � y where

� a is a scalar and x and y are vectors

n � length�x�

for i � � n

z�i� � a�x�i� � y�i�

end

� x and y have the same dimensions n

� Production software should check this

Page 11: F T Luk - Computer Scienceflaherje/pdf/lin1.pdf · Computational Linear Algebra Course MA TH CSCI Semester F all Instructors Joseph E Flahert y aherje cs rpi edu F ranklin T Luk luk

Matrix Multiplication

� Let A � �m�r� B � �r�n� and C � �m�n� then

cij �

rXk��

aikbkj� i � � m� j � � n ���

function C � matmy�A� B�

� matmy compute the matrix product C � AB

�m r� � size�A��

�r n� � size�B��

for i � � m

for j � � n

C�i�j� � ��

for k � � r

C�i�j� � C�i�j� � A�i�k��B�k�j��

end

end

end

� size�A� returns the row and column dimensions of A

� The �s suppress printing of results

� The complexity is mnr multiplications and additions

� Some details are omitted

��

Page 12: F T Luk - Computer Scienceflaherje/pdf/lin1.pdf · Computational Linear Algebra Course MA TH CSCI Semester F all Instructors Joseph E Flahert y aherje cs rpi edu F ranklin T Luk luk

Matrix Multiplication

� Matrix multiplication using the dot product

� Let aj denote the jth column of A

A � �a�� a�� � � � � ar� ���

� Replace the loop in ��� by a dot product

cij � aTi bj� i � � m� j � � n ���

� aTi is the ith column of AT or the ith row of A

function C � matmy�A� B�

� matmy compute the matrix product C � AB using

� the dot product

�m r� � size�A��

�r n� � size�B��

for i � � m

for j � � n

C�i�j� � dot�A�i�� r���B�� r�j���

end

end

� The range � r indicates an operation for an entire row or

column

� A � denotes transposition

� A�i� � r� is the ith row of A

��

Page 13: F T Luk - Computer Scienceflaherje/pdf/lin1.pdf · Computational Linear Algebra Course MA TH CSCI Semester F all Instructors Joseph E Flahert y aherje cs rpi edu F ranklin T Luk luk

Matrix Multiplication with Saxpys

� Reorder the loop structure regard the columns of C as linear

combinations of those of A�� �

� �

� �� �

��

��

��

��

��

��

��

�� �

��

��

� compute C � AB as

cj �

rXk��

bkjak� j � � n ��

� Implement the sum using saxpys

function C � smatmy�A� B�

� smatmy compute the matrix product C � AB

� using saxpys

�m r� � size�A��

�r n� � size�B��

C � zeros�m� n��

for j � � n

for k � � r

C�� m�j� � saxpy�B�k�j�� A�� m�k�� C�� m�j���

end

end

� zeros�m�n� returns a m� n matrix of zeros�

��

Page 14: F T Luk - Computer Scienceflaherje/pdf/lin1.pdf · Computational Linear Algebra Course MA TH CSCI Semester F all Instructors Joseph E Flahert y aherje cs rpi edu F ranklin T Luk luk

Outer Products

� Reorder the loops to perform an outer product

� An outer product of an mvector x and an nvector y is

the m� n matrix

C � xyT ���

or

C �

�����

x�x����

xm

����� �y�y� � � � yn� �

�����

x�y� x�y� � � � x�ynx�y� x�y� � � � x�yn��� ��� � � � ���

xmy� xmy� � � � xmyn

�����

� Problem � do matrix multiplication using outer products

� Show this for the �� � problem�� �

� �

� �� �

��

Page 15: F T Luk - Computer Scienceflaherje/pdf/lin1.pdf · Computational Linear Algebra Course MA TH CSCI Semester F all Instructors Joseph E Flahert y aherje cs rpi edu F ranklin T Luk luk

Outer Product

� Write an algorithm for a general system

� Di�erent loop structures perform di�erently on di�erent com

puters

dot product i� j� k

saxpy j� k� i

outer porduct k� i� j

��