1. NumericalMethods forEngineers andScientists

2. NumericalMethods forEngineers andScientistsSecond EditionRevised and ExpandedJoe D. HoffmanDepartment of Mechanical EngineeringPurdue UniversityWest Lafayette, IndianaMARCELDEKKERMARCEL DEKKER, INC. NEW YORK. BASEL 3. The first edition of this book was published by McGraw-HillI, nc. (NewY ork, 1992).ISBN: 0-8247-0443-6This book is printed on acid-free paper.HeadquartersMarcel Dekker, Inc.270 Madison Avenue, blew York, NY 10016tel: 212-696-9000; fax: 212-685-4540Eastern Hemisphere DistributionMarcel Dekker AGHutgasse 4, Postfach 812, CH-4001B asel, Switzerlandtel: 41-61-261-8482; fax: 41-61-261-8896World Wide Webhttp://www.dekker.comThe publisher offers discounts on this book when ordered in bulk quantities. For more information,write to Special Sales/Professional Marketing at the headquarters address above.Copyright 2001 by Marcel Dekker, Inc. All Rights Reserved.Neither this book nor any part may be reproduced or transmitted in any form or by any means,electronic or mechanical, including photocopying, microfilming, and recording, or by any informa-tionstorage and retrieval system, without permission in writing from the publisher.Current printing (last digit)10987654321PRINTED IN THE UNITED STATES OF AMERICA 4. To Cynthia Louise Hoffman 5. PrefaceThe second edition of this book contains several majori mprovementosv er the first edition.Someo f these improvementsin volve format and presentation philosophy, and someo f thechanges involve old material which has been deleted and new material which has beenadded.Each chapter begins with a chapter table of contents. The first figure carries a sketchof the application used as the examplep roblemi n the chapter. Section 1 of each chapter isan introduction to the chapter, whichd iscusses the examplea pplication, the general subjectmatter of the chapter, special features, and solution approaches. The objectives of thechapter are presented, and the organization of the chapter is illustrated pictorially. Eachchapter ends with a summarys ection, which presents a list of recommendationsd, os anddonts, and a list of what you should be able to do after studying the chapter. This list isactually an itemization of what the student should have learned from the chapter. It servesas a list of objectives, a study guide, and a review guide for the chapter.Chapter 0, Introduction, has been added to give a thorough introduction to the bookand to present several fundamental concepts of relevance to the entire book.Chapters 1 to 6, which comprise Part I, Basic Tools of Numerical Analysis, havebeen expanded to include more approaches for solving problems. Discussions of pitfalls ofselected algorithms have been added where appropriate. Part I is suitable for second-semestersophomoreso r first-semester juniors through beginning graduate students.Chapters 7 and 8, which comprise Part II, Ordinary Differential Equations, havebeen rewritten to get to the methodsf or solving problemsm oreq uickly, with less emphasison theory. A new section presenting extrapolation methods has been added in Chapter 7.All of the material has been rewritten to flow more smoothly with less repetition and lesstheoretical backgroundP. art II is suitable for juniors throughg raduate students.Chapters 9 to 15 of the first edition, which comprised Part III, Partial DifferentialEquations, has been shortened considerably to only four chapters in the present edition.Chapter 9 introduces elliptic partial differential equations. Chapter 10 introduces parabolicpartial differential equations, and Chapter 11 introduces hyperbolic partial differentialequations. These three chapters are a major condensation of the material in Part III of thefirst edition. The material has been revised to flow more smoothly with less emphasis ontheoretical background. A new chapter, Chapter 12, The Finite Element Method, has beenaddedt o present an introduction to that important methodo f solving differential equations.A new section, Programs, has been added to each chapter. This section presentsseveral FORTRApNro grams for implementing the algorithms developed in each chapterto solve the examplea pplication for that chapter. Thea pplication subroutines are written in 6. vi Prefacea form similar to pseudocode to facilitate the implementation of the algorithms in otherprogramming languages.More examples and more problems have been added throughout the book.The overall objective of the second edition is to improve the presentation format andmaterial content of the first edition in a mannerth at not only maintains but enhancest heusefullness and ease of use of the first edition.Many people have contributed to the writing of this book. All of the peopleacknowledgedi n the Preface to the First Edition are again acknowledged,e specially myloving wife, Cynthia Louise Hoffman. My many graduate students provided much helpand feedback, especially Drs. D. Hofer, R. HarwoodR, . Moore,a nd R. Stwalley. Thanks,guys. All of the figures were prepared by Mr. Mark Bass. Thanks, Mark. Once again, myexpert word processing specialist, Ms. Janice Napier, devoted herself unsparingly to thissecond edition. Thanky ou, Janice. Finally, I wouldl ike to acknowledgem yc olleague, Mr.B. J. Clark, Executive Acquisitions Editor at Marcel Dekker, Inc., for his encouragementand support during the preparation of both editions of this book.Joe D. Hoffman 7. ContentsPrefaceChapter 0.0.1.0.2.0.3.0.4.0.5.0.6.0.7.0.8.Part I.1.1.1.2.1.3.1.4.1.5.1.6.1.7.IntroductionObjectives and ApproachOrganization of the BookExamplesProgramsProblemsSignificant Digits, Precision, Accuracy, Errors, and NumbeRr epresentationSoftware Packages and LibrariesThe Taylor Series and the Taylor PolynomialBasic Tools of Numerical AnalysisSystems of Linear Algebraic EquationsEigenproblemsRoots of Nonlinear EquationsPolynomial Approximation and InterpolationNumerical Differentiation and Difference FormulasNumerical IntegrationSummaryChapter 1. Systems of Linear Algebraic Equations1.1. Introduction1.2. Properties of Matrices and Determinants1.3. Direct Elimination Methods1.4. LU Factorization1.5. Tridiagonal Systems of Equations1.6. Pitfalls of Elimination Methods1.7. Iterative Methods1.8. Programs1.9. SummaryExercise ProblemsChapter 2. Eigenproblems2.1. Introduction2.2. Mathematical Characteristics of Eigenproblems2.3. The Power MethodV1122334671111131414151616171821304549525967767781818589vii 8. viii2.4. The Direct Method2.5. The QR Method2.6. Eigenvectors2.7. Other Methods2.8. Programs2.9. Summary ,Exercise ProblemChapter 3. Nonlinear E3.1. Introduction3.2. General Features3.3. Closed Domain (13.4. Open Domain M~3.5. Polynomials3.6. Pitfalls of Root F3.7. Systems of Nonli~3.8. Programs3.9. SummaryExercise Problem.,uationsRoot Finding~racketing) MethodsIhodsnding Methods and Other Methods of Root Finding~ear EquationsChapter 4. Polynomial ~pproximation and Interpolation4.1. Introduction4.2. Properties of Pol aomials4.3. Direct Fit Polynor aials4.4. Lagrange Polynon rials4.5. Divided Differenc Tables and Divided Difference Polynomials4.6. Difference Tables and Difference Polynomials4.7. Inverse Interpolation4.8. Multivariate Approximation4.9. Cubic Splines4.10. Least Squares Approximation4.11. Programs4.12. SummaryExercise ProblemsChapter 5. Numerical Differentiation and Difference Formulas5.1. Introduction ~5.2. Unequally Spaced Data5.3. Equally Spaced Data5.4. Taylor Series Approach5.5. Difference Formulas5.6. Error Estimation and Extrapolation5.7. Programs5.8. SummaryExercise Prol~lemsChapter 6. Numerical Integration6.1. Introduction6.2. Direct Fit Polynomials6.3. Newton-Cotes FormulasContents101104110111112118119127127130135140155167169173179181187188190197198204208217218221225235242243251251254257264270270273279279285285288290 9. Contents ix6.4. Extrapolation and RombergI ntegration6.5. Adaptive Integration6.6. Gaussian Quadrature6.7. Multiple Integrals6.8. Programs6.9. SummaryExercise ProblemsPart II.II.1.11.2.II.3.II.4.II.5.11.6.11.7.Ordinary Differential EquationsIntroductionGeneral Features of Ordinary Differential EquationsClassification of Ordinary Differential EquationsClassification of Physical ProblemsInitial-Value Ordinary Differential EquationsBoundary-ValueO rdinary Differential EquationsSummaryChapter 7.7.1.7.2.7.3.7.4.7.5.7.6.7.7.7.8.7.9.7.10.7.11.7.12.7.13.7.14.7.15.7.16.One-DimensionaIln itial-Value Ordinary Differential EquationsIntroductionGeneral Features of Initial-Value ODEsThe Taylor Series MethodThe Finite Difference MethodThe First-Order Euler MethodsConsistency, Order, Stability, and ConvergenceSingle-Point MethodsExtrapolation MethodsMultipoint MethodsSummaryo f Methods and ResultsNonlinear Implicit Finite Difference EquationsHigher-Order Ordinary Differential EquationsSystems of First-Order Ordinary Differential EquationsStiff Ordinary Differential EquationsProgramsSummaryExercise ProblemsChapter 8.8.1.8.2.8.3.8.4.8.5.8.6.8.7.8.8.8.9.8.10.8.11.One-Dimensional Boundary-Value Ordinary Differential EquationsIntroductionGeneral Features of Boundary-Value ODEsThe Shooting (initial-Value) MethodThe Equilibrium (Boundary-Value) MethodDerivative (and Other) Boundary ConditionsHigher-Order Equilibrium MethodsThe Equilibrium Method for Nonlinear Boundary-Value ProblemsThe Equilibrium Method on Nonuniform GridsEigenproblemsProgramsSummaryExercise Problems297299302306311 315316323323323325326327330332335336340343346352359364378381391393397398401408414416435436439441450458466471477480483488490 10. x ContentsPart III.III. 1.111.2.III.3.111.4.111.5.III.6.Ill.7.111.8.111.9.III. 10.IlL 11.Partial Differential EquationsIntroductionGeneral Features of Partial Differential EquationsClassification of Partial Differential EquationsClassification of Physical ProblemsElliptic Partial Differential EquationsParabolic Partial Differential EquationsHyperbolic Partial Differential EquationsThe Convection-Diffusion EquationInitial Values and Boundary ConditionsWell-Posed ProblemsSummaryChapter9.1.9.2.9.3.9.4.9.5.9.6.9.7.9.8.9.9.9.10.9.11.9.12.9.13.9.14.9. Elliptic Partial Differential EquationsIntroductionGeneral Features of Elliptic PDEsThe Finite Difference MethodFinite Difference Solution of the Laplace EquationConsistency, Order, and ConvergenceIterative Methods of SolutionDerivative Boundary ConditionsFinite Difference Solution of the Poisson EquationHigher-Order MethodsNonrectangular DomainsNonlinear Equations and Three-Dimensional ProblemsThe Control Volume MethodProgramsSummaryExercise ProblemsChapter10.1.10.2.10.3.10.4.10.5.10.6.10.7.10.8.10.9.10.10.10.11.10.12.10A3.10. Parabolic Partial Differential EquationsIntroductionGeneral Features of Parabolic PDEsThe Finite Difference MethodThe Forward-Time Centered-Space (FTCS) MethodConsistency, Order, Stability, and ConvergenceThe Richardson and DuFort-Frankel MethodsImplicit MethodsDerivative Boundary ConditionsNonlinear Equations and Multidimensional ProblemsThe Convection-Diffusion EquationAsymptotic Steady State Solution to Propagation ProblemsProgramsSummaryExercise ProblemsChapter11.1.11.2.11. Hyperbolic Partial Differential EquationsIntroductionGeneral Features of Hyperbolic PDEs501501502504511516519520523524525526527527531532536543546550552557562570571575580582587587591593599605611613623625629637639645646651651655 11. Contents xi11.3.11.4.11.5.11.6.11.7.11.8.11.9.11.10.11.11.Chapter12.1.12.2.12.3.12.4.12.5.12.6.12.7.The Finite Difference Method 657The Forward-Time Centered-Space (FTCS) Method and the Lax Method 659Lax-Wendroff Type Methods 655Upwind Methods 673The Backward-Time Centered-Space (BTCS) Method 677Nonlinear Equations and Multidimensional Problems 682The Wave Equation 683Programs 691Summary 701Exercise Problems 70212. The Finite Element Method 711Introduction 711The Rayleigh-Ritz, Collocation, and Galerkin Methods 713The Finite Element Method for Boundary Value Problems 724The Finite Element Method for the Laplace (Poisson) Equation 739The Finite Element Method for the Diffusion Equation 752Programs 759Summary 769Exercise Problems 770775779795ReferencesAnswers to Selected ProblemsIndex 12. Introduction0.1. Objective and Approach0.2. Organization of the Book0.3. Examples0.4. Programs0.5. Problems0.6. Significant Digits, Precision, Accuracy, Errors, and NumbeRr epresentation0.7. Software Packages and Libraries0.8. The Taylor Series and the Taylor PolynomialThis Introduction contains a brief description of the objectives, approach, and organizationof the book. The philosophy behind the Examples, Programs, and Problems is discussed.Several years experience with the first edition of the book has identified several simple,but significant, concepts which are relevant throughout the book, but the place to includethem is not clear. These concepts, which are presented in this Introduction, include thedefinitions of significant digits, precision, accuracy, and errors, and a discussion of numberrepresentation. A brief description of software packages and libraries is presented. Last,the Taylor series and the Taylor polynomial, which are indispensable in developing andunderstanding many numerical algorithms, are presented and discussed.0.1 OBJECTIVE AND APPROACHThe objective of this book is to introduce the engineer and scientist to numerical methodswhich can be used to solve mathematical problems arising in engineering and science thatcannot be solved by exact methods. With the general accessibility of high-speed digitalcomputers, it is now possible to obtain rapid and accurate solutions to many complexproblems that face the engineer and scientist.The approach taken is as follows:1. Introduce a type of problem. 13. 2 Chapter 02. Present sufficient background to understand the problem and possible methodsof solution.3. Develop one or more numerical methods for solving the problem.4. Illustrate the numerical methods with examples.In most cases, the numerical methods presented to solve a particular problem proceed fromsimple methods to complex methods, which in many cases parallels the chronologicaldevelopment of the methods. Some poor methods and some bad methods, as well as goodmethods, are presented for pedagogical reasons. Why one method does not work is almostas important as why another method does work.0.2 ORGANIZATION OF THE BOOKThe material in the book is divided into three main parts.:I. Basic Tools of Numerical AnalysisII. Ordinary Differential EquationsIII. Partial Differential Equations :Part I considers many of the basic problems that arise in all branches of engineeringand science. These problems include: solution of systems of linear algebraic equations,eigenproblems, solution of nonlinear equations, polynomial approximation and interpola-tion,numerical differentiation and difference formulas, and numerical integration. Thesetopics are important both in their own right and as the foundation for Parts II and III.Part II is devoted to the numerical solution of ordinary differential equations(ODEs). The general features of ODEs are discussed. The two classes of ODEs (i.e.,initial-value ODEs and boundary-value ODEs) are introduced, and the two types ofphysical problems (i.e., propagation problems and equilibrium problems) are discussed.Numerousn umerical methods for solving ODEsa re presented.Part III is devoted to the numerical solution of partial differential equations (PDEs).Some general features of PDEs are discussed. The three classes of PDEs (i.e., ellipticPDEs, parabolic PDEs, and hyperbolic PDEs) are introduced, and the two types of physicalproblems (i.e., equilibrium problems and propagation problems) are discussed. Severalmodel PDEsa re presented. Numerousn umerical methods for solving the model PDEsa represented.The material presented in this book is an introduction to numerical methods. Manypractical problems can be solved by the methods presented here. Many other p[acticalproblems require other or more advanced numerical methods. Mastery of the materialpresented in this book will prepare engineers and scientists to solve manyo f their ex)erydayproblems, give them the insight to recognize when other methods are required, and givethem the background to study other methods in other books and journals.0.3 EXAMPLESAll of the numerical methods presented in this book are illustrated by applying them tosolve an example problem. Each chapter has one or two example problems, which aresolved by all of the methods presented in the chapter. This approach allows the analyst tocompare various methods for the same problem, so accuracy, efficiency, robustness, andease of application of the various methods can be evaluated. 14. Introduction 3Most of the example problems are rather simple and straightforward, thus allowingthe special features of the various methodst o be demonstratedc learly. All of the exampleproblems have exact solutions, so the errors of the various methods can be compared. Eachexample problem begins with a reference to the problem to be solved, a description of thenumericalm ethodto be employedd, etails of the calculations for at least one application ofthe algorithm, and a summaryo f the remaining results. Somec ommentas bout the solutionare presented at the end of the calculations in most cases.0.4 PROGRAMSMost numerical algorithms are generally expressed in the form of a computer program.This is especially true for algorithms that require a lot of computational effort and foralgorithms that are applied manyt imes. Several programmingla nguages are available forpreparing computer programs: FORTRABNa, sic, C, PASCALe,t c., and their variations,to namea few. Pseudocode,w hichi s a set of instructions for implementinga n algorithmexpressed in conceptual form, is also quite popular. Pseudocodec an be expressed in thedetailed form of any specific programmingla nguage.FORTRAisN one of the oldest programmingl anguages. Whenc arefully prepared,FORTRAcaNn approach pseudocode. Consequently, the programs presented in this bookare written in simple FORTRANTh. ere are several vintages of FORT_RAFNO: RTRAI,NFORTRAINI, FORTRA6N6 , 77, and 90. The programs presented in this book arecompatible with FORTRA7N7 and 90.Several programs are presented in each chapter for implementing the moreprominent numerical algorithms presented in the chapter. Each program is applied tosolve the example problem relevant to that chapter. The implementation of the numericalalgorithm is contained within a completely self-contained application subroutine whichcan be used in other programs. These application subroutines are written as simply aspossible so that conversion to other programmingl anguages is as straightforward aspossible. These subroutines can be used as they stand or easily modified for otherapplications.Each application subroutine is accompanied by a program main. The variablesemployed in the application subroutine are defined by comment statements in programmain. The numerical values of the variables are defined in programm ain,whicht hen callsthe application subroutine to solve the example problem and to print the solution. Thesemain programs are not intended to be convertible to other programmingl anguages. Insome problems where a function of some type is part of the specification of the problem,that function is defined in a function subprogram which is called by the applicationsubroutine.FORTRAcoNm pilers do not distinguish between uppercase and lowercase letters.FORTRApNro grams are conventionally written in uppercase letters. However,i n thisbook, all FORTRAprNo grams are written in lowercase letters.0.5 PROBLEMSTwo types of problems are presented at the end of each chapter:1. Exercise problems2. Applied problems 15. 4 Chapter 0Exercise problems are straightforward problems designed to give practice in theapplication of the numerical algorithms presented in each chapter. Exercise problemsemphasize the mechanics of the methods.Applied problems involve more applied engineering and scientific applicationswhich require numerical solutions.Many of the problems can be solved by hand calculation. A large number of theproblems require a lot of computational effort. Those problems should be solved bywriting a computerp rogramt o performt he calculations. Eveni n those cases, however,i t isrecommendetdh at one or two passes through the algorithm be madeb y hand calculation toensure that the analyst fully understands the details of the algorithm. These results also canbe used to validate the computer program.Answertso selected problemsa re presented in a section at the end of the book. All ofthe problems for which answers are given are denoted by an asterisk appearing with thecorresponding problem number in the problem sections at the end of each chapter. TheSolutions Manual contains the answers to nearly all of the problems.0.6 SIGNIFICANT DIGITS, PRECISION, ACCURACY, ERRORS, ANDNUMBER REPRESENTATIONNumerical calculations obviously involve the manipulation (i.e., addition, multiplication,etc.) of numbers. Numbercsa n be integers (e.g., 4, 17, -23, etc.), fractions (e.g.,-2/3, etc.), or an inifinite string of digits (e.g., n--3.1415926535...). Whend ealingwith numerical values and numerical calculations, there are several concepts that must beconsidered: ,1. Significant digits2. Precision and accuracy3. Errors4. Number representationThese concepts are discussed briefly ha this section.Significant DigitsThe significant digits, or figures, in a number are the digits of the number which areknownto be correct. Engineeringa nd scientific calculations generally begin with a set ofdata having a known number of significant digits. When these numbers are processedthrough a numerical algorithm, it is important to be able to estimate how many significantdigits are present in the final computedre sult.Precision and AccuracyPrecision refers to how closely a number represents the number it is representing.Accuracyr efers to howc losely a numbera grees with the true value of the numberi t isrepresenting.Precision is governed by the number of digits being carried in the numericalcalculations. Accuracyis governedb y the errors in the numerical approximation,p recisionand accuracy are quantified by the errors in a numerical calculation. 16. Introduction 5ErrorsThe accuracy of a numerical calculation is quantified by the error of the calculation.Several types of errors can occur in numerical calculations.1. Errors in the parameters of the problem (assumed nonexistent).2. Algebraic errors in the calculations (assumed nonexistent).3. Iteration errors.4. Approximation errors.5. Roundoff errors.Iteration error is the error in an iterative method that approaches the exact solutionof an exact problem asymptotically. Iteration errors must decrease toward zero as theiterative process progresses. The iteration error itself may be used to determine thesuccessivea pproximationtso the exact solution. Iteration errors can be reducedt o the limitof the computing device. The errors in the solution of a system of linear algebraicequations by the successive-over-relaxation (SOR) method presented in Section 1.5 areexampleso f this type of error.Approximation error is the difference between the exact solution of an exactproblem and the exact solution of an approximation of the exact problem. Approximationerror can be reduced only by choosing a more accurate approximation of the exactproblem. The error in the approximation of a function by a polynomial, as described inChapter 4, is an exampleo f this type of error. The error in the solution of a differentialequation where the exact derivatives are replaced by algebraic difference approximations,which have mmcatione rrors, is another exampleo f this type of error.Rouudoff error is the error caused by the finite word length employed in thecalculations. Roundoff error is more significant when small differences between largenumbers are calculated. Most computers have either 32 bit or 64 bit word length,corresponding to approximately 7 or 13 significant decimal digits, respectively. Somecomputers have extended precision capability, which increases the number of bits to 128.Care must be exercised to ensure that enoughs ignificant digits are maintainedi n numericalcalculations so that roundoff is not significant.Number RepresentationNumbers are represented in number systems. Any number of bases can be employed asthe base of a number system, for example, the base 10 (i.e., decimal) system, the base(i.e., octal) system, the base 2 (i.e., binary) system, etc. The base 10, or decimal, systemthe most commonsy stem used for humanc ommunication. Digital computers use the base2, or binary, system. In a digital computer,a binary numbecr onsists of a numbero f binarybits. The numbero f binary bits in a binary numberd etermines the precision with whicht hebinary numberr epresents a decimal number. The most commonsi ze binary numberi s a 32bit number, which can represent approximately seven digits of a decimal number. Somedigital computers have 64 bit binary numbers, which can represent 13 to 14 decimal digits.In manye ngineering and scientific calculations, 32 bit arithmetic is adequate. Howeveri,nmanyo ther applications, 64 bit arithmetic is required. In a few special situations, 128 bitarithmetic may be required. On 32 bit computers, 64 bit arithmetic, or even 128 bitarithmetic, can be accomplished using software enhancements. Such calculations arecalled double precision or quad precision, respectively. Such software enhanced pre-cisioncan require as much as 10 times the execution time of a single precision calculation. 17. 6 Chapter 0Consequently, some care must be exercised when deciding whether or not higher precisionarithmetic is required. All of the examples in this book are evaluated using 64 bitarithmetic to ensure that roundoff is not significant.Except for integers and some fractions, all binary representations of decimalnumbers are approximations, owing to the finite word length of binary numbers. Thus,some loss of precision in the binary representation of a decimal number is unavoidable.When binary numbers are combined in arithmetic operations such as addition, multi-plication,etc., the true result is typically a longer binary number which cannot berepresented exactly with the number of available bits in the binary number capability ofthe digital computer.T hus, the results are roundedo ff in the last available binary bit. Thisrounding off gives rise to roundoff error, which can accumulate as the number ofcalculations increases.0.7 SOFTWARE PACKAGES AND LIBRARIESNumeroucso mmercials oftware packages and libraries are available for implementingt henumerical solution of engineering and scientific problems. Two of the more versatilesoftware packages are Mathcad and Matlab. These software packages, as well as severalother packages and several libraries, are listed below with a brief description of each oneand references to sources for the software packages and libraries.A. Software PackagesExcel Excel is a spreadsheet developed by Microsoft, Inc., as part of Microsoft Office. Itenables calculations to be performed on rows and columns of numbers. The calculations tobe performed are specified for each column. When any number on the spreadsheet ischanged, all of the calculations are updated. Excel contains several built-in numericalalgorithms. It also includes the Visual Basic programmingla nguage and somep lottingcapability. Although its basic function is not numerical analysis, Excel can be usedproductively for many types of numerical problems. Microsoft, Inc. www.microsoft.com/office/Excel.MacsymMa acsymias the worlds first artificial intelligence .based math engineproviding easy to use, powerful math software for both symbolic and numerical comput-ing.Macsyma, Inc., 20 Academy St., Arlington, MA 02476-6412. (781) 646-4550,webmaster@macsyma.com, www.macsyma.com.Maple Maple 6 is a technologically advanced computational system with bothalgorithms and numeric solvers. Maple 6 includes an extensive set of NAG(N umericalAlgorithms Group) solvers forcomputational linear algebra. Waterloo Maple, Inc., 57 ErbStreet W., Waterloo, Ontario, Canada N2L 6C2. (800) 267-6583, (519) 747-2373,info@maplesoft.com, www.maplesoft.com.Mathematica Mathematica 4 is a comprehensive software package which p,erformsboth symbolic and numeric computations. It includes a flexible and intuitive programminglanguage and comprehensive plotting capabilities. Wolfram Research, Inc., 100 TradeCenter Drive, Champaign IL 61820-7237. (800) 965-3726, (217) 398-0700, info@wolfram.corn, www.wolfram.com.MathcadM athcad8 provides a free-form interface which permits the integration ofreal math notation, graphs, and text within a single interactive worksheet. It includesstatistical and data analysis functions, powerful solvers, advanced matrix manipulation, 18. Introduction 7and the capability to create your own functions. Mathsoft, Inc., 101 Main Street,Cambridge, MA 02142-1521. (800) 628-4223, (617) 577-1017, info@mathsoft.com,www.mathcad.com.Matlab Matlab is an integrated computing environment that combines numericcomputation, advanced graphics and visualization, and a high-level programminglanguage. It provides core mathematics and advanced graphics tools for data analysis,visualization, and algorithm and application development, with more than 500 mathe-matical,statistical, and engineering functions. The MathworksI,n c., 3 AppleH ill Drive,Natick, MA 01760-2090. (508) 647-7000, info@mathworks.com, www.mathworks.com.B. LibrariesGAMGS AMS(G uide to Available Mathematical Software) is a guide to over 9000software modules contained in some 80 software packages at NIST (National Institute forStandards and Technology) and NETLIBg. ams.nist.gov.IMSLI MSL(I nternational Mathematicsa nd Statistical Library) is a comprehensiveresource of more than 900 FORTRAsNub routines for use in general mathematics andstatistical data analysis. Also available in C and Java. Visual Numerics,I nc., 1300 W. SamHouston Parkway S., Suite 150, Houston TX 77042. (800) 364-8880, (713) 781-9260,info@houston.vni.com, www.vni.com.LAPACKLA PACKis a library of FORTRA7N7 subroutines for solving linearalgebra problems and eigenproblems. Individual subroutines can be obtained throughNETLIB. The complete package can be obtained from NAG.NAGN AGis a mathematical software library that contains over 1000 mathematicaland statistical functions. Available in FORTRAanNd C. NAGI, nc., 1400 Opus Place,Suite 200, Downers Grove, IL 60515-5702. (630) 971-2337, naginfo@nag.com,www.nag.com.NETLIBN ETLIBis a large collection of numerical libraries, netlib@research.att.com, netlib@ornl.gov, netlib@nec.no.C. Numerical RecipesNumericaRl ecipes is a book by William H. Press, Brian P. Flarmery, Saul A. Teukolsky,and William T. Vetterling. It contains over 300 subroutines for numerical algorithms.Versions of the subroutines are available in FORTRACN, , Pascal, and Basic. The sourcecodes are available on disk. CambridgeU niversity Press, 40 West2 0th Street, NewY ork,NY 10011. www.cup.org.0.8 THE TAYLOR SERIES AND THE TAYLOR POLYNOMIALA power series in powers of x is a series of the formY~.anx~ = ao + a~x + azx2 +...n=OA power series in powers of (x - x0) is given~ an(x - Xo)" = o +at(x - Xo) + az(x -- X0)2 --{ - n=0(0.1)(0.2) 19. 8 Chapter 0Within its radius of convergence, r, any continuous function, f(x), can be representedexactly by a power series. Thus,f(x) = ~ a,(x - Xo)" (0.3)n=0is continuous for (x0 - r) < x < (xo + r).A. Taylor Series in One Independent VariableIf the coefficients, an, in Eq. (0.3) are given by the rule:a0 =f(xo) al = ~.f,(xo) a2 1 " x ,= ~f (0) ... (0.4)then Eq. (0.3) becomes the Taylor series off(x) at x o.Thus,f(x) =f(xo) ~.f(xo)(X - Xo) + ~ f" (xo)(X - Xo)2 +... (0.5)Equation (0.5) can be written in the simpler appearing form, 1 ,, ~ f(x) =fo +f~Ax + -~f~ ~ +. .. +l~ f(nn)!Aox + "" (0.6)wherefo = f (xo),f (n) = df(") / ", and Ax= (x - Xo)Equation (056)can bewrittenin thecompact formf(x) = Y~ ~fot")(x - Xo)" , (0.7)n=0 n!Whenx 0 = 0, the Taylor series is knowna s the Madaurins eries. In that case, Eqs.(0.5) and (0.7) becomef(x) =f(0) + f(O)x + f"(O)x2 +... (0.8)f(x) = (n)(0)x~ (0.9)It is, of course, impractical to evaluate an infinite Taylor series term by term. TheTaylor series can be written as the finite Taylor series, also knowna s the Taylor formulao rTaylor polynomial with remainder, as follows:f(x) =/(Xo) + f(xo)(X - Xo) + l_g f,,(Xo)( x _ Xo)2 +...(0.10) 1+ ~.f(")(Xo)(X n + Rn+lwhere the term Rn+l is the remainder term given byR,+I _ 1 f(n+l)(~)(x -" +1Xo) (0.11)(~n + 1)!where ~ lies between x0 and x. Equation (0. i0) is quite useful in numerical analysis, wherean approximation off@) is obtained by truncating the remainder term. 20. Introduction 9B. Taylor Series in Two Independent VariablesPower series can also be written for functions of more than one independent variable. For afunction of two independenvt ariables,f (x, y), the Taylors eries off(x, y) at 0, Y)0 is givenbyf(x,y) =fo + ~-~fx o(X-Xo)+-~ o(y-yo)+ I~x2 (-xo)~+ oXoylo -Xo)(y-yo)+~l /2fl x 2.02( (x ~--~f2 o(y-yo)2)""(0.12)Equation (0.12) can be written in the general formf(x,Y) = n~=o~ (X- Xo)-~x+ (y- yo)- ~ f(x,y)lo (0.13)where the term (...)n is expanded by the binomial expansion and the resulting expansionoperates on the function f (x, y) and is evaluated at (xo,Yo).The Taylor formula with remainder for a function of two independent variables isobtained by evaluating the derivatives in the (n + 1)st term at the point (, r/), where (,lies in the region betweenp oints (xo, Yo)a nd (x, y). 21. BasicT oolso f NumericaAl nalysis1.1. Systems of Linear Algebraic Equations1.2. Eigenproblems1.3. Roots of Nonlinear Equations1.4. Polynomial Approximation and Interpolation1.5. Numerical Differentiation and Difference Formulas1.6. Numerical Integration1.7. SummaryManyd ifferent types of algebraic processes are required in engineering and science. Theseprocesses include the solution of systems of linear algebraic equations, the solution ofeigenproblems, finding the roots of nonlinear equations, polynomial approximation andinterpolation, numerical differentiation and difference formulas, and numerical integration.These topics are not only important in their own right, they lay the foundation for thesolution of ordinary and partial differential equations, which are discussed in Parts II andIII, respectively. FigureI . 1 illustrates the types of problemsc onsideredi n Part I.The objective of Part I is to introduce and discuss the general features of each ofthese algebraic processes, which are the basic tools of numerical analysis.1.1 SYSTEMS OF LINEAR ALGEBRAIC EQUATIONSSystems of equations arise in all branches of engineering and science. These equationsmay be algebraic, transcendental (i.e., involving trigonometric, logarithmic, exponential,etc., functions), ordinary differential equations, or partial differential equations. Theequations may be linear or nonlinear. Chapter 1 is devoted to the solution of systems oflinear algebraic equations of the following form:a] ix1 + al2x 2 -q- al3x3 q- .. + alnXn = b~a21xI q- a22x2 q- a23x3 q- q- a2,xn = b2anlX1 q-- an2x2 q- an3X3 q- ... q- annXn n= b(I.la)(I. lb)(I. 1 n)11 22. 12 Part Iwherex j (j = 1, 2 ..... n) denotes the unknowvna riables, aid (i,j = 1,2 ..... n) denotesthe coefficients of the unknownv ariables, and bi (i = 1, 2 ..... n) denotes the nonhomo-geneousterms. For the coefficients aid, the first subscript i correspondst o equationi , andthe second subscriptj corresponds to variable xj. The numbero f equations can range fromtwo to hundreds, thousands, and even millions.Systems of linear algebraic equations arise in many different problems, for example,(a) networkp roblems( e.g., electrical networks), (b) fitting approximatingf unctionsChapter 4), and (c) systems of finite difference equations that arise in the numerical solution of differential equations (see Chapters 7 to 12). The list is endless. Figure I.laillustrates a static spring-mass system, whose static equilibrium configuration is governedby a system of linear algebraic equations. That system of equations is used throughoutChapter 1 as an example problem.(a) Static spring-masssy stem. (b) Dynamiscp ring-masss ystem.f(x)(c) Rootso f nonlineare quations.f(x)Figure 1.1(d) Polynomiaalp proximatioann d interpolation.f(x)x x(f) Numericainl tegration.(e) Numericadli fferentiation.Basict ools of numericaal nalysis. (a) Static spring-masssy stem.( b) Dynamsicp ring-masss ystem. (c) Rootso f nonlinear equations. (d) Polynomiaal pproximatioann d interpolation.(e) Numericadli fferentiation. (f) Numericainl tegration. 23. Basic Tools of NumericaAl nalysis 13Systemso f linear algebraic equations can be expressed very conveniently in terms ofmatrix notation. Solution methods can be developed very compactly in terms of matrixnotation. Consequently, the elementary properties of matrices and determinants arereviewed at the beginning of Chapter 1.Two fundamentally different approaches can be used to solve systems of linearalgebraic equations:1. Direct methods2. Iterative methodsDirect methods are systematic procedures based on algebraic elimination. Several directelimination methods, for example, Gauss elimination, are presented in Chapter 1. Iterativemethods obtain the solution asymptotically by an iterative procedure in which a trialsolution is assumed, the trial solution is substituted into the system of equations todetermine the mismatch, or error, and an improved solution is obtained from the mismatchdata. Several iterative methods, for example, successive-over-relaxation (SOR), arepresented in Chapter 1.The notation, concepts, and procedures presented in Chapter 1 are used throughoutthe remainder of the book. A solid understanding of systems of linear algebraic equationsis essential in numerical analysis.1.2 EIGENPROBLEMSEigenproblems arise in the special case where a system of algebraic equations ishomogeneoust;h at is, the nonhogeneoust erms, bi in Eq. (I.1), are all zero, and thecoefficients contain an unspecified parameter, say 2. In general, when bi -~ O, the onlysolution to Eq. (I.1) is the trivial solution, 1 =x2... .. x n = 0. However, when thecoefficients aid contain an unspecified parameter, say 2, the value of that parameter can bechosen so that the system of equations is redundant, and an infinite numbero f solutionsexist. The unspecified parameter 2 is an eigenvalue of the system of equations. Forexample,(all -- )~)x1 + al2x2 = 0 (I.2a)azlx1 ~- (a22 -- ),)x2 = 0 (I.2b)is a linear eigenproblem. The value (or values) of 2 that make Eqs. (I.2a) and (I.2b)identical are the eigenvalueso f Eqs. (I.2). In that case, the two equationsa re redundant,the only unique solution is xI = x2 = 0. Howevera, n infinite numbero f solutions can beobtained by specifying either xl or x2, then calculating the other from either of the tworedundant equations. The set of values ofx1 and x2 correspondingt o a particular value of 2is an eigenvector of Eq. (I.2). Chapter 2 is devoted to the solution of eigenproblems.Eigenproblems arise in the analysis of many physical systems. They arise in theanalysis of the dynamicb ehavior of mechanical,e lectrical, fluid, thermal, and structuralsystems. They also arise in the analysis of control systems. Figure I.lb illustrates adynamic spring-mass system, whose dynamic equilibrium configuration is governed by asystem of homogeneousl inear algebraic equations. That system of equations is usedthroughout Chapter 2 as an examplep roblem. Whenth e static equilibrium configuration ofthe system is disturbed and then allowed to vibrate freely, the system of masses willoscillate at special frequencies, which depend on the values of the masses and the spring 24. 14 Part Iconstants. These special frequencies are the eigenvalues of the system. The relative valuesof x~, x2, etc. corresponding to each eigenvalue 2 are the eigenvectors of the system.The objectives of Chapter 2 are to introduce the general features of eigenproblemsand to present several methods for solving eigenproblems. Eigenproblems are specialproblems of interest only in themselves. Consequently, an understanding of eigenproblemsis not essential to the other concepts presented in this book.1.3 ROOTS OF NONLINEAR EQUATIONSNonlinear equations arise in many physical problems. Finding their roots, or zeros, is acommonp roblem. The problem can be stated as follows:Given the continuous nonlinear functionf(x), find the value of x = e such thatf(~) =where ~ is the root, or zero, of the nonlinear equation. Figure I.lc illustrates the problemgraphically. The function f (x) may be an algebraic function, a transcendental function, thesolution of a differential equation, or any nonlinear relationship between an input x and aresponse f(x). Chapter 3 is devoted to the solution of nonlinear equations.Nonlinear equations are solved by iterative methods. A trial solution is assumed, thetrial solution is substituted into the nonlinear equation to determine the error, or mismatch,and the mismatch is used in some systematic manner to generate an improved estimate ofthe solution. Several methods for finding the roots of nonlinear equations are presented inChapter 3. The workhorse methods of choice for solving nonlinear equations are Newtonsmethod and the secant method. A detailed discussion of finding the roots of polynomials ispresented. A brief introduction to the problems of solving systems of nonlinear equationsis also presented.Nonlinear equations occur throughout engineering and science. Nonlinear equationsalso arise in other areas of numerical analysis. For example, the shooting method forsolving boundary-valueo rdinary differential equations, presented in Section 8.3, requiresthe solution of a nonlinear equation. Implicit methods for solving nonlinear differentialequations yield nonlinear difference equations. The solution of such problems is discussedin Sections 7.11, 8.7, 9.11, 10.9, and 11.8. Consequently, a thorough understanding ofmethods for solving nonlinear equations is an essential requirement for the numericalanalyst.1.4 POLYNOMIAL APPROXIMATION AND INTERPOLATIONIn many problems in engineering and science, the data under consideration are known onlyat discrete points, not as a continuousf unction. For example,a s illustrated in FigureI . 1 d,the continuous function f(x) may be known only at n discrete values of x:Yi = y(xi) (i = 1, 2,..., (1.3)Values of the function at points other than the known discrete points may be needed(i.e., interpolation). The derivative of the function at some point may be needed (i.e.,differentiation). The integral of the function over some range may be required (i.e.,integration). These processes, for discrete data, are performed by fitting an approximatingfunction to the set of discrete data and performing the desired processes on theapproximating function. Many types of approximating functions can be used. 25. Basic Tools of NumericaAl nalysis 15Because of their simplicity, ease of manipulation, and ease of evaluation, poly-nomialsare an excellent choice for an approximating function. The general nth-degreepolynomial is specified byPn(X) = 0 q- alx q - a2~x2 +... q - anx (1.4)Polynomials can be fit to a set of discrete data in two ways:1. Exact fit2. Approximate fitAn exact fit passes exactly through all the discrete data points. Direct fit polynomials,divided-difference polynomials, and Lagrange polynomials are presented in Chapter 4 forfitting nonequally spaced data or equally spaced data. Newtond ifference polynomialsa represented for fitting equally spaced data. The least squares procedure is presented fordetermining approximate polynomial fits.Figure I.ld illustrates the problem of interpolating within a set of discrete data.Procedures for interpolating within a set of discrete data are presented in Chapter 4.Polynomiaal pproximationis essential for interpolation, differentiation, and integra-tionof sets of discrete data. A good understanding of polynomial approximation is anecessary requirement for the numerical analyst.1.5 NUMERICAL DIFFERENTIATION AND DIFFERENCE FORMULASThe evaluation of derivatives, a process known as differentiation, is required in manyproblems in engineering and science. Differentiation of the function f(x) is denoted byd-~x (f(x)) =f(x)(1.5)The function f(x) may be a known function or a set of discrete data. In general, knownfunctions can be differentiated exactly. Differentiation of discrete data requires anapproximate numerical procedure. Numerical differentiation formulas can be developedby fitting approximating functions (e.g., polynomials) to a set of discrete data anddifferentiating the approximating function. For polynomial approximating functions, thisyieldsFigure I.le illustrates the problemo f numericald ifferentiation of a set of discretedata. Numerical differentiation procedures are developed in Chapter 5.The approximating polynomial may be fit exactly to a set of discrete data by themethods presented in Chapter 4, or fit approximately by the least squares proceduredescribed in Chapter 4. Several numerical differentiation formulas based on differentiationof polynomials are presented in Chapter 5.Numerical differentiation formulas also can be developed using Taylor series. Thisapproach is quite useful for developing difference formulas for approximating exactderivatives in the numerical solution of differential equations. Section 5.5 presents a tableof difference formulas for use in the solution of differential equations.Numerical differentiation of discrete data is not required very often. However, thenumerical solution of differential equations, which is the subject of Parts II and III, is one 26. 16 Part Iof the most important .areas of numerical analysis. The use of difference formulas isessential in that application.h6 NUMERICAL INTEGRATIONThe evaluation of integrals, a process knowna s integration, or quadrature, is required inmany problems in engineering and science. Integration of the functionf(x) is denotedI = f(x) (I.7)aThe function f(x) may be a known function or a set of discrete data. Some knownfunctions have an exact integral. Many known functions, however, do not have an exactintegral, and an approximate numerical procedure is required to evaluate Eq. (I.7). Whenknownf unction is to be integrated numerically, it must first be discretized. Integration ofdiscrete data always requires an approximate numerical procedure. Numerical integration(quadrature) formulas can be developed by fitting approximating functions (e.g., poly-nomials)to a set of discrete data and integrating the approximating function. Forpolynomial approximating functions, this givesI = dx ~- P,(x) (I.8)FigureI . 1 f illustrates the problemo f numericali ntegration of a set of discrete data.Numerical integration procedures are developed in Chapter 6.The approximatingf unction can be fit exactly to a set of discrete data by direct fitmethods, or fit approximately by the least squares method. For unequally spaced data,direct fit polynomialsc an be used. For equally spaced data, the Newtonfo rward-differencepolynomials of different degrees can be integrated to yield the Newton-Coteqs uadratureformulas. The most prominent of these are the trapezoid rule and Simpsons 1/3 rule.Romberg integration, which is a higher-order extrapolation of the trapezoid rule, isintroduced. Adaptive integration, in which the range of integration is subdivided auto-maticallyuntil a specified accuracy is obtained, is presented. Gaussian quadrature, whichachieves higher-order accuracyf or integrating knownf unctions by specifying the locationsof the discrete points, is presented. The evaluation of multiple integrals is discussed.Numerical integration of both known functions and discrete data is a commonproblem. The concepts involved in numerical integration lead directly to numericalmethodsf or solving differential equations.1.7 SUMMARYPart I of this book is devoted to the basic tools of numerical analysis. These topics areimportant in their own fight. In addition, they provide the foundation for the solution of -ordinary and partial differential equations, which are discussed in Parts II and III,respectively. The material presented in Part I comprises the basic language of numericalanalysis. Familiarity and masteryo f this material is essential for the understandinga nd useof more advanced numerical methods. 27. 1Systemos f Linear AlgebraicE quations1.1. Introduction1.2. Properties of Matrices and Determinants1.3. Direct Elimination Methods1.4. LU Factorization1.5. Tfidiagonal Systems of Equations1.6. Pitfalls of Elimination Methods1.7. Iterative Methods1.8. Programs1.9. SummaryProblemsExamples1.1.1.2.1.3.1.4.1.5.1.6.1.7.1.8.1.9.1.10.1.11.1.12.1.13.1.14.1.15.1.16.1.17.1.18.1.19.1.20.1.21.1.22.1.23.Matrix additionMatrix multiplicationEvaluation of a 3 x 3 determinant by the diagonal methodEvaluation of a 3 3 determinant by the cofactor methodCramers ruleEliminationSimple eliminationSimple elimination for multiple b vectorsElimination with pivoting to avoid zero pivot elementsElimination with scaled pivoting to reduce round-off errorsGauss-Jordan eliminationMatrix inverse by Gauss-Jordan eliminationThe matrix inverse methodEvaluation of a 3 3 determinant by the elimination methodThe Doolittle LU methodMatrix inverse by the Doolittle LU methodThe Thomas algorithmEffects of round-off errorsSystem conditionNorms and condition numbersThe Jacobi iteration methodThe Gauss-Seidel iteration methodThe SOR method17 28. 18 Chapter 11.1 INTRODUCTIONThe static mechanical spring-mass system illustrated in Figure 1.1 consists of three massesm~ to m3, having weights W1 to W3, interconnected by five linear springs K~ to K5. In theconfiguration illustrated on the left, the three masses are supported by forces F~ to F3 equalto weightsW ~to W3,r espectively, so that the five springs are in a stable static equilibriumconfiguration. When the supporting forces F1 to F3 are removed, the masses move3d,ownwarda nd reach a new static equilibrium configuration, denoted by x~, x2, and xwherex ~, x2, and x3 are measuredf rom the original locations of the correspondingm asses.Free-body diagrams of the three masses are presented at the bottom of Figure 1.1.Performing a static force balance on the three masses yields the following system of threelinear algebraic equations:(X 1 q-X 2-~x3)xI -X2x 2-x3x3 1= m-X2x I + (X2 -~- X4)x2 - X4x3 2= m-K3x~- X~x~+ (~3 + x4 + X~)x3= w3(1.1a)(1.1b)(1.1c)Vvqaen values ofK1 to Ks and W1 to W3 are specified, the equilibrium displacements xI tox3 can be determined by solving Eq. (1.1).The static mechanical spring-mass system illustrated in Figure 1.1 is used as theexample problem in this chapter to illustrate methods for solving systems of linearIKIX K2 (I)x2-Ix) K3 (x3-xK3(X3-X1) W1 K2(x2-x1) W2 K4(x3-x2) W3Figure 1.1 Static mechanical spring-mass system.K4(X3-X2)m3K5x3 29. Systemso f Linear AlgebraicE quations 19algebraic equations. For that purpose, let K1 = 40 N/cm, K2 = K3 = K4 = 20 N/cm, andK5 = 90N/cm. Let W1 = W2 = W3 = 20N. For these values, Eq. (1.1) becomes:80xt - 20x2 - 20x3 = 20-20xl + 40x2 - 20x3 = 20-20x~ - 20x2 + 130x3 = 20(1.2a)(1.2b)(1.2c)The solution to Eq. (1.2) is x~ = 0.6 cm, 2 =1. 0 cm, an d x3= 0.4cm, which can beverified by direct substitution.Systems of equations arise in all branches of engineering and science. Theseequations may be algebraic, transcendental (i.e., involving trigonometric, logarithmetic,exponential, etc. functions), ordinary differential equations, or partial differential equa-tions.The equations may be linear or nonlinear. Chapter 1 is devoted to the solution ofsystems of linear algebraic equations of the following form:allXt q--al2X2 q-al3x3 q-... q-alnXn 1= b (1.3a)a211x q-- az2x2 -+- az3x3 -~ ... q- a2nnX 2= b (1.3b)a,~lx~+ a ,,2x2+ a n3X"3~ - " " -}- a,,,,x,~ = b,~ (1.3n)wherex j (j = 1, 2 ..... n) denotes the unknowvna riables, ai, j (i,j = 1, 2 ..... n) denotesthe constant coefficients of the unknowvna riables, and bi (i = 1, 2 ..... n) denotes thenonhomogeneotuersm s. For the coefficients ai,j, the first subscript, i, denotes equation i,and the second subscript, j, denotes variable xj. The numbero f equations can range fromtwo to hundreds, thousands, and even millions.In the most general case, the numbero f variables is not required to be the samea sthe numbero f equations. However,i n most practical problems, they are the same. That isthe case considered in this chapter. Even when the number of variables is the same as thenumbeor f equations, several solution possibilities exist, as illustrated in Figure 1.2 for thefollowing system of two linear algebraic equations:a~ix~ + al2x2 = b~ (1.4a)azlx1 q- a22x2 2= b " (1.4b)The four solution possibilities are:1. A unique solution (a consistent set of equations), as illustrated in Figure 1.2a2. No solution (an inconsistent set of equations), as illustrated in Figure 1.2b3. Ani nfinite numbeor f solutions (a redundants et of equations), as illustratedFigure 1.2c4. The trivial solution, xj = 0 (j = 1,2 ..... n), f7or a homogeneousse t of equations, as illustrated in Figure 1.2dChapter t is concerned with the first case where a unique solution exists.Systems of linear algebraic equations arise in many different types of problems, forexample:1. Networkp roblems( e.g., electrical networks)2. Fitting approximating functions (see Chapter 4) 30. 20 Chapter 1x2(a) Uniques olution. (b) Nos olution../(c) Infinite numbeorf solutions.~(6) Trivial solution.xIFigure 1.2 Solution of a system of two linear algebraic equations.3. Systems of finite difference equations that arise in the numerical solution ofdifferential equations (see Parts II and III)The list is endless.There are two fundamentally different approaches for solving systems of linearalgebraic equations:1. Direct elimination methods2. Iterative methodsDirect elimination methods are systematic procedures based on algebraic elimination,whicho btain the solution in a fixed numbero f operations. Exampleso f direct eliminationmethodsa re Gauss elimination, Gauss-Jordane limination, the matrix inverse method, andDoolittle LUfactorization. Iterative methods, on the other hand, obtain the solutionasymptotically by an iterative procedure. A trial solution is assumed, the trial solution issubstituted into the system of equations to determine the mismatch, or error, in the trialsolution, and an improved solution is obtained from the mismatch data. Examples ofiterative methods are Jacobi iteration, Gauss-Seidel iteration, and successive-over-relaxa-tion(SOR).Although no absolutely rigid rules apply, direct elimination methods are generallyused when one or more of the following conditions holds: (a) The number of equationssmall (100 or less), (b) most of the coefficients in the equations are nonzero, (c) theof equations is not diagonally dominant [see Eq. (1.15)], or (d) the system of equationsill conditioned (see Section 1.6.2). Iterative methods are used when the numberequations is large and most of the coefficients are zero (i.e., a sparse matrix). Iterativemethods generally diverge unless the system of equations is diagonally dominant [seeEq. (1.15)1.The organization of Chapter 1 is illustrated in Figure 1.3. Following the introductorymaterial discussed in this section, the properties of matrices and determinants arereviewed. The presentation then splits into a discussion of direct elimination methods 31. Systemso f Linear Algebraic Equations 21Systemosf LinearAlgebraic EquationsPropertieosf Matricesand DeterminantsDirectMethodsIterativeMethods~.~ GaEulsims-iJnoartdioannGaussEliminationMatrixInverseLUFactodzationDeterminantsTridiagonalSystemsJacobiIteration~__.~ ACcocnuvrearcgyea nncdeGauss-SeidelIterationSuccessiveOverrelaxationPogramsSummaryFigure 1.3 Organization of Chapter 1.followed by a discussion of iterative methods. Several methods, both direct eliminationand iterative, for solving systems of linear algebraic equations are presented in this chapter.Procedures for special problems, such as tridiagonal systems of equations, are presented.All these procedures are illustrated by examples. Although the methods apply to largesystems of equations, they are illustrated by applying them to the small system of onlythree equations given by Eq. (1.2). After the presentation of the methods, three computerprograms are presented for implementing the Gauss elimination method, the Thomasalgorithm, and successive-over-relaxation (SOR). The chapter closes with a Summary,which discusses some philosophy to help you choose the right method for every problemand lists the things you should be able to do after studying Chapter 1.1.2 PROPERTIES OF MATRICES AND DETERMINANTSSystems of linear algebraic equations can be expressed very conveniently in terms ofmatrix notation. Solution methods for systems of linear algebraic equations can be 32. 22 Chapter 1developed very compactly using matrix algebra. Consequently, the elementary propertiesof matrices and determinants are presented in this section.1.2.1. Matrix DefinitionsA matrix is a rectangular array of elements (either numbers or symbols), which arearranged in orderly rows and columns. Each element of the matrix is distinct and separate.The location of an element in the matrix is important. Elements of a matrix are generallyidentified by a double subscripted lowercase letter, for example, ai,j, where the firstsubscript i identifies the row of the matrix and the second subscriptj identifies the columnof the matrix. The size of a matrix is specified by the numbero f rows times the numbero fcolumns. A matrix with n rows and m columns is said to be an n by m, or n x m, matrix.Matrices are generally represented by either a boldface capital letter, for example, A, thegeneral element enclosed in brackets, for example, [ai4], or the full array of elements, asillustrated in Eq. (1.5):A = [ai,j] =Iall a12 ...... aim(i=1,2 ..... n; j=l,2 ..... m)(1.5)ComparinEg qs. (1.3) and (1.5) showst hat the coefficients of a system of linear algebraicequations form the elements of an n n matrix.Equation (1.5) illustrates a convention used throughout this book for simplicityappearance. When the general element ai4 is considered, the subscripts i and j areseparated by a comma. When a specific element is specified, for example, a31, thesubscripts 3 and 1, which denote the element in row 3 and column 1, will not be separatedby a commau, nless i orj is greater than 9. For example,a 37 denotes the element in row 3and column 7, whereas a1~,17 denotes the element in row 13 and column 17.Vectors are a special type of matrix which has only one column or one row. Vectorsare represented by either a boldface lowercase letter, for example, x or y, the generalelement enclosed in brackets, for example, [xi] or [Yi], or the full column or row ofelements. A column vector is an n 1 matrix. Thus,x = [xi] = x2 (i = 1,2 ..... n) (1.6a)A row vector is a 1 x n matrix. For example,Y=[Yj]=[Yl Y2 "" Y,I (j=l,2 ..... n) (1.6b)Unit vectors, i, are special vectors which have a magnitude of unity. Thus,;2-~1/2 l (1.7)wheret he notation Ilill denotes the length of vector i. Orthogonals ystemso f pnit vectors,in which all of the elements of each unit vector except one are zero, are used to definecoordinate systems. 33. Systemso f Linear AlgebraicE quations 23There are several special matrices of interest. A square matrix S is a matrix whichhas the same number of rows and columns, that is, m = n. For example,Falla12 .. 1alnS = / .a~].,, .a.~., . : :., .a.~: / (1.8)[_ant an2 "" ann_]is a square n x n matrix. Our interest will be devoted entirely to square matrices. The left-to-right downward-slopinlgin e of elements from all to ann is called the major diagonal ofthe matrix. A diagonal matrix D is a square matrix with all elements equal to zero exceptthe elements on the major diagonal. For example,Fall0 0 i 1D = L i a2200 a3300 a44 (1.9)is a 4 x 4 diagonal matrix. The identity matrix I is a diagonal matrix with unity diagonalelements. The identity matrix is the matrix equivalent of the scalar number unity. ThematrixI---- 0 1 0 (1.10) 0 0 10 0 0is the 4 x 4 identity matrix.A triangular matrix is a square matrix in which all of the elements on one side of themajor diagonal are zero. The remaining elements may be zero or nonzero. An uppertriangular matrix U has all zero elements below the major diagonal. The matrixiall a12 a13 al4 1U =0 a22 a23 a24 ]0 0 a330 a34 / 0 0 a44 /(1.11)is a 4 x 4 upper triangular matrix. A lower triangular matrix L has all zero elements abovethe major diagonal. The matrixFall 0 0 0 ]a22 0 0(1.12)L~ /a31/a21 a32 a33 0L a41 a42 a43 a44is a 4 x 4 lower triangular matrix. 34. 24 Chapter 1A tridiagonal matrix T is a square matrix in which all of the elements not on themajor diagonal and the two diagonals surrounding the major diagonal are zero. Theelements on these three diagonals may or may not be zero. The matrix~all a12 0 0i ]/a2t a22 a23 0T = [ i0 a43 a44 a450 0 a54 a55 _]a32 a33 a34is a 5 x 5 tridiagonal matrix.A banded matrix B has allexample,all a12 0 a14 0 1a22 a23 0 a25a~lB = a32 a33 a34 01~ a~)0 a43 a44 a45a52 0 a54 a55(1.13)zero elements except along particular diagonals. For(1.14)is a 5 x 5 banded matrix.The transpose of an n x m matrix A is the m x n matrix, AT, which has elementsa. r. = aj, i. The transpose of a column vector, is a row vector and vice versa. Symmetricsquare matrices have identical corresponding elements on either side of the majordiagonal. That is, aid = aj, i. ITn. that case, A = AA sparse matrix is one in which most of the elements are zero. Most large matricesarising in the solution of ordinary and partial differential equations are sparse matrices.A matrix is diagonally dominant if the absolute value of each element on the majordiagonal is equal to, or larger than, the sum of the absolute values of all the other elementsin that row, with the diagonal element being larger than the correspondings umo f the otherelements for at least one row. Thus, diagonal dominanceis defined as[ai,il >_ ~ laid I (i = 1 ..... n) (1.15)with > true for at least one row.1.2.2. Matrix AlgebraMatrix algebra consists of matrix addition, matrix subtraction, and matrix multiplication.Matrix division is not defined. An analogous operation is accomplished using the matrixinverse.Matrix addition and subtraction consist of adding or subtracting the correspondingelements of two matrices of equal size. Let A and B be two matrices of equal size. Then,A + B = [aid] -[- [bid ] = [aid + bid] -~- [cid ] ~-- C (1.16a)A - B = [aid ] - [bi,j] = [aid - bid] ~- [cid ] = C (1.16b)Unequal size matrices cannot be added or subtracted. Matrices of the same size areassociative on addition. Thus,A + (B + C) = (A+B) (1.17) 35. Systemso f Linear AlgebraicE quationsMatrices of the same size are commutative on addition. Thus,A+B=B+AExample 1.1. Matrix addition.25(1.18)Add the two 3 x 3 matrices A and B to obtain the 3 x 3 matrix C, whereA = 1 4 and B = -4 1 24 3 2 3 -1(1.19)From Eq. (1.16a),Ci~j d= aid + bi (1.20)Thus, ell = all + bll = 1 + 3 = 4, 12 = a12 + b12 = 2 + 2 = 4, etc. The result isI(1+3)(2+2) (3+1)] I 4 4A+B = (2-4) (1+1) (4+2)|= -2 2 6(1+2) (4+3) (3-1)/ 3 7 2(1.21)Matrix multiplication consists of row-elementt o column-elemenmt ultiplication andsummationo f the resulting products. Multiplication of the two matrices Aa nd B is definedonly when the number of columns of matrix A is the same as the number of rows of matrixB. Matrices that satisfy this condition are called conformablein the order AB.T hus, if thesize of matrix A is n x m and the size of matrix B is m x r, thenAB = [aid][bid ] = [ci,j] = C ci~ j = ~ ai,kbkjk=l(i= 1,2 ..... n, j= 1,2 ..... r)(1.22)The size of matrix C is n x r. Matrices that are not conformable cannot be multiplied.It is easy to makee rrors whenp erformingm atrix multiplication by hand. It is helpfulto trace across the rows of A with the left index finger while tracing down the columns ofB with the right index finger, multiplying the corresponding elements, and summingth eproducts. Matrix algebra is much better suited to computers than to humans.Multiplication of the matrix A by the scalar ~ consists of multiplying each elementof A by ~. Thus,eA = e[aij ] = [eaid] = [bid] = B (1.23)Example 1.2. Matrix multiplication.Multiply the 3 x 3 matrix A and the 3 x 2 matrix B to obtain the 3 x 2 matrix C, whereA= 2 1 4 and B= 2 (1.24)1 4 3 1 36. 26 Chapter 1From Eq. (1.22),3Ci~[ = Z ai,kblq~ (i = 1, 2, 3, j = 1,2)k=lEvaluating Eq. (1.25) yieldsC~l = allbll + al2b21 + a13b31 = (1)(2) + (2)(1) + (3)(2)C12 = allbl2 -~ a12bz2 + a13b32 = (1)(1) + (2)(2) + (3)(1)C32- ~- a31b12+ a32b22+ a33b32= (1)(1) + (4)(2) + (3)(1)Thus,(1.25)(1.26a)(1.26b)(1.26c)C = [cij ] = 13 8 (1.27)12 12Multiply the 3 2 matrix C by the scalar ~ = 2 to obtain the 3 2 matrix D. FromEq. (1.23), dll = O~Cll = (2)(10) = 20, d12 = ~c12 = (2)(8) = 16, etc. The result0=~c=2c= (2)(13) (2)(8) = 26 16(2)(12) (2)(12)Matrices that are suitably conformablea re associative on multiplication. Thus,A(BC) = (AB)C (1.29)Square matrices are conformable in either order. Thus, if A and B are n x n matrices,AB = C and BA = Dwhere C and D are n n matrices. However squarecommutativeo n multiplication. That is, in general,AB -~ BA(1.30)matrices in general are not(1.31)Matrices A, B, and C are distributive if B and C are the sames ize and A is conformablet oB and C. Thus,A(B + C) = AB + AC (1.32)Consider the two square matrices A and B, Multiplying yieldsAB = C (1.33)It might appear logical that the inverse operation of multiplication, that is, division, wouldgiveA = C/B (1.34)Unfortunately, matrix division is not defined. However, for square matrices, an analogousconcept is provided by the matrix inverse. 37. Systemso f Linear Algebraic Equations 27Consider the two square matrices A and B. If AB = I, then B is the inverse of A,which is denoted as A-1. Matrix inverses commute on multiplication. Thus,AA-1 = A-1A = I (1.35)The operation desired by Eq. (1.34) can be accomplished using the matrix inverse.Thus, the inverse of the matrix multiplication specified by Eq. (1.33) is accomplishedmatrix multiplication using the inverse matrix. Thus, the matrix equivalent of Eq. (1.34)given byA = B-~C (1.36)Procedures for evaluating the inverse of a square matrix are presented in Examples1 .12and 1.16.Matrix factorization refers to the representation of a matrix as the product of twoother matrices. For example, a known matrix A can be represented as the product of twounknown matrices B and C. Thus,A ----- BC (1.37)Factorization is not a uniquep rocess. Therea re, in general, an infinite numbeor f matricesB and C whose product is A. A particularly useful factorization for square matrices isA = LU (1.38)where L and I5 are lower and upper triangular matrices, respectively. The LU factorizationmethod for solving systems of linear algebraic equations, which is presented in Section1.4, is based on such a factorization.A matrix can be partitioned by grouping the elements of the matrix into submatrices.These submatrices can then be treated as elements of a smaller matrix. To ensure that theoperations of matrix algebra can be applied to the submatrices of two partitioned matrices,the partitioning is generally into square submatrices of equal size. Matrix partitioning isespecially convenient whens olving systems of algebraic equations that arise in the finitedifference solution of systems of differential equations.1.2.3. Systems of Linear Algebraic EquationsSystems of linear algebraic equations, such as Eq. (1.3), can be expressed very compactlyin matrix notation. Thus, Eq. (1.3) can be written as the matrix equation~ (1.39)whereFalla12 "" alnlA : ::: ...a/.2.nL an 1 an2 " " " ann ~JEquation (1.3) can also be written~ ai,jx j = bi (i : 1 ..... n)j=l(1.40)(1.41) 38. 28 Chapter1or equivalently asai,ix j = bi (i,j = 1 ..... n) (1.42)where the summation convention holds, that is, the repeated index j in Eq. (1.42)summedo ver its range, 1 to n. Equation (1.39) will be used throughout this bookrepresent a system of linear algebraic equations.There are three so-called row operations that are useful when solving systems oflinear algebraic equations. They are:1. Any row (equation) may be multiplied by a constant (a process knownscaling).2. The order of the rows (equations) may be interchanged (a process knownpivoting).3. Any row (equation) can be replaced by a weighted linear combination of thatrow (equation) with any other row (equation) (a process known eli mination).In the context of the solution of a system of linear algebraic equations, these threerow operations clearly do not change the solution. The appearance of the system ofequations is obviously changed by any of these row operations, but the solution isunaffected. When solving systems of linear algebraic equations expressed in matrixnotation, these row operations apply to the rows of the matrices representing the systemof linear algebraic equations.1.2,4. DeterminantsThe term determinant of a square matrix A, denoted det(A) or IAI, refers to both thecollection of the elements of the square matrix, enclosed in vertical lines, and the scalarvalue represented by that array. Thus,all a12 .. alndet(A) IAI = a2 1 a22 "" azn (1.43)anl an2 " " " annOnly square matrices have determinants.The scalar value of the determinant of a 2 2 matrix is the product of the elementson the major diagonal minus the product of the elements on the minor diagonal.Thus,det(A) =IAI ~__all a12 = alla22 -- a21a12 (1.44)a21 a22The scalar value of the determinanto f a 3 3 matrix is composedo f the sumo f six tripleproducts which can be obtained from the augmented determinant:all a12 a13 all a12a21 a22 a23 a21 a22 (1.45)a31 a32 a33 a31 a32The 3 3 determinant is augmentedb y repeating the first two columnso f the determinanton the right-hand side of the determinant. Three triple products are formed, starting withthe elements of the first row multiplied by the two remaining elements on the right- 39. Systemso f Linear AlgebraicE quations 29downward-slopingd iagonals. Three more triple products are formed, starting with theelements of the third row multiplied by the two remaining elements on the right-upward-slopingdiagonals. The value of the determinant is the sum of the first three triple productsminus the sum of the last three triple products. Thus,det(A) IAI = al la22a33 + a12a23a31 + a13a21a32-- a31a22a13- - a32a23all -- a33a21a12 (1.46)Example 1.3. Evaluation of a 3 x 3 determinant by the diagonal method.Lets evaluate the determinant of the coefficient matrix of Eq. (1.2) by the diagonalmethod. Thus,80 -20A = -20 40 -20 (1.47)-20 -20 130The augmented determinant is80 -20 -20 80 -20-20 40 -20 -20 40 (1.48)-20 -20 130 -20 -20Applying Eq. (1.46) yieldsdet(A) IAI = (80)(40)(130) + (- 20)(-20)(-20) + (-- (-20)(40)(-20) - (-20)(-20)(80)- (130)(-20)(-20) = 416,000 - 8,000 -- 16,000 - 32,000 - 52,000 = 300,000 (1.49)The diagonal method of evaluating determinants applies only to 2 x 2 and 3 x 3determinants. It is incorrect for 4 x 4 or larger determinants. In general, the expansion ofan n x n determinant is the sum of all possible products formed by choosing one and onlyone element from each row and each column of the determinant, with a plus or minus signdetermined by the number of permutations of the row and column elements. One formalprocedure for evaluating determinants is called expansion by minors, or the method ofcofactors. In this procedure there are n! products to be summedw, heree ach product has nelements. Thus, the expansion of a 10 x 10 determinant requires the summation of 10!products (10! = 3,628,800), where each product involves 9 multiplications (the product10 elements). This is a total of 32,659,000 multiplications and 3,627,999 additions, notcounting the work needed to keep track of the signs. Consequently, the evaluation ofdeterminantsb y the methodo f cofactors is impractical, except for very small determinants.Although the method of cofactors is not recommendefdo r anything larger than a4 x 4 determinant, it is useful to understand the concepts involved. The minor Mij is thedeterminant of the (n - 1) x (n - 1) submatrix of the n x n matrix A obtained by deletingthe ith row and the jth column. The cofactor Aij associated with the minor Mi~ isdefined asAi, j = (--1)i+Jmi,j (1.50) 40. 30 Chapter 1Using cofactors, the determinant of matrix A is the sum of the products of the elements ofany row or column, multiplied by their corresponding cofactors. Thus, expanding acrossany fixed row i yieldsn ndet(A) = IAI = ai jAi, j = ~- ~(-1)i+JaijMi,j (1.51)Alternatively, expanding down any fixed column j yieldsdet(A) = IAI = ai,iAi J = ~(- 1) i+JaijMid (1.52)i=1 i=1Each cofactor expansion reduces the order of the determinant by one, so there are ndeterminants of order n- 1 to evaluate. By repeated application, the cofactors areeventually reduced to 3 3 determinants which can be evaluated by the diagonalmethod. The amount of work can be reduced by choosing the expansion row or columnwith as many zeros as possible.Example 1.4. Evaluation of a 3 3 determinant by the cofactor method.Lets rework Example1 .3 using the cofactor method. Recall Eq. (1.47):A = -20 40 -20-20 -20 130Evaluate IAI by expanding across the first row. Thus,IA] (=80 ) -20 40 1-3200_ (-20) -20-20130 (-20)-20 + -20 -20[AI = 80(5200 400) - (-20)(-2600 + 400) + (-20)(400= 384000 - 60000 - 24000 = 300000(1.53)(1.54)(1.55)If the value of the determinanto f a matrixi s zero, the matrix is said to be singular. Anonsingular matrix has a determinant that is nonzero. If any row or column of a matrix hasall zero elements, that matrix is singular.The determinant of a triangular matrix, either upper or lower triangular, is theproduct of the elements on the major diagonal. It is possible to transform any nonsingularmatrix into a triangular matrix in such a way that the value of the determinant is eitherunchangedo r changedi n a well-defined way. That procedure is presented in Section 1.3.6.The value of the determinant can then be evaluated quite easily as the product of theelements on the major diagonal.1.3 DIRECT ELIMINATION METHODSThere are a number of methods for the direct solution of systems of linear algebraicequations. One of the more well-known methods is Cramers rule, which requires theevaluation of numerousd eterminants. Cramersr ule is highly inefficient, and thus notrecommended. More efficient methods, based on the elimination concept, are recom- 41. Systemso f Linear Algebraic Equations 31mended. Both Cramers rule and elimination methods are presented in this section. Afterpresenting Cramersr ule, the elimination concept is applied to develop Gausse limination,Gauss-Jordan elimination, matrix inversion, and determinant evaluation. These conceptsare extended to LU factorization and tridiagonal systems of equations in Sections 1.4 and1.5, respectively.1.3.1. Cramers RuleAlthough it is not an elimination method, Cramers rule is a direct method for solvingsystems of linear algebraic equations. Consider the system of linear algebraic equations,Ax = b, which represents n equations. Cramers rule states that the solution forxy (j = 1 ..... n) is givenjd)et(Axj- det(A) (j = 1 ..... n) (1.56)where Aj is the n x n matrix obtained by replacing column j in matrix A by the columnvector b. For example, consider the system of two linear algebraic equations:allX1 t-t- a12x2= b (1.57a)a21xI 2+ a22x2- -- b (1.57b)Applying Cramers rule yieldsb1]a12 all blb2 a22] a2 b2xt -all (1.58)a12 and x2 -- lall-----1 a12a21 a22 [ a21 a22The determinants in Eqs. (1.58) can be evaluated by the diagonal method describedSection 1.2.4.For systems containing more than three equations, the diagonal method presented inSection 1.2.4 does not work. In such cases, the method of cofactors presented in Section1.2.4 could be used. The number of multiplications and divisions N required by themethod of cofactors is N = (n - 1)(n + 1)!. For a relatively small system of 10 equations(i.e., n = 10), N = 360,000,000, which is an enormous number of calculations. Forn = 100, N = 10157, which is obviously ridiculous. The preferred method for evaluatingdeterminants is the elimination method presented in Section 1.3.6. The number ofmultiplications and divisions required by the elimination method is approximatelyN = n3 n 2 -- n. Thus, for n = 10, N = 1090, and for n = 100, N = 1,009,900.Obviously, the elimination method is preferred.Example 1.5. Cramers rule.Lets illustrate Cramer;s rule by solving Eq. (1.2). Thus,80X1 -- 20x2 -- 20x3 = 20--20x1 + 40x2 -- 20x3 = 20--20x1 -- 20x2 + 130x3 = 20(1.59a)(1.59b)(1.59c) 42. 32 Chapter1First, calculate det(A). FromE xample1 .4,det(A)80 -20 -20-20 40 -20-20 -20 130= 300,000Next, calculate det(A1), 1d)e,t(A2), and det(A3). For det(Aii-20 -20det(A l) = 20 40 -20-20 130(1.60)= 180,000 (1.61)In a similar manner, det(A2) = 300,000 and det(A3) = 120,000. Thus,det(A ~) 180,000 300,000 120,000x~ det(A) - 300,000 - 0.60 x2 - 300,000 - 1.00 x3 - 300,000 -- 0.40(1.62)1.3.2. Elimination MethodsElimination methods solve a system of linear algebraic equations by solving one equation,say the first equation, for one of the unknowns, say x~, in terms of the remainingunknowns, x2 to x,, then substituting the expression for x1 into the remaining n- 1equations to determine n - 1 equations involving x2 to xn. This elimination procedure isperformedn - 1 times until the last step yields an equation involving only x,. This processis called elimination.The value of xn can be calculated from the final equation in the eliminationprocedure. Then x,_~ can be calculated from modified equation n - 1, which containsonly x, and x~_l. Then x,_2 can be calculated from modified equation n- 2, whichcontains only x,, x,_~, and x,_2. This procedurei s performedn - 1 times to calculate x,_~to x~. This process is called back substitution.1.3.2.1. Row OperationsThe elimination process employs the row operations presented in Section 1.2.3, which arerepeated below:1. Any row (equation) may be multiplied by a constant (scaling).2. The order of the rows (equations) may be interchanged (pivoting).3. Any row (equation) can be replaced by a weighted linear combination of thatrow (equation) with any other row (equation) (elimination).These row operations, which change the values of the elements of matrix A and b, do notchange the solution x to the system of equations.The first row operation is used to scale the equations, if necessary. The second rowoperation is used to prevent divisions by zero and to reduce round-off errors. The third rowoperation is used to implement the systematic elimination process described above. 43. Systemso f Linear Algebraic Equations 331.3.2.2. EliminationLets illustrate the elimination methodb y solving Eq. (1.2). Thus,80xI - 20x2 - 20x3 = 20 (1.63a)-20xI + 40x2 - 20x3 = 20 (1.63b)-20x1 - 20x2 + 130x3 = 20 (1.63c)Solve Eq. (1.63a) for xl. Thus,xt = [20 - (-20)x2 - (-20)x3]/80 (1.64)Substituting Eq. (1.64) into Eq. (1.63b) gives-20{[20 - (-20)x 2 - (-20)x3]/80 } + 40x2 - 20x3 = 20 (1.65)which can be simplified to give35x2 - 25x3 = 25 (1.66)Substituting Eq. (1.64) into Eq. (1.63c) gives-20{[20 - (-20)x2 - (-20)x3]/80} - 20x2 + 130x3 -- 20 (1.67)which can be simplified to give-25x~ + 125x3 = 25 (1.68)Next solve Eq. (1.66) for 2. Thus,x2 = [25 - (-25)x3]/35 (1.69)Substituting Eq. (1.69) into Eq. (1.68) yields-25{[25 - (-25)x3]/35 } + 125x3 = 25 (1.70)which can be simplified to give"~9"X 3 = ~ (1.71)Thus, Eq. (1.63) has been reduced to the upper triangular system80x1 - 20x2 - 20x3 -- 20 (1.72a)35x2 - 25x3 --- 25 (1.72b)7_570 X--3 3070 (1.72c)which is equivalent to the original equation, Eq. (1.63). This completes the eliminationprocess.1.3.2.3. Back SubstitutionThe solution to Eq. (1.72) is accomplished easily by back substitution. Starting with Eq.(1.72c) and working backward yieldsx3 = 300/750 = 0.40 (1.73a)x2 = [25 - (-25)(0.40)]/35 = (1.73b)Xl = [20 - (-20)(1.00) - (-20)(0.40)]/80 (1.73c) 44. 34Example 1.6. Elimination.Chapter 1Lets solve Eq. (1.2) by elimination. Recall Eq. (1.2):80xZ - 20x2 - 20x3 = 20-20x1 + 40x2 - 20x3 = 20-20xl - 20x2 + 130x3 = 20(1.74a)(1.748)(1.74c)Elimination involves normalizing the equation above the element to be eliminated by theelement immediately above the element to be eliminated, which is called the pivot element,multiplying the normalized equation by the element to be eliminated, and subtracting theresult from the equation containing the element to be eliminated. This process system-aticallyeliminates terms below the major diagonal, column by column, as illustratedbelow. The notation Ri - (em)Rj next to the ith equation indicates that the ith equation isbe replaced by the ith equation minus em times the jth equation, where the eliminationmultiplier, em, is the quotient of the element to be eliminated and the pivot element.For example, R2 -(-20/40)R~ beside Eq. (1.75.2) below means replace(1.75.2) by Eq. (1.75.2)-(-20/40)xEq. (1.75.1). The elimination multiplier,em = (-20/40), is chosen to eliminate the first coefficient in Eq. (1.75.2). All ofcoefficients below the major diagonal in the first columaa are eliminated by linearcombinations of each equation with the first equation. ,Thus,I80x 1-20xz- 230=x201-20x1 -t- 40x2 - 20x3 = 20J R2 -1 (-20/80)R-20xI - 20x2 -t- 135x3 = 20 R3 - (-20/80)R=(1.75.1)(1.75.2)(1.75.3)The result of this first elimination step is presented in Eq. (1.76), which also showsthe elimination operation for the second elimination step. Next the coefficients below themajor diagonal in the second column are eliminated by linear combinations with thesecond equation. Thus,I80x - 20x2 - 20x3 = 2070x1 + 35x2 25x 3 250x1 25x2 + 125x3 25 R3 -2 (-25/35)R(1.76)The result of the second elimination step is presented in Eq. (1.77):I28-0x~ -20x 20x3= 201z0x~+35x 25x3= 25Oxi+ Ox2+750/7x3=300/7(1.77)This process is continued until all the coefficients below the major diagonal are eliminated.In the present example with three equations, this process is now complete, and Eq. (1.77)is the final result. This is the process of elimination.At this point, the last equation contains only one unknown, x3 in the presentexample, which can be solved for. Using that result, the next to last equation can be solved 45. Systemso f Linear Algebraic Equations 35for x2. Using the results for x3 and x2, the first equation can be solved for x~. This is theback substitution process. Thus,x3 = 300/750 = 0.40 (1.78a)x2 = [25 - (-25)(0.40)]/35 = (1.78b)x~ = [20 - (-20)(1.00) - (-20)(0.40)/80 (1.78c)The extension of the elimination procedure to n equations is straightforward.1.3.2.4. Simple EliminationThe elimination procedure illustrated in Example 1.6 involves manipulation of thecoefficient matrix A and the nonhomogeneouvse ctor b. Componentso f the x vector arefixed in their locations in the set of equations. As long as the colurnns are not interchanged,column j corresponds to x/. Consequently, the xj notation does not need to be carriedthroughout the operations. Only the numerical elements of A and b need to be considered.Thus, the elimination procedure can be simplified by augmenting the A matrix with the bvector and performing the row operations on the elements of the augmented A matrix toaccomplish the elimination process, then performing the back substitution process todetermine the solution vector. This simplified elimination procedure is illustrated inExample 1.7.Example 1.7. Simple elimination.Lets rework Example 1.6 using simple elimination. From Example 1.6, the A matrixaugmented by the b vector is80 -20 -201I 21[AIb] = -20 40 -201 20-20 -20 130120Performing the row operations80 -20 -20120-20 40 -20120-20 -20 130120to accomplish the elimination process yields:IR - (-20/80)R~R3 -1- (-20/80)R(1.79)(1.80)iI8-2035 -25125-25 -125125 20120183 -- (2-25/35)R(1.81)I8 i -20-20120JXl = [20 - (-20)(100) - (-20)(0"40)]/8035 = 0.60 -25125 --~0 750/71300/7x2=[25-(-25)(0.4)]/35 =x3 = 300/750 = 0.40(1.82)The back substitution step is presented beside thetfiangulafized augmented Amatfix. 46. 36 Chapter11.3.2.5. Multiple b VectorsIf moret han one b vector is to be considered, the A matrix is simply augmentedb y all ofthe b vectors simultaneously. The elimination process is then applied to the multiplyaugmented A matrix. Back substitution is then applied one column at a time to themodified b vectors. A more versatile procedure based on matrix factorization is presentedin Section 1.4.Example 1.8. Simple elimination for multiple b vectors.Consider the system of equations presented in Example 1.7 with two b vectors,bl ~ = [20 20 20] and b2~ = [20 10 20]. The doubly augmented A matrix isI 80 -20 -20[20120J[Alb 1 b2] = -20 40 -20120 I 10 (1.83)-20 -20 130120120Performing the elimination process yields-20 -20 [ 2035 -25 I 25 150 750/7 1300/7 250/7201(1.84)Performing the back substitution process one column at a time yieldsxl = 1.00 and x2 = /2/3|0.40 0.60]L 1/3 ](1.85)1.3.2.6. PivotingThe element on the major diagonal is called the pivot element. The elimination proceduredescribed so far fails immediatelyif the first pivot elementa lx is zero. The procedurea lsofails if any subsequent pivot element ai, i is zero. Even though there may be no zeros on themajor diagonal in the original matrix, the elimination process may create zeros on themajor diagonal. The simple elimination procedure described so far must be modified toavoid zeros on the major diagonal. This result can be accomplished by rearranging theequations, by interchanging equations (rows) or variables (columns), before each elimina-tionstep to put the element of largest magnitude on the diagonal. This process is calledpivoting. Interchanging both rows and columns is calledfullpivoting. Full pivoting is quitecomplicated, and thus it is rarely used. Interchanging only rows is called partialpivoting.Only partial pivoting is considered in this book.Pivoting eliminates zeros in the pivot element locations during the eliminationprocess. Pivoting also reduces round-off errors, since the pivot element is a divisor duringthe elimination process, and division by large numbersi ntroduces smaller round-off errorsthan division by small numbers. Whenth e procedure is repeated, round-off errors cancompound. This problem becomes more severe as the number of equations is increased. 47. Systemso f Linear Algebraic EquationsExample1 .9. Elimination with pivoting to avoid zero pivot elements.37Use simple elimination with partial pivoting to solve the following system of linearalgebraic equations, Ax = b:4 t -1 x2 = -3 (1.86)-2 3 -3 x 3 5Lets apply the elimination procedure by augmenting A with b. The first pivot element iszero, so pivoting is required. The largest number( in magnitude)i n the first columnu nderthe pivot element occurs in the second row. Thus, interchanging the first and second rowsand evaluating the elimination multipliers yields0 2 1 I R2- (0/4)R 1 (1.87)-2 3 -3 I R3 - (-2/4)R1Performing the elimination operations yields2 l 15 (1.88)7/2 -7/2 [7/2Althoughth e pivot elementi n the secondr owi s not zero, it is not the largest elementi n thesecond column underneath the pivot element. Thus, pivoting is called for again. Note thatpivoting is based only on the rows below the pivot element. The rows above the pivotelement have already been through the elimination process. Using one of the rows abovethe pivot element would destroy the elimination already accomplished. Interchanging thesecond and third rows and evaluating the elimination multiplier yields7/2 -7/217/2 (1.89)2 1 I 5 R3 -2- (4/7)RIi 1 -111-3Performing the elimination operation yields7/2 -7/2 ] --~ x2 : 2 (1.90)0 313 ] x3=lThe back substitution results are presented beside the triangularized augmentedA matrix.1.3.2.7. ScalingThe elimination process described so far can incur significant round-off errors when themagnitudes of the pivot elements are smaller than the magnitudes of the other elements inthe equations containing the pivot elements. In such cases, scaling is employedto selectthe pivot elements. After pivoting, elimination is applied to the original equations. Scalingis employedo nly to select the pivot elements.Scaled pivoting is implementeads follows. Beforee limination is applied to the firstcolumn,a ll of the elementsi n the first columna re scaled (i.e., normalized)b y the largestelements in the correspondingr ows. Pivoting is implementedb ased on the scaled elements 48. 38 Chapter 1in the first column, and elimination is applied to obtain zero elements in the first columnbelow the pivot element. Before elimination is applied to the second column, all of theelements from 2 to n in column2 are scaled, pivoting is implementeda, nd elimination isapplied to obtain zero elements in column 2 below the pivot element. The procedure isapplied to the remaining rows 3 to n - 1. Back substitution is then applied to obtain x.Example 1.10. Elimination with scaled pivoting to reduce round-off errors.Lets investigate the advantage of scaling by solving the following linear system:-3 = (1.91)1which has the exact solution x1 = -1.0, x2 = 1.0, and x3 = 1.0. To accentuate the effectsof round-off, carry only three significant figures in the calculations. For the first column,pivoting does not appear to be required. Thus, the augmentedA matrix and the first set ofrow operations are given by2 105 I 104"]-13 103 I 98| R: -(0.667)R (1.92)3 ]3 ] R3 -1- (0.333)Rwhich gives-4.330.3341051 104 ~33.01 28.6| (1.93)-32.0 I -31.6] R3 -2 (-0.0771)RPivoting is not required for the second column. Performing the elimination indicated inEq. (1.93) yields the triangularized matrix-