numerical algorithms - university of massachusetts amherstjohnston/m551f15/chapt1.pdf · numerical...
TRANSCRIPT
-
Preface
, graduate schoolso use our more
putational errorslofi‘error, which -2 systematically>r 3 and Chaptersthe iteration, or
e main focus in
B and expect the
[ringthe material1d programming5 tutorial.
.ercise numbered
ating the process
nationto answer
:her. As a result,iettle for readingursue.
over the years to
11input is simply
‘3.
Chapter 1
Numerical Algorithms
This opening chapter introduces the basic concepts ofnumerical algorithms and scientific comput—
ing.
more substantial Sections 1.2 and 1.3. Section 1.2 discusses the basic errorsthat may be encountered
when applying numerical algorithms. Section 1.3 isconcerned with essential properties of such
algorithms and the appraisal of the results they produce.We get to the “meat” of the material in later chapters.
1 1.1 Scientific computing. Scientific cdmputing is a discipline concerned with the development
and study of numerical al-
gorithms for solvingmathematical problems that arise in various disciplinesin science and engin—
eering.»
Typically, the starting point is a given mathematicalmodel which has been formulated in
an attempt to explain and understand an observed phenomenonin biology, chemistry, physics, eco—
nomics, or any other scientific or engineering discipline. Wewill concentrate on those mathematical
models which are continuous (or piecewise continuous) and are difficult or impossibleto solve ana—
lytically; this is usually the case in practice. Relevant applicationareas within computer science and
related engineering fields include graphics, vision andmotion analysis, image and signal processing,
1
search engines and data mining, machine learning, and hybridand embedded systems.
In order to solve such a model approximately on a computer, the continuousor piecewise
[continuous problem is approximated by a discreteone. Functions are approximated by finite arrays
of values. Algorithms are then sought which approximatelysolve the mathematical problem effi—
ciently, accurately, and reliably. This is the heartof scientific computing. Numerical analysis may
be viewed as- the theory behind such algorithms.The next step after devising suitable algorithms is their implementation.
This leads to ques—
tions involving programming languages, data structures, computingarchitectures, etc. The big pic—
ture is depictedin Figure 1.1. ,.
'
' “i‘
The set of requirements that good scientific computing algorithmsmust satisfy, which seems
’
lementary and obvious, may actually pose ratherdifficult and complex practical challenges. The
t»
main purpose of this book is to equip you withbasic methods and analysis tools for handling such
7
challengesas they arise in futureendeavors.
~
. a
We begin with a general, brief introduction to the field inSection 1.1. This is followed by the
I
l
ill
lll
1ll
-
Chapter 1. Numerical Algorithms
Figure 1.1. Scientificcomputing.
Problem solving environment
1, we will be using MATLAB:this is an interactive computer language,
which
(1 as a convenient problem solving environment.V
guage based on simple data arrays;it is truly a complete
environment. Its interactivity and graphics capabilitiesmake it more suitable and convenientin our
context than general-purpose languages suchas C++, Java, Scheme, or
Fortran 90. In fact,
many of the algorithms that we Willlearn are already implemented in MATLAB. ..
So why learn
them at all? Because they provide the basis for muchmore complex tasks, not quite availablelthat
is to say, not already solved) in MATLABor anywhere else, which you may
encounter in the future.
Rather than producing yet another MATLABtutorial or introduction in this text (there
are
several very good ones available in othertexts as well as on the Internet) we will
demonstrate the
use of this language on examples as we go along.
As a computing too
for our purposes may best beviewe
MATLAB is much more than a 1an
-
arical Algorithms‘
language, whichM
3:truly a complete
convenient in our
gran 90. In fact,So why learn
ite available (thatinter in the future.
iris text (there are
1 demonstrate the
iriNumerical algorithms and errors
'
NUmerical algorithms and errors
The most fundamental feature of numerical computing is the inevitable presenceof error. The result
ofany interesting computation (andof many uninteresting ones) is typically only approximate,
and
ur goal is to ensure that the resulting error is tolerablysmall.
'
‘atiVe and absolute errors
,
ere-are in general two basic types of measured error.Given a scalar quantity u and its approxima-
tion‘vt,
:- The absolute error in. u is‘ In a vl.
o The relative error (assuming it 750) is
lu—vllul
fiTherelative error is usually a more meaningful measure. Thisis especially true for errors in floating
point representation, a point to which we returnin Chapter 2. For example, we record absolute
and
_I'relativeerrors for various hypotheticalcalculations in the following table:
_________—___—’——
‘u 1) Absolute Relative
error error
1 0.99 0.01 0.01
q
1 1.01 0.01 0.01i
—1.5 —1.2 0.3 0.2
100 99.99 0.01 0.0001
100 99 l 0.01
Evidently, when |u| % 1 there is not much differencebetween absolute and relative error measures.
But when lul >> 1, the relative error is more meaningful.In particular, we expect the approximation
in the last row of the above table to be similarin quality to the one in the first row. This expectation
is borne out by the value of the relative error butis not reflected by the value of the absolute error.
When the approximated value is small in magnitude, thingsare a little more delicate, and here
is where relative errors may not be so meaningful. But letus not worry about this at this early point.
Example 1.1. The Stirling approximationn n.
v=Sn= 27rn-(—>eis used to approximateu = n! = 1 -2 - - -n for large n. The formula involves
the constant e = exp(1) =
'
2.7182818 The following MATLAB script computesand displays 11! and S”, as well as their
absoluteand relative differences,for 1 ___ n g 10:
I
‘3‘e:=exp(1);m.n=1£10;
' % array
Sn=sqrt (2*pi*n) .* ( (n/e) .An) ;7% the Stirling approximation.
-
\ vectors or matrices. Finally, our printing instructions (thelast two in the script) are a bit primitive
4Chapter 1. Numerical Algorithms
fact_n=factoria1(n);abs_err=abs(Sn—fact_n);re1_err=abs_err./fact_n;format short 9
,
[n; fact_n; Sn; abs_err; re1_err]’ % printout Values
instance5, Thus, it
,-_,_'_.resultin1§
At the 1';"3.....nothing
consider;two typi‘,
Approx
% absolute error
% relative error
Given that this is our first MATLAB script, let us providea few additional details, though we
hasten to add that we will not make a habitout of this. The commands exp, factorial,
and abs
use built—in'functions. The command n=1 : 10 (along with a semicolon,which simply suppresses
screen output) defines an array of length10 containing the integers 1,2,. . ., 10.
This illustrates a
fundamental concept in MATLAB of workingwith arrays whenever possible. Along with
it come
array operations: for example, in the thirdline “.*” corresponds to elementwise!multiplication
of Such er
_
i I_ .
.
evaluate
here, a sacrifice made for the sake of srmphcltyin this, our first program.
*
The resulting output is
we Will I
1 1 0.922140.077863 0.077863
1
2 21.919 0.080996 0.040498
3 6 5.83620.16379 0.027298
4 24 23.5060.49382 0.020576
5 120 118.021.9808 0.0165071
6 720 710.089.9218 0.01378
7 5040 4980.459.604 0.011826 fin
8 40320 39902417.6 0.010357
9 3.6288e+005 3.5954e+0053343.1 0.0092128 Round(
10 3.6288e+006 3.5987e+00630104 0.008296
1 Any cor
The values of n! become very large very quickly,and so are the values of the approximation error
is
S”. The absolute errors grow as n grows,but the relative errors stay well behaved and
indicate that elimme
in fact the larger 11 is, the better the quality of the approximationis. Clearly, the relative errors are
of the fil
much more meaningful as a measure of the qualityof this approximation. I represer
Discreti:
Error types,
and we will 8
Knowing how errors are typically measured,we now move to discuss their source;
There are seVeral ,..structure whk
types of error that maylimit the accuracy of a numerical calculation.
‘
errors domina
1. Errors in the problem to be solved.‘
. ..
.
.
» Theorer
These may be approxrmation errorsin the mathematical model. For instance: Assume
o Heavenly bodies are often approximatedby sphereswhen calculating their properties;
Then
an example here is the approximate calculationof their motion trajectory, attempting
to
answer the question (say) whether a particularasteroid will collide with Planet Earth
before 11.12.2016.
“3“
o Relatively unimportant chemicalreactions are often discarded in complex
chemical
modeling in order to obtain a mathematical problemof a manageable size. I. ,. _, ‘
. .
.V
. .
.
1 where‘g
It IS important to realize, then,that often approx1mat10n errors of
the type stated aboveare
deliberately made: the assumption isthat simplificationof the problem is worthwhile
even if
it generates an error in themodel. Note, however, that we are
still talking about the math- Discretizafi‘
ematical model itself; approximationerrors relatedto the numerical solution of the problem _
Let us show 2
are discussed below.
-
etical Algorithms‘
2% Numerical algorithms and errors
1‘ 1Another-typical sourceof error in the problem is error in the input data.
This may arise, for
"5 instance, from physical measurements, which are never infinitelyaccurate.
,
. Thus, it may be that after a careful numericalsimulation of a given mathematical problem, the
b;_ resulting solution wouldnot quite match observations on the phenomenonbeing
examined.
Atthe level of numerical algorithms, which is the focusof our interest here, there is really,
l. nothing we can do about the above-described
errors. Nevertheless, they should be taken into
consideration, for instance, when determiningthe accuracy (tolerance with respect to
the next
two types of error mentioned below)to which the numerical problem should be solved.
details,though we:orial, and abs
simply suppressesThis illustrates: a
dong with it come
:multiplic‘atidnof“are a bit primitive
.
2. Approximation errors
Such errors arise when an approximate formula is usedin place of the actual function to be
evaluated.'
We will often encounter two types of approximationerrors:
0 Discretization errors arise from discretizations ofcontinuous processes, such as inter-
polation, differentiation, and integration.
0 Convergence errors arise in iterativemethods. For instance, nonlinear problems must
generally be solved approximatelybyan iterative process. Such a process
would con—
verge, to the exact solutionin infinitely many iterations, but we cut it off
after a finite
(hopefully small!) number of suchiterations. Iterative methods in fact often
arise in
linear algebra.
(3. Roundofferrors
error is produced (as in the direct evaluationof a straight line, or the solution by Gaussian
elimination of a linear system of equations), roundofferrors are present. These arise because
of the finite precision representation of real numberson any computer, which affects
both data
representation and computer arithmetic.‘
the approximation:1and indicate that
j relative errors are'
Discretization and convergence errors maybe assessed by an analysis of the method used,
and We will see a lot of that in thistext. Unlike roundoff errors, they have a relatively'smooth
structure which may occasionally be exploited. Ourbasic assumption will be that approximation
errors dominate roundoff errors in magnitude in actual,successful calculations._.
There are several
Theorem:Taylor Series. ,Assume that f (x) has k +1
derivatives in an interval containing the points x0and x0 + h.
Then '5
‘;ince:
ig their properties;
izory,attempting tohz hk
iWlth PlanetEarth f(xo + h) = f(xo) + hf'(xo) + —2-f”(xo>
+ - - - + g f(k)(xo)
I
I»I
ilk“ (k-ll—l)’
yzomplex chemical+ ——--f . (S),
:,
l
t i
(k 1).l
,,size.
l, Stated above arewhere; is some point between x0
and x0 + h. .
vorthwhile even if
g about the math— r- Discretization errors in actionon of the problem
-
\
Let us show an example that illustrates the behaviorof discretization errors.
4“
Any computation with real numbersinvolves roundoff error. Even when no approximation
-
6 - _ Chapter 1. Numerical Algorithms
'
Example 1.2. Consider the problem of approximatingthe derivative
'
f’(x0) of a given smooth
function f (x) at the point x = x0. For instance, let f (x)= sin(x) be defined on the real line
—00 < x < 00, and set x0 = 1.2. Thus, f(x0) = sin(1.2)% 0.932....
)
,
Further, consider a situation where f (x) may be evaluatedat any point x near x0, but f’ (x0)
may not be directly available or is computationally expensiveto evaluate. Thus, we seek ways to
;
approximate f’(x0) by evaluating f at x near x0.
I
.
A simple algorithm may be constructed'using. Taylor’s series.This fundamental theorem is
given on the preceding page. For some small, positivevalue h that we will choose in a moment, write
/’12 ll [’13 Ill II/l
f(Xo +11) = f(xo) +hf (X0) + if (x0) + —6-f(x0) + if (xo)+- “-
Then
h 5
Our algorithm for approximating f’ (x0) is to calculate
f(x0 +h) — f(x0)h
'
The obtained approximation has the discretization error
1,13I — h ‘12x
f/(XQ)__f(x0+ f(X0)
Ef//(x'0)+l6_f///(XO)+flfII/I(XO)+I _ . ‘-
Geometrically, we approximate the slope of the tangentat the point x0 by the slope of the chord
through neighboring points of f. In Figure 1.2,the tangent is in blue and the chord
is in red.
h __ ] h2 h3_
:
f/(XO)=f(X0+ ) f(x0).—
-
nerical Algorithm Numerihalalgorithms and errors7
is bbunded,and it is @(h) if also f’ ’
(x0) # O. In any case, even without knowing f’ ’
(x) we
the discretizationerror to decrease at leastas fast" as h when h is decreased.
of a given smoothed on the real line
near x0, but f’ (x0) ,yl
'S’ we seek ways to. considerivariouscomputationalerrorsdependin'gon‘adisCretizaL
as'
ow. ey decreaseash1decreases._In other:instances;such as
H
particularalgorithm”we'
are interested_ inT’alboundon
ncreaseS‘nnboundedlyxeigqan;=3h). i ’ziw‘lixI
we'denote‘
'
'
"
amental theorem is'
:in a moment, write'
' e.=(9(hq)
w=f0(ii16gn)~= r
H _.
.
Ottawa'
‘
LSIOPCof the chordm C"? log”
Irdisinred.
.
ext whichof thesiertwo:‘I‘neaningsisI,
i.
g , stronger}relation than“the (9 notation::;aafunction ¢(h)fo 1” is @(1/I(h))\(resp.,@(1/r(n)))if; q);is asymptotically
For our ‘particularinstance, f (x) = sin(x), we have the exact value f’ (x0): cos(1.2)
362357754476674. . .. Carrying out our short algorithm we obtainfor h = 0.1 the approxi—
on f’ (x‘o)% (sin(l .3) - sin(1.2))/0.1 = 0.315.. .. The absolute
error thus equals approximately
47. The relative error is not qualitatively different here. .
This approximation of f’(x0) using h = 0.1 is not very accurate. We
therefore apply the same
llgorithmusing several increasingly smaller valuesof h. The resulting errors are as follows:
h Absolute error
0.1 4.7166766—2
0.01 4.666196e—3
0.001 4.660799e-4
1.e—4 4.660256e—5
I,1.e-7 4.619326e—8
f’ (x0) is approxi—
iiscretization error
Indeed, the error appears to decrease like h.More specifically(and less importantly), using
‘
our explicit knowledge of f”
(x) = f (x) =— sin(x), in this case we have that %f
”(m) m —0.466.
Therquantity 0:466h is seen to provide a ratheraccurate estimate for the above-tabulated absolute
3 (90!) SO long-'33I
error values.
-
8.
'
Chapter 1. Numerical AlgorithmsmThe damagingeffect of roundoff errorsThe calculations in Example 1.2, and the ones reportedbelow, were carried out using MATLAB’sstandard arithmetic. Let us stay for'a few more moments with the approximation algorithm featuredin that example and try to push the envelope a little further. ‘ '
Absoluteerror
sin(l.2 + h) —' sin(l .2)h
< 10—10.cos( 1.2) —-
Can’t we just set It 5 10—10/0.466in our algorithm?Not quite! Let us record results for very small, positive values of h:
12 Absolute error
1.e—8 4.361050e—10’
1.e-9 5 .594726e-8
1.e—10 1.669696e—7 P’erpolatesthe‘1.e-11 7.938531e—61._e—l3 4.250484e—4
1.e-15 8.173146e-2
1.e-16 3.623578e—1_ _ _‘ rrorwhenh istA log-log plot1 of the error versus 11 is provided In Flgure 1.3. We can clearly see that as h 18decreased, at first (from right to left in the figure) the error decreases along a straight line?but thistrend is altered and eventually reversed. The MATLAB script that generates the plot ingFlgure1.3
A is given next. ‘
x0 = 1_2-I
r-
. creasesashdelf0 = sin(x0); - ''-
. errorwhensolvjfp = cos (x0); 'i = -20:0'.5:0;
-
i
L
h = 10.*i,~ ‘_
V
pular theornerr = abs (fp — (sin(x0+h) — f0) ./h );d_err = f0/2*h;loglog (h, err, ' —1=’ ) ;
‘
513‘ Other Pollhold on-
age. They areloglog (h,d_err, ’ r— . ’ );xlabel(’h')
V
ylabel (’Absolute error’)
Perhapsthe most mysterious line in this script is that defining d4err: it calculatesélf'txonh- I.
7
:3 Algal.I this section vs
‘lGraphingerror values using a logarithmic scale is rather common in scientific computing,: because a logarithmic umerical algorscale makes it easier to trace values that are close to zero. As you will use such plotting often, let us mention at this ould have.early stage the MATLAB commands plot, semilogy, and loglog.
-
merical Algorithm“
1
tusinglVIATLAlB’In algorithmifeature
'
‘Absoluteerror
—15
10
10-20 .5 0
10 10
1I
T
Figure1.3. The combined efi‘ectof discretization and roundofi‘errors. The solid curveerpolates the computed values of lf
’(x0) — W |for f (x) = sin(x), x0 = 1.2. Also shown
Hashédot style is a straight-line depicting the discretization error without roundofi error
The reason the error “bottoms out” at about h = 10—8in the combined Examples 1.2—1.3that the total, measured error consists of contributions of both discretization and roundoff errors.e discretization error decreases in an orderlyfashion as h decreases, and it dominates the roundoffor iii/henlhis relatively large. But When h gets below approximately 10—8the discretization error
e‘
meslvery small and roundoff error starts to dominate (i.e., it becomes larger in magnitude).;, Roundoff error has a somewhat erratic behavior, as is evident from the small oscillations that
present in the graph in a few places.Moreover, for the algorithm featured in the last two examples, overall the roundoff error in—
,reases as h decreases. This is one reason why we want it always dominated by the discretizationrror when solving problems involving numerical differentiation such as differential equations.
rlyseethatas his‘raightline, but this1e plot in Figure 1.3l
m:
o ular theorems from calculusL
7
e Taylor Series Theorem, given on page 5, is by far the most cited theorem in numerical anal—ysis. Other popular calculus theorems that we will use in this text are gathered on the followingpage. They are all elementary and not difficult to prove: indeed, most are special cases of Taylor’sTheorem.
Specific exercises for this section: Exercises 1—3. Iv
rr: it calculates -. ':'
€11.63.- Algorithm propertiesIn this section we briefly discuss performance features that may or may not be expected from a goodnumerical algorithm, and we define some basic properties, or characteristics, that such an algorithm
-
'
should have.
because a logarithmiclet us mention at this
-
10___—_______.—_—_-——-——-——-—————
Chapterl. Numerical Algorithms . - Algorithm
’
t
Theorem: Useful Calculus Results.
0 Intermediate Valueevaluauon
If f e C[a,b] and s is a valuevsuch that f(&) 5 s 5 f0?) for two numbers6,5 Ex 95 ASSw[(1,1)], then there exists a real number c 6 [a,b] for which f (c) = s.
v
g‘
% in 2
'
I
% (X,
0 Mean Value . p E cIf f E C [a,b] and f. is differentiable on the open interval (a,b), then thjereiexistsa for 3real number c 6 (a,b) for which f
’(c) = (1713;!(a).
' i “i a=
,
en
0 Rolle’s- -
_
If f ‘E C [a,b] and f is differentiable on (0,1)), and in addition f (a) f'(b)= 0,
then there is a real number c 6 (a,b)‘ for which f’ (c) = O.
The “onic
introducir
It is imp01
. . . .
a ', algorithmCriteria for assessmg an algorithm , ' they do n
An assessment of the quality and usefulness of an algorithm may be based on a number of criteria:-~ ablY-Fun
' I' well. Cur
0 Accuracy integral p
This issue is intertwined with the issue of error types and was discussed at the start of Sec—-
ago. Ind‘
tion 1.2 and in Example 1.2. (See also Exercise 3.) The important point is that the accuracy of'
other par:used for i
2See page 7 for the (9 notation.
a numerical algorithm is a crucial parameter in its assessment, and when designing numerical
algorithms it is necessary to be able to point out what magnitude of error is to be expectedThose, 111
when the computation is carried out.this text"
.
portanceEfficrencyA good computation is one that terminates before we lose our patience. A numerical algo—
° RObusm
rithm that features great theoretical properties is useless if carrying it out takes an unreason-Often, th
able amount of computational time. Efficiency depends on both CPUltime and storage spaceLAB for
requirements. Details of an algorithm implementation within a given computer languageand tegration
an underlying hardware configurationmay-play an important role in yieldingcode efficiency.would w
Other theoretical properties yield indicators of efficiency, for instance, the rate of conver—result to
gence. We return to this in later chapters.with aw
Often a machine-independent estimate of the number of elementary operationsrequired, namelyThere an
additions, subtractions, multiplications, and divisions, gives an idea of the algorithm’seffi—'
algorithr'
must be.ciency. Normally, a floating point representation is used for,real numbers and then the costs
of these different elementary floating point operations, called flops, may be assumed to be
roughly equal to one another. roblem cont
view of the 1
Example 1.4. A polynomial of degree n, given as arises regardin
17,106)= co+61x + - - - +Cnx”,
requires (9(n2)operations2to evaluate at a fixed point x, if done in a brute force way withouintermediate storing of p0wers of x. But using the nested form, also known as Homer’s rul
and given by191105)= ' ‘ (@1115+ Cn—l)x+ Cn—Z)x
' )X + Cf),
-
suggests an evaluation algorithm which requires only (9(n)elementary operations,i.e., re—
quiring linear (in n) rather than quadratic computationtime. A MATLAB script for nested
evaluation follows:'
I, “Hf-U]. IV: -A 112.
numbers“’17E % Assume the polynomialcoefficients are already stored
% in array 0 such that for anyreal x,
I
% p(x) = ,c(l) + c(2)x + C(3)XA2+ + c(n+l)x"n
.t . p=cmnh
fi‘thereexistsa for j= n:-1:l
p=p*x+c(j);'J end
The “onion shell” evaluation formula thus unravels quite simply.Note also the manner of
introducing comments into the script. Il)=f(b)=0,
It is important to note that while operationcounts as in Example 1.4 often give a rough idea
of
* algorithm efficiency, they) do not give the complete pictureregarding execution speed, since
.they do not take into account the price (speed)of memory access which may vary
consider—
, ably. Furthermore, any, setting of parallel computingis ignored in a simple operation count
as
well. Curiously, this is part of the reason theMATLAB command f lops, which had been
an
integral part of this language for many years,was removed from further releases several years
ago. Indeed, in modern computers,cache access, blocking and vectorization features,
and
other parametersare cruCial in the determinationof execution time. The computer language
% used for implementationcan also affect the comparative timingof algorithm implementations.
Thofse, unfortunately, are'much moredifficult to assess compared to an operation count.
In
this'text‘we 'will not get into the gory details of these issues, despitetheir relevance and im—
'
portanCe:
number of criteria:
‘ at thestart of. Sec—that the accuracy of
'
esigning numericalit is to be expected
Robustness
Often, the major effort in writing numerical software,such as the routines available in MAT-
LAB for solving linear systems of algebraic equationsor for function approximation and in-
tegration, isspent not on implementingthe essence of an algorithm but on ensuring that
it
would work under all weather conditions Thus,the routine should either yield the correct
result to within an acceptable error tolerance level,or it should fail gracefully (i.e., terminate
with aiwarning) if it does not succeed to guaranteea “correct result.”
A numerical algo-'
‘
takes an unreason-
‘: and storage spaceIPuter language and -
jng code efficiency. -16 rate of conver-
PnsreQUifed,namely Thereare intrinsic numerical properties that account for the
robustness and reliability of an
516algonthm’seffi-‘ '
algOrithm. Chief among these is therate of accumulation of errors. In particular, the algorithm
and then the COStS mubt be stable; see Example 1.6.
f be assumed to beiProblem'cov'nditioningand algorithm stability
I
In View of the fact that the problem and the numerical algorithmboth yield errors, a natural question
l’arises regardingthe appraisal of a given computedsolution. Here notions such as problem sensitivity
and algorithm stability play an importantrole. If the problem is too sensitive, or
ill-conditioned,
vtmeaningithat even a small perturbationin the data produces a large difference in
the result,3 then no
salgorithrrinlaybe found for that problem whichwould meet our requirement of solution robustness;
'
see Figure 11.4for an illustration. Somemodification in the problem definition may be
called for in
» such cases.
«5force way without
In as Homer’s rule
g ,H3HereWe refer to intuitive notions of “large”vs. “small” quantities and of values being
“close to” vs. “far from”
one another. While these notions can be quantifiedand thus bemade more precise, such a move would typically
make
'
definitions cumbersome and harder to understandat this preliminary stage of the discussion.
-
12 Chapter 1. Numerical Algorithms,
y
Figure 1.4. An ill-conditioned problem of computing output values y given in terms ofinput values x by y = g(x): when the input x is slightly perturbed to x, the result y = g(x) is farfrom y. If the problem were well-conditioned, we would be expecting the distance betWeen y and y lto be more comparable in magnitude to the distance between x and x.
I
For instance, the problem of numerical differentiation depicted in Examples 1.2 and 13 turns »out to be ill—conditioned when‘extreme accuracy (translating to very small values of h) is required.
The job of a stable algorithm for a given problem is to yield a numerical solution which is‘
the exact solution of an only slightly perturbed problem; see the illustration in Figure 1.5. Thus,‘
if the algorithm is stable and the problem is well-conditioned (i.e., not ill—conditioned),then the lcomputed result 52is close to the exact y.
‘1
Figure 1.5. An instance of a stable algorithm for computing y = g(x): the output 5: istheexact result, 52= g(x'), for a slightly perturbed input, Le, x which is close to the input x. Thus, if thealgorithm is stable and the problem is well-conditioned, then the computed result y is close'to theexact y.
‘ '
Example 1.5. The problem of evaluating the square root function for an argument near the value 1is well—conditioned,as we show below.
-
lmericalAlgorithms.\- l3
t:g(x) = «N +xand note that g’(x) = LAT;Suppose we fix x so that le
-
'1F
l4
Chapterl. Numerical AlgorithmAn algorithm which may come to mind is therefore as follows:
1. Evaluateyo = ln(11) —ln(10).2. Form = 1,...,30, evaluate
by 10 each time the recursion is applied. Thus, there is exponential error growth with c1 = 10. InMATLAB (which automatically employs the IEEE double precision floatingpoint arithmetic seeSection 2.4) we obtain yo = 9.53106 — 02, ylg = '—9.1694e+01, y19 = 9.1694e+02,...,y3o =—9.1694e + 13. It is not difficult to see that the exact values all satisfy 0 < y” < l, and hence thecomputed solution, at least for n 3 18, is meaningless! I ‘ s
Thankfully, such extreme instances of instability as illustrated in Example1.6 will not occur "in any of the algorithms developed in this text from here on. ‘
Specific exercises for this section: Exercises .4—5.
1.4 Exercises
0. Review questions 'V
'
i
(a) Whatris the difference, according to Section 1.1, between scientific computing and nu—merical analysis?(b) Give a simple example where relative error is a more suitable measure than? absolute 5 t
error, and another example where the absolute error measure is more suitable.(c) State a major difference between the nature of roundoff errors and discretization errors.((1) Explain brieflywhy accumulation of roundoff errors is inevitable when arithmetic opera- Ltions are performed in a floatingpoint system. Under which circumstances is it tolerable 1;.in numerical computations?(e) Explain the differences between accuracy, efficiency,and robustness as criteria for eval—
uating an algorithm.
(i) Show that nested evaluation of a polynomial of degree 11 requires only 211 elementaryoperations and hence has_0(n) complexity. ' ‘
(g) Distinguish between problem conditioning and algorithm stability.1. Carry out calculations similar to those of Example 1.3 for approximating the derivative of thefunction f (x) = e‘2x evaluated at x0 = 0.5. Observe similarities and differences by comparing
'
your graph against that in Figure 1.3. '
2. Carry out derivation and calculations analogous to those in Example 1.2, using the expression
f(xo+h)-f(xo—h)2h
for approximating the first derivative f’ (x0). Show that the error is (9012).More precisely, theleading term of the error is — g f ”’ (x0) when f ’ ” (x0) 75 O.
-
iericaiAlgorithms
off errors were not
ors getsmultiplied1 with c] = 10. Inint arithmetic;see
.,y30 =: Land hence the
1.6 will not occur
mputing and nu-
re than; absolute‘
itable..
etization errors.
rithmetic opera-es is it tolerable
riten'a for eval-
Zn elementary
:rivative of the: by .comparin g
the expression
precisely, the
- Additional notes . . ‘ 15m
33. Carry out similar calculationsto those of Example 1.3 using the approximation from Exer—cise ’2.
,_ Observe similarities and differences by comparing your graph against that in Fig—ure 1.3;
’4.FollOwing Example1.5, assess the conditioning of the problem of evaluating
exp(cx) — exp(—cx)g(x) = tanmcx) =
exp(cx) + exp(—cx)
near x = 0 as the positive parameter c grows.
,5. Consider the problem presented in Example 1.6. There we saw a numerically unstable proce-dure for carrying out the task.
(a) Derive a formula for approximately computing these integrals based on evaluating yn_1given y".
(b) Show that for any given value 8 > 0 and positive integer no, there exists an integer. n1 3 no such that taking y,” = O as a starting value will produce integral evaluations 32,,
with an absolute error smaller than 8 for all 0 < n 5 no.
(0) Explain why your algorithm is stable.
(d) Write a MATLAB function that computes the value of yzo within an absolute error ofat most 10—5.Explain how you choose 11] in this case. -
5 Additionalnotes
lay with experiment and theory. On the one hand, improvements in computing power allow for
ornputersare still not (and may never be) powerful enough to handle.A potentiallysurprising amount of attention hasbeen given throughout the years to the defini-
ons of scientificcomputing and numerical analysis. An interesting account of the evolution of thisseemingly esoteric but nevertheless important issue can be found in Trefethen and Bau [70].‘
The conceptof problem conditioning is both fundamental and tricky to discuss so early ine game. If you feel a bit lost somewhere around Figures 1.4 and 1.5, then rest assured that these’conceptsieventuallywill become clearer as we gain experience, particularly in the more specificcontexts of Sections 5.8, 8.2, and 14.4. A
‘'
Many‘computerscience theory books deal extensively with 0 and 8 notations and complexityissues. One widely used such book is Graham, Knuth, and Patashnik [31].
and what’s in Mathworks. One helpful survey of some of those can be found at http://www.cs.ubc.ca/Nmitchell/matlabResources.html .
s
There are many printed books and Internet introductions to MATLAB. Check out wikipedia I