1. foundations of numerics from advanced mathematics · mathematical essentials and notation linear...

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

1. Foundations of Numerics from Advanced Mathematics


Numerical Programming I (for CSE), Hans-Joachim Bungartz page 1 of 50

http://www5.in.tum.de/persons/bungartz.html


The main purpose of this first chapter (about 4 weeks) is to recall those topics fromyour Advanced Mathematics courses (linear algebra, calculus, stochastics) typical forthe first two years of bachelor’s programs in science and engineering that are ofparticular importance for numerical algorithms and, hence, for the whole CSE master’sprogram.

We do this,• since you can hardly go successfully through a thorough numerical education

without these foundations;

• since we made the experience that the CSE freshmen’s mathematicalbackgrounds are quite heterogeneous (and not always at hand ...);

• since TUM’s CSE program has a methodological (i. e. mathematical andinformatical) point of view that goes beyond the usual and widespread engineeringapproach and way of thinking;

• and since the two numerics courses have been the most serious roadblock forCSE students since the program’s launch (too high failure rates – something wewant to reduce without touching the level of education).

If you are familiar with all this stuff, don’t feel bored – just consider this as a warm-up tothe numerical contents to be discussed later on in this course.





We also changed the name of the courses from “Numerical Analysis” to “NumericalProgramming”, to indicate that there are mathematical topics to be addressed, but witha clear focus on algorithmics, programming, and applications (instead of proofs etc.).

This introductory part won’t be a complete lecture with all explanations etc. Rather, itwill be a “guided tour” through important topics, mentioning notions and buzzwords thatshould have some meaning for you. If they don’t, you know that you have to close thegaps as soon as possible, with the help of the references provided or by doingadditional exercises etc.

Also use the tutorials to refresh your knowledge!





1.1. Mathematical Essentials and Notation

Symbols and Notions

Everyone familiar with• the symbol ∞;

• the symbols ∃, ∃1, and ∀ (so-called quantifiers);

• the symbolsnX

i=1

andYi6=k

;

• the notions maximum, minimum, infimum, and supremum;

• Kronecker symbol δij ;

• the Landau symbol O(N), O(h2)

• the symbol ⇒;

• the meaning of sufficient and necessary;

• the meaning of iff: sufficient and necessary;

• the meaning of associative, commutative, and distributive?





Numbers

• Booleans: true/false; logical operations; relations of logics to set theory (seebelow)

• natural numbers, integers N, Z: factorials; binomial coefficients; Pascal’s triangle

• rational numbers Q: countable/non-countable

• real numbers R:

– field property (allows for arithmetic operations)– order property (allows for comparison)– completeness property (each interval nesting defines exactly one real

number)– supremum/infimum property

• Q is dense in R• different classes of irrational numbers:

√2, e, ...

• complex numbers C: imaginary unit i, Re, and Im; conjugate complex

• fundamental theorem of algebra: each polynomial of degree n with complexcoefficients has at least one complex root

• what else can be said of roots of polynomials?





Sets

• notions of sets, subsets, and elements

• set operations: union, intersection, difference, complement

• symbols ∈, ⊂, ⊆

• power set

• Cartesian product of sets

• appearances:

– explicit 1, 2, 3, ...– implicit x ∈ R : f(x) = 0

• already here a bit of topology: open sets, closed sets, bounded sets, compactsets





Relations

• definition: relation R between two sets A and B as a subset of A×B:R ⊆ A×B

• notation: aRb or (a, b) ∈ R

• important examples for A = B: <, ≤, >, ≥, ...

• properties of relations:

– transitive– reflexive– symmetric– asymmetric– antisymmetric– connex

• notion of an equivalence relation





Mappings and Functions

• mapping or function (here used in a synonymous way)

f : A → B : ∀x ∈ A ∃1y ∈ B such that y = f(x);

write x 7→ y

• properties of mappings:

– injective– surjective– bijective

• f−1(x) = ?

• inverse mapping: existence and meaning





Building Blocks ...

... of a math course / book / presentation:

• definition: new notions etc. are defined and, thus, introduced

• theorem / proposition:

– a central statement, typically consisting of conditions and a conclusion (“if thisand that holds, then the following is valid ...”)

– the more restrictions are made, the more can be concluded (but also the lessgeneral the statements are)

• lemma: similar to a theorem w.r.t. its structure, but usually only an auxiliarystatement of minor importance by itself (that marks just a step on the way to atheorem, e.g.)

• corollary: a statement that follows immediately from a previous theorem etc.

• proof: a precise argumentation showing clearly that a theorem, lemma, orcorollary is correct

Note that all this is typically formulated as general and generic as possible – a factwhich is frequently misinterpreted as “not concrete” or “without practical relevance”.





A Short Remark on Proofs

Why proofs – or how much of proofs?

• Proofs are the essence of mathematical argumentation – they make the latterrigorous.

• Proofs are a permanent source of misunderstandings and problems:

– math professors often do not want to do anything without proofs – even incourses for non-mathematicians

– non-math students often think that only the results or statements are relevant,but not the proofs (which they suppose to be something for hardcoremathematicians only)

– note that both points of view are problematic

• hence: proofs for non-mathematicians (such as CSE students)?

– yes, if the way of proving a statement helps to understand it– no, if just for itself (i.e. just to prove it)





Standard Proof Techniques

• forward: A ⇒ B ⇒ C ⇒ D

• by contradiction (“what if”): ¬D ⇒ ... ⇒ ¬A

• by counterexample: to refute the assertion that all students are smart, just findone stupid and the job is done

• complete search: to prove that all students are smart, check them all

• mathematical / complete induction: show the statement for n = 1, and showthe conclusion from n to n + 1 (does it work for the smart student example?)





1.2. Linear Algebra

Mathematical Structures

• a mathematical structure consists of one or several sets and one or severaloperations defined on the set(s)

• special elements:

– neutral element (of an operation)– inverse element (of some element x)

• a group: a structure to add and subtract

• a field: a structure to add, subtract, multiply, and divide

• a vector space: a set with additional properties, allows for addition andmultiplication with scalars

• note: sometimes, the association with classical (geometric) vectors is helpful,sometimes it is more harmful





Vector Spaces

• a linear combination of vectors

• linear (in)dependence of a set of vectors

• the span of a set of vectors

• a basis of a vector space

– definition?– why do we need a basis?– is a vector’s basis representation unique?– is there only one basis for a vector space?

• the dimension of a vector space

• does infinite dimensionality exist?

• important applications:

– (analytic) geometry– numerical and functional analysis: function spaces are vector spaces

(frequently named after mathematicians: Banach spaces, Hilbert spaces,Sobolev spaces, ...)





Linear Mappings

• definition in the vector space context; notion of a homomorphism

• image and kernel of a homomorphism

• matrices, transposed and Hermitian of a matrix

• relations of matrices and homomorphisms

• meaning of injective, surjective, and bijective for a matrix; rank of a matrix

• meaning of the matrix columns for the underlying mapping

• matrices and systems of linear equations

• basis transformation and coordinate transformation

• mono-, epi, iso-, endo-, and automorphisms





Determinants

• definition

• properties

• meaning

• occurrences

• Cramer’s rule





Eigenvalues

• notions of eigenvalue, eigenvector, and spectrum

• similar matrices A, B:∃S : B = SAS−1

(i.e.: A and B as two basis representations of the same endomorphism)

• resulting objective: look for the best / cheapest representation (diagonal form)

• important: matrix A is diagonalizable iff there is a basis consisting ofeigenvectors only

• characteristic polynomial, its roots are the eigenvalues

• Jordan normal form

• important:

– spectrum characterizes a matrix– many situations / applications where eigenvalues are crucial





Scalar Products and Vector Norms

• notions of a linear form and a bilinear form

• scalar product: a positive-definite symmetric bilinear form

• examples of vector spaces and scalar products

• vector norms:

– definition: positivity, homogeneity, triangle inequality– meaning of triangle inequality– examples: Euclidean, maximum, and sum norm

• normed vector spaces

• Cauchy-Schwarz inequality

• notions of orthogonality and orthonormality

• turning a basis into an orthonormal one: Gram-Schmidt orthogonalization





Matrix Norms

• definition:

– properties corresponding to those of vector norms– plus sub-multiplicativity:

‖AB‖ ≤ ‖A‖ · ‖B‖

– plus consistency‖Ax‖ ≤ ‖A‖ · ‖x‖

• matrix norms can be induced from corresponding vector norms: Euclidean,maximum, sum

‖A‖ := max‖x‖=1

‖Ax‖

• alternative: completely new definition, for example Frobenius norm (considermatrix as a vector, then take Euclidean norm)





Classes of Matrices

• symmetric: A = AT

• skew-symmetric: A = −AT

• Hermitian: A = AH = AT

• s.p.d. (symmetric positive definite): xT Ax > 0 ∀x 6= 0

• orthogonal: A−1 = AT (the whole spectrum has modulus 1)

• unitary: A−1 = AH (the whole spectrum has modulus 1)

• normal: AAT = AT A or AAH = AHA, resp. (for those and only those matricesthere exists an orthonormal basis of eigenvectors)





1.3. Calculus

Functions Revisited

• notions of a function, its range, and its image

• graph of a function

• isolines and isosurfaces

• sums and products of functions

• composition of functions

• inverse of a function: when existing?

• simple properties: (strictly) monotonous

• explicit and implicit definition

• parametrized representations (curves, ...)





Continuity

• remember the ε and the δ!

• definition of local (“in x0”) and global continuity (“∀x”)

• what about sums, products, quotients, ... of continuous functions?

• what about compositions of continuous functions?

• what about continuity of the inverse?

• intermediate value theorem

• continuous functions on compact sets – maximum and minimum value

• uniform continuity





Limits

• meaning of ε → 0 and N →∞ and x →∞

• accumulation point of a set

• limit (value) of a set

• limits from the left or from the right, respectively: f(x+), f(x−)

• limits at infinity: limx→∞ f(x)

• infinite limits: f(x) →∞

• how can discontinuities look like?

– jumps: f(x+) 6= f(x−)

– holes: f(x+) = f(x−) 6= f(x)

– second kind: f(x) = 0 in x = 0 and f(x) = sin`

1x

´elsewhere





Sequences

• definition of a sequence: a function f defined on N• if f(n) = an, write

(an) or a1, a2, a3, ...

• bounded / monotonously increasing / monotonously decreasing sequences

• notion of convergence of a sequence: existence of a limit for n →∞

• Cauchy sequence

• subsequences





Series

• notion of an (infinite) series

– elements of a series– partial sums of a series– convergence defined by convergence of the sequence of the partial sums– convergence and absolute convergence

• examples:

– geometric series:P∞

k=1 xk = 11−x

– harmonic series:P∞

k=11k

= ∞– alternating harmonic series:

P∞k=1(−1)k−1 1

k= ln(2)

• criteria for convergence: quotient and root criterion

• power series:P∞

k=0 ak(z − a)k

– coefficients ak and centre point a

– radius of convergence R: absolute convergence for |z − a| < R

– identity theorem for power series

• re-arrangement

• sums of series, nested series, products of series (Cauchy product)





Differentiation

• first step: functions f of one real variable, complex values allowed

• derivative or differential quotient of f :

– defined via limit process of difference quotients– write f ′ or f or df

dx

– geometric meaning?– local and global differentiability– derivative from the left / from the right

• rules for the daily work:

– derivative of f + g, fg, and f/g?– derivative of f(g) (chain rule)?– derivative of the inverse function?

• higher derivatives f (k)(x); meaning

• notion of continuous differentiability

• smoothness of a function

• space of k-times continuously differentiable functions: Ck





Differential Calculus of one Real Variable

• notion of a global/local minimum/maximum

• local extrema and the first derivative

• mean value theorem:

∃ξ ∈ (a, b) : f ′(ξ) =f(b)− f(a)

b− a

• monotonous behaviour and the first derivative

• local extrema and the second derivative

• rule of de l’Hospital

• notions of convexity and concavity

• convexity/concavity and the second derivative

• notion of a turning point

• turning points and the second derivative





Function Classes (1)

• polynomials– definition, degree, sums and products, division with rest, identity theorem,

roots and their multiplicity• rational functions

– poles and their multiplicity, partial fraction decomposition• exponential function and logarithm

– characterising law of the exponential function:

exp(s + t) = exp(s) · exp(t) or y′ = y

(functional equation of natural growth)– series expansion of the exponential function, speed of growth– natural logarithm as exp’s inverse:

y = exp(x) = ex, x = ln(y)

– functional equation: ln(xy) = ln(x) + ln(y)

– exponential function and logarithm for general basis a:

ax := ex ln a, loga(y) :=ln y

ln a





Function Classes (2)

• hyperbolic functions

– cosh(z), sinh(z), ...

• trigonometric functions

– sin(x), cos(x): solutions of y(2) + y = 0

– geometric meaning?– Euler’s formula: eix = cos(x) + i · sin(x)

– derivatives, addition theorem– periodicity– series expansion





Integral Calculus of one Variable

• Riemann integral, upper and lower sums• approximation by staircase functions• properties:

– linearity– monotonicity

• mean value theorem:

ξ ∈ (a, b) :

Z b

af(x)dx = (b− a) · f(ξ)

• main theorem of differential and integral calculus:– define F (x) :=

R xa f(t)dt

– thenR b

a f(t)dt = F (b)− F (a)

• rules for everyday work:– partial integration: Z

uv′dx = uv −Z

vu′dx

– substitution: Z b

af(t(x))t′(x)dx =

Z t(b)

t(a)f(t)dt





Local Approximation: Taylor Polynomials and Series

• local approximation of functions with polynomials• generalization of the tangent approximation used for the definition of the derivative• Taylor polynomials:

– let f be n-times differentiable in a

– we look for a polynomial T with T (k) = f (k) for k = 0, 1, ..., n

– obviously:

T (x) :=nX

k=0

1

k!f (k)(a)(x− a)k

– unique, degree n, write Tnf(x; a)

– remainder Rn+1(x) := f(x)− Tnf(x : a)

Rn+1(x) =f (n+1)(ξ)

(n + 1)!(x− a)n+1

• Taylor series:– for infinitely differentiable functions (exp, sin, cos, ...)– sum up to ∞ instead of n only– examples





Global Approximation: Uniform Convergence

• convergence of sequences of functions fn defined on D:

• pointwise: for each x ∈ D; then

f(x) := limn→∞

fn(x)

defines a function

– problems: are properties such as continuity or differentiability inherited fromthe fn to f , and how to calculate derivatives or integrals of f?

– i.e., can the order of limit processes be changed?

• therefore the notion of uniform convergence:

– definition: ‖fn − f‖D → 0 for n →∞– with that, the inheritance and change-order problems from above are solved!– criteria: Cauchy, ...

• approximation theorem of Weierstrass: each continuous function f on acompact set can be arbitrarily well approximated with some polynomial





Simple Differential Equations

• notion of a differential equation

– ordinary: one variable– partial: more than one variable (several spatial dimensions or space and

time)

• examples:

– growth: y = k · y or y = k(t, y) · y– oscillation: y + y = 0 or similar

• example of an analytic solution strategy: separation of variables

y′ = g(x) · h(y), y(x) = y0

– formal separation:dy

h(y)= g(x)dx

– integration of the left and right side– some requirements for applicability





Periodic Functions

• target now: periodic functions, period typically 2π

• trigonometric polynomials

• definition:

T (x) :=nX

k=−n

ckeikx =a0

2+

nXk=1

(ak cos(kx) + bk sin(kx))

(coefficients ck, ak, and bk are unique)

• formula for the coefficients:

ck =1

2π

Z 2π

0T (x)e−ikxdx

• T is real iff all ak, bk are real iff ck = c−k

• Weierstrass: 2π-periodic continuous functions can be arbitrarily wellapproximated by trigonometric polynomials





Fourier Series

• consider vector space of 2π-periodic complex functions f on R

• Fourier coefficients: f(k) :=1

2π

Z 2π

0f(x)e−ikxdx

• Fourier polynomial: Snf(x) :=nX

k=−n

f(k)eikx

• Fourier series:∞X−∞

f(k)eikx

• sine-cosine representation of Snf :

Snf(x) =a0

2+

nXk=1

(ak cos(kx) + bk sin(kx))

• coefficients:ak = f(k) + f(−k) =

1

π

Z π

−πf(x) cos(kx)dx

bk = i(f(k)− f(−k)) =1

π

Z π

−πf(x) sin(kx)dx

• all ak vanish for odd f , all bk vanish for even f





Functions of Several Variables

• f now defined on Rn or a subset of it

• notion of differentiability: now via existence of a linear map, the differential

• directional derivatives

• partial derivatives

• prominent differentiability criterion: existence and continuity of all partialderivatives

• the gradient of a scalar function f and its interpretation

• the Jacobian of a vector-valued function f

• mean value theorem

• higher partial derivatives, Taylor approximation, Hessian

• local minima and maxima, criteria

• saddle points





Integration over Domains

• a huge field, from which we only mention a few results

• theorem of Fubini:– shows that, in many cases, a multi-dimensional integration domain can be

tackled dimension by dimension– statement (we neglect the requirements, for which more integration theory is

needed):ZX×Y

f(x, y)d(x, y) =

ZY

„ZX

f(x, y)dx

«dy =

ZX

„ZY

f(x, y)dy

«dx

– related to Cavalieri’s principle– will also be of relevance for numerical quadrature

• transformation theorem:– a generalisation of integration by substitution– statement, again without requirements:Z

Uf(T (x)) ·

‚‚det T ′(x)‚‚ dx =

ZV

f(y)dy

– allows for a change of the coordinate system (polar coordinates), e.g.





Gauss Theorem

• we further generalise integration, now allowing for integration over hyper-surfaces(a sphere, e.g.)

• this is important for the physical modelling in many scenarios (heat flux through apot’s surface, ...)

• the famous Gauss theorem allows to combine integrals over volumes andsurfaces, which occurs in the derivation of many physical models (conservationlaws) and, hence, is of special relevance for CSE

• prerequisites:– a vector field: a vector-valued function on Rn (example: the velocity field in

fluid mechanics)– the divergence of a vector field F :

div F (x) =nX

i=1

∂iFi(x)

• finally the Gauss theorem:– several regularity assumptions neededZ

Gdiv Fdx =

Z∂G

F ~dS





1.4. Stochastics and StatisticsNot a primary ingredient of CSE, but you should have at hand at least some basicknowledge (for things to come later: Monte-Carlo-based simulations, stochasticPDE, ...):

• probability theory: subfield of mathematics providing the apparatus to formaliserandom events and studying what can be derived within that formalism; typicalquestion: if the world looks this and that, what can we say concerning the resultsof a random experiment?

• mathematical statistics: not that firmly connected to mathematics; descriptivestatistics deals with the presentation of huge data sets, inductive statistics dealswith conclusions from measured or observed values of random variables to theirunderlying properties

• stochastics: sometimes used in a synonymous way to probability theory,sometimes as a notion covering probability theory and statistics





Combinatorics

Combinatorics deals with counting possible cases of a certain characteristics in adiscrete and finite universe. The most famous and important example: Select k ≤ nelements from a set of n elements.

Remember the four cases:

• with repetition, ordered:nk

• without repetition, ordered:

nk =

k−1Yi=0

(n− i) =n!

(n− k)!

• without repetition, unordered:“n

k

”=

n!

(n− k)!k!

• with repetition, unordered: “n + k − 1

k

”1. Foundations of Numerics from Advanced Mathematics




Discrete Probability Spaces: Events (1)

• random experiments have results ωi from a discrete (possibly infinite) result setΩ = ω1, ...

• events as subsets of this result set: A, B ⊆ Ω

• A ∩B, A ∪B, A, A \B

• elementary events ωi with probabilities pi

• rules: ∀i : pi ∈ [0, 1],P

Ω pi = 1

• handling probabilities:

p(A) = 1−p(A), p(A∪B) = p(A)+p(B)−p(A∩B), A ⊆ B ⇒ p(A) ≤ p(B), ...

• conditional probabilities (check that p(.|B) are probabilities):

p(A|B) :=p(A ∩B)

p(B)





Independence of Events

• intuitive definition: two events A, B are independent, if there is no influenceamong each other:

p(A|B) = p(A), p(B|A) = p(B)

• formal definition: independence, if

p(A ∩B) = p(A) · p(B)

• examples?

• independence of a set of events A1, ..., An:

p

0@\i∈I

Ai

1A =Yi∈I

p(Ai) ∀I ⊆ 1, ..., n





Discrete Random Variables

• definition: X : Ω → R• X defines events:

“X = x” := ωi ∈ Ω : X(ωi) = x, ...

• discrete density (function):

fX(x) := p(X = x)

• discrete distribution (function):

FX(x) := p(X ≤ x)

• expectation or mean value:

E(X) :=X

x

x · p(X = x)

• variance:V (X) := E

`(X − E(X))2

´• rules how to work with E(X) and V (X):

E(aX + b) = aE(X) + b, V (aX + b) = a2V (X), ...





Important Discrete Distributions

• Bernoulli distribution:

– binary random variable– results 1 and 0 with probabilities p and 1− p, resp.

• Binomial distribution

– sum of n Bernoulli variables– p(X = k) =

`nk

´pk(1− p)n−k

• geometric distribution

– repeat a Bernoulli experiment until success (result 1)– p(X = i) = p(1− p)i−1

• Poisson distribution

– counts events– possible results 0, 1, 2, 3, ...

– density fX(i) = e−λλi

i!

• exercise: think about examples, calculate E(X) and V (X) for all





Continuous Probability Spaces

• motivation:

– think about the differences of discrete and continuous realities and probabilityspaces

• sums get integrals now:

– expectation value:

E(X) :=

Z ∞

−∞tfX(t)dt

– density:

p(A) :=

ZA

fX(t)dt,

Z ∞

−∞fX(t)dt = 1

– distribution:FX(x) :=

Z x

−∞f(t)dt

– properties of fX and FX?





Important Continuous Distributions

• uniform distribution

– no differences in probability– density:

fX(x) =

1b−a

for x ∈ [a, b] ,

0 otherwise.

• normal or Gaussian distribution

– frequent in reality, especially in limit cases, see slide on asymptotics– density:

fX(x) =1

√2πσ

· e−(x−µ)2

2σ2 , FX(x) =: Φ(x) ∼ N (µ, σ)

• negative exponential distribution

– describes time between events– density:

fX(x) =

λ · e−λx if x ≥ 0 ,0 otherwise

• exercise: think about examples and calculate E(X) and V (X) for all





Asymptotics (1)

• inequality of Markov:

p(X ≥ t) ≤E(X)

t.

• inequality of Chebyshev:

p(| X − E(X) |≥ t) ≤V (X)

t2.

• theorem of large numbers:

– given random variables Xi, i = 1, 2, ..., iid (independent and identicallydistributed) with mean value µ and variance σ2

– define arithmetic average Zn of X1, ..., Xn:

Zn :=1

n·

nXi=1

Xi .

– given ε, δ > 0 and n ∈ N with n ≥ σ2

εδ2

– then it holdsp(| Zn − µ |≥ δ) ≤ ε





Asymptotics (2)

• central limit theorem:

– given random variables Xi, i = 1, 2, ..., iid with mean value µ and varianceσ2 > 0

– define Yn, n = 1, 2, ... as n-th partial sum of Xi:

Yn :=nX

i=1

Xi

– let Zn be the normalized random variable of Yn defined as

Zn :=Yn − nµ

σ√

n

– Then: All Zn are asymptotically standardized normally distributed, i. e.

limn→∞

Zn ∼ N (0, 1)

Particularly, it holds for the series of distribution functions FZn of Zn

limn→∞

FZn (x) = Φ(x) ∀x ∈ R





Inductive Statistics (1)

• (random) sample:

– repeat a random experiment– record results as an iid sequence (ωi) or (xi), resp.

• objective: learn about reality with the help of the sample

• first technique: estimators: Y := g(X1, ..., Xn)

– properties: unbiased, variance-reducing, efficient, consistent– examples:

Y := X :=1

n

nXi=1

Xi for E(X)

Y := S2 :=1

n− 1

nXi=1

(Xi − X)2 for V (X)

• maximum likelihood principle and estimators





Inductive Statistics (2)

• second technique: confidence intervals

– construct two estimators U1, U2 to define an interval for an unknownparameter θ (E(X), V (X), ...):

p(U1 ≤ θ ≤ U2) ≥ 1− α

• third technique: tests

– hypothesis H0 and alternative H1 on the size of an unknown parameter

E(X) = 27.2, p < 0.9, ...

– make a sample– determine a range where to reject the hypothesis– error types: error of first kind, error of second kind– maximum risk of first kind: significance level of the test– exercise: study simple examples




1. foundations of numerics from advanced mathematics · mathematical essentials and notation linear...

Documents