functions of matrices - department of mathematicshigham/misc/ggsss13/fun.pdf · in the interview....

Research Matters

February 25, 2009

Nick HighamDirector of Research

School of Mathematics

1 / 6

Functions of Matrices

Nick HighamSchool of Mathematics

The University of Manchester

http://www.maths.manchester.ac.uk/~higham/misc/ggsss13.php

@nhigham, nickhigham.wordpress.com

Gene Golub SIAM Summer School 2013Fudan University, Shanghai

http://www.ma.man.ac.uk/~higham/

http://www.ma.man.ac.uk

http://www.man.ac.uk



https://twitter.com/nhigham

http://nickhigham.wordpress.com

An Interview with Gene Golub1

by Nicholas J. Higham2

On July 3, 2005 I interviewed Gene Golub (1932–2007) during a visit he made toThe University of Manchester to attend a workshop. This document provides an editedtranscript of the interview.

The bibliography contains some books and papers mentioned directly or indirectlyin the interview. For more on Golub and his work, I highly recommend Milestones inMatrix Computation: The Selected Works of Gene H. Golub, with Commentaries [3].

I am grateful to Louise Stait for transcribing the interview, and to Gail Corbett andSven Hammarling for help with the editing.

Gene Golub, July 2005. Photograph by N. J. Higham.

Gene Golub by John de Pillis (http://math.ucr.edu/˜jdp), 2007.

1Document dated February 5, 2008.2School of Mathematics, The University of Manchester, Manchester, M13 9PL, UK

([email protected], http://www.ma.man.ac.uk/˜higham).

1

An InterviewwithGene Golub(H, 2008).

http://eprints.ma.man.ac.uk/1024



History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras

Free SIAM Student Membership

All graduate students attending the Summer School areeligible for free SIAM Student Membership, which givesyou

Free membership in two SIAM Activity Groups (SIAG).Subscription to SIAM News.Subscription to SIAM Review (electronic).30% discount on all SIAM books.

I will be happy to nominate you.

University of Manchester Nick Higham Matrix Functions 3 / 162

http://www.siam.org/membership/individual/free.php

http://www.siam.org/students/memberships.php

http://www.siam.org/students/memberships.php


Outline

1 History2 Definitions3 Properties, Formulas4 Applications5 Fréchet Derivative & Condition No.6 Problem Classification7 Methods for f (A)8 Methods for f (A)b9 Software

10 Case11 Quiz12 Extras



Cayley and Sylvester

Term “matrix” coined in 1850by James Joseph Sylvester,FRS (1814–1897).

Matrix algebra developed byArthur Cayley, FRS (1821–1895).Memoir on the Theory of Ma-trices (1858).



Cayley and Sylvester on Matrix Functions

Cayley considered matrix squareroots in his 1858 memoir.

Tony Crilly, Arthur Cayley: Mathemati-cian Laureate of the Victorian Age,2006.

Sylvester (1883) gave first defini-tion of f (A) for general f .

Karen Hunger Parshall, James JosephSylvester. Jewish Mathematician in aVictorian World, 2006.



Buchheim’s Formula

Buchheim (1886) extended Sylvester’s 1883interpolation formula to arbitrary eigenvalues:


http://nickhigham.wordpress.com/2013/01/31/arthur-buchheim

Laguerre (1867):

Peano (1888):


Matrices in Applied Mathematics

Frazer, Duncan & Collar, Aerodynamics Division ofNPL: aircraft flutter, matrix structural analysis.

Elementary Matrices & Some Applications toDynamics and Differential Equations, 1938.Emphasizes importance of eA.

Arthur Roderick Collar, FRS(1908–1986): “First book to treatmatrices as a branch of appliedmathematics”.



What’s New?

Advances in Matrix Functions and Matrix Equations,University of Manchester, April 10–12, 2013.

Workshop report.


http://www.mims.manchester.ac.uk/events/workshops/FUN13

http://nickhigham.wordpress.com/2013/05/22/workshop-on-matrix-functions-and-equations


Outline





Defining by Substitution

Want to define f : Cn×n → Cn×n, but not elementwise.Given f (t), can define f (A) by substituting A for t :

f (t) =1 + t2

1− t⇒ f (A) = (I − A)−1(I + A2).

log(1 + x) = x − x2

2+

x3

3− x4

4+ · · · , |x | < 1

⇒ log(I + A) = A− A2

2+

A3

3− A4

4+ · · · , ρ(A) < 1.

Works for fa polynomial,a rational,or with a convergent power series.



Jordan Canonical Form

Z−1AZ = J = diag(J1, . . . , Jp), Jk︸︷︷︸mk×mk

=

λk 1

λk. . .. . . 1

λk

Definition

f (A) = Zf (J)Z−1 = Zdiag(f (Jk))Z−1,

f (Jk) =

f (λk) f ′(λk) . . .

f (mk−1)(λk)

(mk − 1)!

f (λk). . . .... . . f ′(λk)

f (λk)

.



“Deriving” The Formula for f (Jk)

Write Jk = λk I + Ek ∈ Cmk×mk . For mk = 3 we have

Ek =

0 1 00 0 10 0 0

, E2k =

0 0 10 0 00 0 0

, E3k = 0.

Assume f has Taylor expansion

f (t) = f (λk) + f ′(λk)(t − λk) + · · ·+f (j)(λk)(t − λk)

j

j!+ · · · .

Then

f (Jk) = f (λk)I + f ′(λk)Ek + · · ·+f (mk−1)(λk)E

mk−1k

(mk − 1)!.



Interpolation (1)

Definition (Sylvester, 1883; Buchheim, 1886)Distinct e’vals λ1, . . . , λs, ni = max size of Jordan blocks forλi . Then f (A) = p(A), where p is unique Hermiteinterpolating poly of degree <

∑si=1 ni satisfying

p(j)(λi) = f (j)(λi), j = 0 : ni − 1, i = 1 : s.

Example. Let f (t) = t1/2, A =

[2 21 3

], λ(A) = 1,4.

Taking +ve square roots,

r(t) = f (1)t − 41− 4

+ f (4)t − 14− 1

=13(t + 2).

⇒ A1/2 = r(A) =13(A + 2I) =

13

[4 21 5

].



Interpolation (2)

Propertiesf (A) = r(A) is a polynomial in A.Poly r depends on A.f (A) commutes with A.

f (AT ) = f (A)T .



Cayley–Hamilton Theorem

Theorem (Cayley, 1857)If A,B ∈ Cn×n, AB = BA, and f (x , y) = det(xA− yB) thenf (B,A) = 0.

Usual Cayley–Hamilton is:p(t) = det(tI − A) implies p(A) = 0.An =

∑n−1k=0 cnAk .

eA =∑n−1

k=0 dnAk .



Cauchy Integral Theorem

Definition

f (A) =1

2πi

∫Γ

f (z)(zI − A)−1 dz,

where f is analytic on and inside a closed contour Γ thatencloses λ(A).



Schwerdtfeger’s Formula (1938)

DefinitionFor A with distinct e’vals λ1, . . . , λs with indices ni ,

f (A) =s∑

i=1

Ai

ni−1∑j=0

f (j)(λi)

j!(A− λi I)j =

s∑i=1

ni−1∑j=0

f (j)(λi)Zij ,

Ai are Frobenius covariants, Zij depend on A but not f .



Multiplicity of Definitions

There have been proposed in the literature since 1880eight distinct definitions of a matric function,

by Weyr, Sylvester and Buchheim,Giorgi, Cartan, Fantappiè, Cipolla,

Schwerdtfeger and Richter.

— R. F. Rinehart,The Equivalence of Definitions

of a Matric Function (1955)



Equivalence of Definitions

TheoremThe four definitions are equivalent, modulo analyticityassumption for Cauchy.

Interpolation: for basic properties.JCF: for solving matrix equations (e.g., X 2 = A,eX = A). For evaluation (normal A).Cauchy: various uses.

For computation:

Use the definitions (with care).Schur decomposition for general f .Methods specific to particular f and A.



Root Oddities (1)

B2n = In, where

B4 =

1 1 1 10 −1 −2 −30 0 1 30 0 0 −1

.Arises in BDF solvers for ODEs.

Turnbull (1927): A3n = In, where

A4 =

−1 1 −1 1−3 2 −1 0−3 1 0 0−1 0 0 0

.



Root Oddities (1)

B2n = In, where

B4 =

1 1 1 10 −1 −2 −30 0 1 30 0 0 −1

.Arises in BDF solvers for ODEs.Turnbull (1927): A3

n = In, where

A4 =

−1 1 −1 1−3 2 −1 0−3 1 0 0−1 0 0 0

.University of Manchester Nick Higham Matrix Functions 24 / 162


Root Oddities (2)

C2n = I, where

C4 = 2−3/2

1 3 3 11 1 −1 −11 −1 −1 11 −3 3 −1

.

Hill (1932): US patent for involutory matrices incryptography.

Bauer (2002): “since then the value of mathematicalmethods in cryptology has been unchallenged.”



Nonprimary Matrix Functions

If A is derogatory, and a different branch of f is taken in thetwo different Jordan blocks for λ, a nonprimary matrixfunction of A is obtained.

I2 =

[1 00 1

]2

=

[−1 00 −1

]2

primary

=

[1 00 −1

]2

=

[cos θ sin θsin θ − cos θ

]2

nonprimary

Primary: expressible as a polynomial in A.Nonprimary: never so-expressible.

Not all nonprimary functions are obtainable from the JCFdefinition. Consider A = 0.



Square Roots of Rotations

G(θ) =

[cos θ sin θ− sin θ cos θ

].

G(θ/2) is the natural square root of G(θ).

For θ = π,

G(π) =

[−1 00 −1

], G(π/2) =

[0 1−1 0

].

G(π/2) is a nonprimary square root.

Virtually all existing theory and methods are for primaryfunctions.



Matrix Square Root Example

Find a matrix X such that

X 2 = A =

1 1 00 1 00 0 1

.

A solution is

X =

1 1/2 00 1 00 0 1

.All square roots are given by ±X and

Y = ±U

1 1/2 00 1 00 0 −1

U−1, U =

a b d0 a 00 e c

.





X 2 = A =

1 1 00 1 00 0 1

.A solution is

X =

1 1/2 00 1 00 0 1

.

All square roots are given by ±X and

Y = ±U

1 1/2 00 1 00 0 −1

U−1, U =

a b d0 a 00 e c

.





X 2 = A =

1 1 00 1 00 0 1

.A solution is

X =

1 1/2 00 1 00 0 1

.All square roots are given by ±X and

Y = ±U

1 1/2 00 1 00 0 −1

U−1, U =

a b d0 a 00 e c



Some Open Problems

Find function of minimal norm (possibly nonprimary) orwith particular structure.Does a stochastic A have a stochastic pth root? (H &Lin, 2011).

Very little work. Singer & Spilerman (1976) remains one ofthe best references.



Principal Logarithm and Root

Let A ∈ Cn×n have no eigenvalues on R− .

Principal logX = log A denotes unique X such that

eX = A.−π < Im

(λ(X )

)< π.

Principal pth root

For integer p > 0, X = A1/p is unique X such thatX p = A.−π/p < arg(λ(X )) < π/p.



Principal Power

Arbitrary Power

For s ∈ R, As = es log A, where log A is the principallogarithm.

As =sin(sπ)

sπA∫ ∞

0(t1/sI + A)−1 dt , s ∈ (0,1).



Outline





Basic Properties

f (XAX−1) = Xf (A)X−1;

E’vals of f (A) are f (λi), where the λi are theeigenvalues of A (but f may change the Jordanstructure);

if A = (Aij) is block triang then F = f (A) is block triangwith the same block structure as A, and Fii = f (Aii);



More Advanced

f (A) = 0 iff (with JCF as earlier)

f (j)(λi) = 0, j = 0 : ni − 1, i = 1 : s.

Sum, product, composition of functions work “asexpected”:

(sin+ cos)(A) = sin A + cos A,f (t) = cos(sin t) ⇒ f (A) = cos(sin A),

Functional relations preserved: G(f1, . . . , fp) = 0, whereG is a polynomial. E.g.

sin2 A + cos2 A = I,(A1/p)p = A,eiA = cos A + i sin A.



What Can Go Wrong?

f (A∗) 6= f (A)∗ in general.

elog A = A but log eA 6= A in general.

(AB)1/2 6= A1/2B1/2 in general.

eA 6= (eA/α)α in general.

e(A+B)t = eAteBt for all t if and only if AB = BA.



Function of 2× 2 Triangular Matrix

f([

λ1 t12

0 λ2

])=

[f (λ1) t12f [λ1, λ2]

0 f (λ2)

],

where

f [λ1, λ2] =

f (λ2)− f (λ1)

λ2 − λ1, λ1 6= λ2,

f ′(λ1), λ1 = λ2.

Note: (1,2) element is prone to cancellation.

Proof?



Function of n × n Triangular Matrix

Theorem (Davis, 1973; Descloux, 1963; Van Loan, 1975)If T is upper triangular, so is F = f (T ) and fii = f (tii),

fij =∑

(s0,...,sk )∈Sij

ts0,s1ts1,s2 . . . tsk−1,sk f [λs0 , . . . , λsk ],

whereλi = tii ,Sij is set of all strictly increasing sequences of integersstarting at i and ending at j, andf [λs0 , . . . , λsk ] is kth order divided difference.



Diagonalizable Matrices

Theorem

Let D be an open subset of R or C and let f be n − 1 timescontinuously differentiable on D. Then f (A) = 0 for allA ∈ Cn×n with spectrum in D if and only if f (A) = 0 for alldiagonalizable A ∈ Cn×n with spectrum in D.

Theorem (Richter)

For A ∈ Cn×n with no eigenvalues on R−,

log A =

∫ 1

0(A− I)

[t(A− I) + I

]−1 dt .



Outline





Toolbox of Matrix Functions

d2ydt2 + Ay = 0, y(0) = y0, y ′(0) = y ′0

has solution

y(t) = cos(√

At)y0 +(√

A)−1 sin(

√At)y ′0.

But [y ′

y

]= exp

([0 −tA

t In 0

])[y ′0y0

].

In software want to be able to evaluate interesting f atmatrix args as well as scalar args.MATLAB has expm, logm, sqrtm, funm.



Linear, Constant Coefficient ODE

In nuclear magnetic resonance (NMR) spectroscopy,Solomon equations

dMdt

= −RM, M(0) = I,

where M(t) is matrix of intensities and R a symmetricrelaxation matrix. Thus M(t) = e−Rt .

NMR workers need to solve both forward and inverseproblems.

Burnup calculations in nuclear reactor analysis involve

dXdt

= AX , X (0) = X0,

where A is often upper triangular.University of Manchester Nick Higham Matrix Functions 41 / 162


Application to Complex NetworksAdjacency matrices of undirected network:

1 2

43 A =

0 0 1 00 0 1 11 1 0 10 1 1 0

.Network measures (Estrada et al., 2005–)

Centrality: (eA)ii , how important node i is.Communicability: (eA)ij , how well information istransferred between nodes i and j .

Can use resolvent (I − αA)−1 in place of eA.trace(cosh(A))/trace(eA) is measure of how closegraph is to bipartite.



The Average Eye

First order character of optical system characterized bytransference matrix

T =

[S δ0 1

]∈ R5×5,

where S ∈ R4×4 is symplectic:

ST JS = J =

[0 I2−I2 0

].

Average m−1∑mi=1 Ti is not a transference matrix.

Harris (2005) proposes the average exp(m−1∑mi=1 log(Ti)).



A1/2b: Application in StatisticsChen, Anitescu & Saad (2011): sample y ∼ N(µ,C),C ∈ Rm×m, m ∈ [1012,1015].

Let x ∼ N(0, I).

If C = LLT is Cholesky factorization,y = µ+ Lx ∼ N(µ,C).

Cholesky factorization may not be computable orstorable.

y = µ+ C1/2x ∼ N(µ,C).



Markov Models (1)

Time-homogeneous continuous-time Markov processwith transition probability matrix P(t) ∈ Rn×n.Transition intensity matrix Q: qij ≥ 0 (i 6= j),∑n

j=1 qij = 0, P(t) = eQt .

For discrete-time Markov processes:

Embeddability problemWhen does a given stochastic P havea real logarithm Q that is an intensitymatrix?



Markov Models (2)—Example

With x = −e−2√

3π ≈ −1.9× 10−5,

P =13

1 + 2x 1− x 1− x1− x 1 + 2x 1− x1− x 1− x 1 + 2x

.P diagonalizable, Λ(P) = 1, x , x.Every primary log complex (can’t have complexconjugate ei’vals).Yet a generator is the non-primary log

Q = 2√

3π

−2/3 1/2 1/61/6 −2/3 1/21/2 1/6 −2/3



Markov Models (3)–Practicalities

Let P be transition probability matrix for discrete-timeMarkov process.If P is transition matrix for 1 year,P(1/12) = P1/12 = e

112 log P is matrix for 1 month.

Problem: log P, P1/k may have wrong sign patterns⇒“regularize”.In credit risk, P is strictly diagonally dominant.



Chronic Disease Example

Estimated 6-month transition matrix.Four AIDS-free states and 1 AIDS state.2077 observations (Charitos et al., 2008).

P =

0.8149 0.0738 0.0586 0.0407 0.01200.5622 0.1752 0.1314 0.1169 0.01430.3606 0.1860 0.1521 0.2198 0.08150.1676 0.0636 0.1444 0.4652 0.1592

0 0 0 0 1

.Want to estimate the 1-month transition matrix.

Λ(P) = 1,0.9644,0.4980,0.1493,−0.0043.

H & Lin (2011).Lin (2011) for survey of regularization methods.



Phi Functions: Definition

ϕ0(z) = ez , ϕ1(z) =ez − 1

z, ϕ2(z) =

ez − 1− zz2 , . . .

ϕk+1(z) =ϕk(z)− 1/k !

z.

ϕk(z) =∞∑

j=0

z j

(j + k)!.



Phi Functions: Solving DEs

y ∈ Cn, A ∈ Cn×n.

dydt

= Ay , y(0) = y0 ⇒ y(t) = eAty0.

dydt

= Ay + b, y(0) = 0 ⇒ y(t) = t ϕ1(tA)b.

dydt

= Ay + ct , y(0) = 0 ⇒ y(t) = t2ϕ2(tA)c.

...



Exponential Integrators

Considery ′ = Ly + N(y), y(0) = y0.

N(y(t)) ≈ N(y0) implies

y(t) ≈ etLy0 + tϕ1(tL)N(y0).

Exponential Euler method:

yn+1 = ehLyn + hϕ1(hL)N(yn).

Lawson (1967); recent resurgence.



Implementation of Exponential Integrators

u ′(t) = Au(t) + g(t ,u(t)), u(0) = u0, t ≥ 0.

Let uk = g(k−1)(t ,u(t)) |t=0 and ϕ`(z) =∑∞

k=0 zk/(k + `)!.We need to compute

u(t) = etAu0 +∑p

k=1 ϕk(tA)tk uk .



Saad’s Trick (1992)

ϕ1(z) =ez − 1

z.

exp([

A b0 0

])=

[eA ϕ1(A)b0 1

]



Evaluating Sum of Phi Functions

Theorem (Al-Mohy & H, 2011)

Let A ∈ Cn×n, U = [u1,u2, . . . ,up] ∈ Cn×p, τ ∈ C, and define

B =

[A U0 J

]∈ C(n+p)×(n+p), J =

[0 Ip−1

0 0

]∈ Cp×p.

Then for X = eτB we have

X (1 : n,n + j) =∑j

k=1 τk ϕk(τA)uj−k+1, j = 1 : p.

u(t) =[

In 0]

exp(

t[

A U0 J

])[u0

ep

].



Outline





Fréchet Derivative

Fréchet derivative of f : Cn×n → Cn×n at X ∈ Cn×n

A linear mapping L : Cn×n → Cn×n s.t. for all E ∈ Cn×n

f (X + E)− f (X )− L(X ,E) = o(‖E‖).

Example For f (X ) = X 2 we have

f (X + E)− f (X ) = XE + EX + E2,

so L(X ,E) = XE + EX .



Fréchet Derivative of eA

L(A,E) =

∫ 1

0eA(1−s)EeAs ds.

Simplifies to L(A,E) = EeA = eAE when AE = EA.Another representation:

L(A,E) = E +AE + EA

2!+

A2E + AEA + EA2

3!+ · · · .



Condition Number

cond(f ,A) := limε→0

sup‖E‖≤ε‖A‖

‖f (A + E)− f (A)‖ε‖f (A)‖ .

‖L(A)‖ := maxE 6=0

‖L(A,E)‖‖E‖ .

Lemma

cond(f ,A) =‖L(A)‖‖A‖‖f (A)‖ .



Condition Number of eA

κexp(A) =‖L(A)‖‖A‖‖eA‖ .

‖L(A)‖ ≥ ‖L(A, I)‖ = ‖eA‖ ⇒ κexp(A) ≥ ‖A‖ .

TheoremFor normal A ∈ Cn×n, κexp(A) = ‖A‖2.If A ∈ Rn×n is a nonnegative scalar multiple of astochastic matrix then in the∞-norm, κexp(A) = ‖A‖∞.



Computing Lf : Via 2n × 2n Matrix

TheoremIf f is 2n − 1 times ctsly diffble,

f([

A E0 A

])=

[f (A) Lf (A,E)

0 f (A)

].

Note that Lf (A, αE) = αLf (A,E), but α may effect algused for the evaluation.



Computing Lf : Complex Step

Assume that f : Rn×n → Rn×n and A,E ∈ Rn×n. Then

f (A + ihE)− f (A)− ihLf (A,E) = o(h).

Thus (Al-Mohy & H, 2010)

f (A) ≈ Re f (A + ihE),

Lf (A,E) ≈ Imf (A + ihE)

h.

h not restricted by fl pt arith considerations. Can takeh = 10−100.f alg must not employ complex arith.



Condition Estimation (1)

Since Lf is a linear operator,

vec(Lf (A,E)) = K (A)vec(E)

where K (A) ∈ Cn2×n2 is the Kronecker form of the Fréchetderivative.

Can show ‖Lf (A)‖F = ‖K (A)‖2 .

Let L?f (A) be the adjoint of Lf (A) wrt inner product〈X ,Y 〉 = trace(Y ∗X ). In general, L?f (A) = Lf (A

∗), wheref (z) := f (z).



Power Method

Algorithm

Power method applied to A∗A to produce γ ≤ ‖A‖2.

1 Choose a nonzero starting vector z0 ∈ Cn

2 for k = 0:∞3 wk+1 = Azk

4 zk+1 = A∗wk+1

5 γk+1 = ‖zk+1‖2/‖wk+1‖2

6 if converged, γ = γk+1, quit, end7 end

Normalization needed.Linear convergence.



Condition Estimation (2)

Algorithm (power method on Fréchet derivative)

2-norm power method to produce γ ≤ ‖Lf (A)‖F .

1 Choose a nonzero starting matrix Z0 ∈ Cn×n

2 for k = 0:∞3 Wk+1 = Lf (A,Zk)4 Zk+1 = L?f (A,Wk+1)5 γk+1 = ‖Zk+1‖F/‖Wk+1‖F

6 if converged, γ = γk+1, quit, end7 end

In practice we use instead the block 1-norm estimatorof H & Tisseur (2000).



Outline





Small/Medium Scale Problems

Decompositions:Normal A: if can compute Schur/spectraldecomposition A = QDQ∗, D = diag(di), thenf (A) = Qdiag(f (di))Q∗.Nonnormal A: if can compute Schur decompositionA = QTQ∗ then use Schur–Parlett method.

Matrix iterations: Xk+1 = g(Xk), X0 = A, for matrix roots,sign function, polar decomposition. Require only matrixmult and multiple RHS linear systems.

Approximation methods: polynomial and rational(Taylor, Padé, . . . ), specific to f .



Large Scale f (A)b Problems (1)

A large and sparse,f (A) cannot be stored,the problem is f (A)b: action of f (A) on b.

Case 1: Can solve Ax = b (sparse direct methods) butnot compute Schur decomp: “backslash matrix”.

Cauchy integral formula can be used (Hale, H &Trefethen, 2008) .Rational Krylov can be used with direct solves.



Large Scale f (A)b Problems (2)

Case 2: Can only compute matrix–vector products Ax(and maybe A∗x).

Perhaps: A symmetric, can estimate [λmin, λmax].

Krylov methods.Polynomial approximations.Others?

Exponential integrators for sufficiently large problems are inthis case.



Accuracy Requirements

Full double precision.

Variable tolerance, e.g., within an ODE integrator.

Given tolerance, where matrix A is subject tomeasurement error, e.g., ≈ 10−4 in engineering,healthcare.



Outline





Classic MATLAB< M A T L A B >

Version of 01/10/84

HELP is available

<>helpType HELP followed byINTRO (To get started)NEWS (recent revisions)ABS ANS ATAN BASE CHAR CHOL CHOP CLEA COND CONJ COSDET DIAG DIAR DISP EDIT EIG ELSE END EPS EXEC EXITEXP EYE FILE FLOP FLPS FOR FUN HESS HILB IF IMAGINV KRON LINE LOAD LOG LONG LU MACR MAGI NORM ONESORTH PINV PLOT POLY PRIN PROD QR RAND RANK RCON RATREAL RETU RREF ROOT ROUN SAVE SCHU SHOR SEMI SIN SIZESQRT STOP SUM SVD TRIL TRIU USER WHAT WHIL WHO WHY< > ( ) = . , ; / ’ + - * :



Classic MATLAB<>help fun

FUN For matrix arguments X , the functions SIN, COS, ATAN,SQRT, LOG, EXP and X**p are computed using eigenvalues Dand eigenvectors V . If <V,D> = EIG(X) then f(X) =V*f(D)/V . This method may give inaccurate results if Vis badly conditioned. Some idea of the accuracy can beobtained by comparing X**1 with X .For vector arguments, the function is applied to eachcomponent.

The availability of [FUN] in early versions of MATLABquite possibly contributed to

the system’s technical and commercial success.

— Cleve Moler (2003)



Taylor Series

Matrix Taylor series converges if eigenvalues ofincrement matrix lie within radius of convergence ofseries. Thus for all A,

cos(A) = I − A2

2!+

A4

4!− A6

6!+ · · · .

Can bound error in truncated Taylor series in terms ofappropriate derivative at matrix argument.Usual concerns about numerical cancellation in theevaluation.



George Forsythe “Pitfalls” (1970)



Padé Approximation

Rational rkm(x) = pkm(x)/qkm(x) is a[k ,m] Padé approximant tof (x) =

∑∞i=0 αix i if pkm and qkm are polys

of degree at most k and m and

f (x)− rkm(x) = O(xk+m+1).

Generally more efficient thantruncated Taylor series.Possible representations:

Ratio of polys.Continued fraction.Partial fraction.

Henri Padé1863–1953



Similarity Transformations

Can use the formula

A = XBX−1 ⇒ f (A) = Xf (B)X−1,

provided f (B) is easily computable.E.g. B = diag(λi) if A diagonalizable.

Problem : any error ∆B in f (B) magnified by up toκ(X ) = ‖X‖‖X−1‖ ≥ 1.

Prefer to work with unitary X : thus can useeigendecomposition (diagonal B) when A is normal(AA∗ = A∗A),Schur decomposition (triangular B) in general.



Example: Eigendecompositionfunction F = funm_ev(A,fun)[V,D] = eig(A);F = V * diag(feval(fun,diag(D))) / V;

>> A = [3 -1; 1 1]; X = funm_ev(A,@sqrt)X =

1.7678e+000 -3.5355e-0013.5355e-001 1.0607e+000

>> norm(A-X^2) % cond(V) = 9.4e7ans =

9.9519e-009>> Y = sqrtm(A); norm(A-Y^2)ans =

6.4855e-016



Block Diagonalization

Could instead use a block diagonalization A = XDX−1,where

X well conditioned,D = diag(Di) block diagonal.

Can compute starting with Schur:

Need a parameter : max condition of individualtransformations.Di are triangular but have no particular eigenvaluedistribution, so f (Di) nontrivial.



Parlett’s Recurrence

fii = f (tii) is immediate.

Parlett (1976): from FT = TF obtain recurrence

fij = tijfii − fjjtii − tjj

+

j−1∑k=i+1

fik tkj − tik fkj

tii − tjj.

Used in funm in MATLAB 6.5 (2002) and earlier.

Fails when T has repeated eigenvalues.



Matrix pth Root

Square root: Björck & Hammarling (1983). Compute Schurdecomp. A = QTQ∗, solve U2 = T by

rii =√

tii , rij =tij −

∑j−1k=i+1 uijukj

uii + ujj,

form X = QUQ∗.

Extended to pth roots by Smith (2003)—much morecomplicated recurrence.

These algs

Have essentially optimal numerical stability.Generalize to real Schur decomp.



Parlett vs. Björck & Hammarling

Parlett recurrence is not “optimal”, as clear from sq. rootcase: x12 obtained from

Parlett :a12(√

a11 −√

a22)

a11 − a22=

a12√a11 +

√a22

: B & H.



Block Parlett Recurrence

T = (Tij) block upper triangular with square diagonal blocks.

F = (Fij) has same block structure and Fii = f (Tii).TF = FT leads to Sylvester equations

TiiFij − FijTjj = FiiTij − TijFjj +

j−1∑k=i+1

(FikTkj − TikFkj), i < j .

Compute F a block superdiagonal at a time.

Singular systems possible.So re-order and re-block to produce “well conditioned”systems.



Schur–Parlett Algorithm

H & Davies (2003):

Compute Schur decomposition A = QTQ∗.Re-order T to block triangular form in whicheigenvalues within a block are “close” and those ofseparate blocks are “well separated”.Evaluate Fii = f (Tii).Solve the Sylvester equations

TiiFij − FijTjj = FiiTij − TijFjj +

j−1∑k=i+1

(FikTkj − TikFkj).

Undo the unitary transformations.



Reordering Step

Break the eigenvalues into sets: λi and λj go in sameset if

|λi − λj | ≤ δ = 0.1.

Choose ordering of sets on the diagonal and determinesequence of swaps to produce that ordering.Carry out the swaps by unitary transformations(LAPACK).



Function of Atomic Block

Assume f has Taylor series with∞ radius of cgce andderivatives available.

For diagonal blocks T use

T = σI + M, σ = trace(T )/n : f (T ) =∞∑

k=0

f (k)(σ)k !

Mk .

Truncate series based on strict error bound, not usingsize of terms. NB: for n = 2,

M =

[ε α0 −ε

]⇒ M2k =

[ε2k 00 ε2k

], M2k+1 =

[ε2k+1 αε2k

0 −ε2k+1

].



Features of Algorithm

Costs O(n3) flops, or up to n4/3 flops if large blocksneeded (close or repeated eigenvalues).Needs derivatives if block of size > 1: price to pay fortreating general f and nonnormal A.Parameter δ controls blocking. Default δ = 0.1 goodmost of time.Know example where alg unstable for all δ.Best general f (A) alg. Benchmark for comparing otherf (A) algs—general and specific.The basis of funm in MATLAB.



Example Use of funm

To compute exp(A) + cos(A) with one call to funm, useF = funm(A,@fun_expcos)

where

function f = fun_expcos(x,k)% Return k’th derivative of EXP+COS at X.g = mod(ceil(k/2),2);if mod(k,2)

f = exp(x) + sin(x)*(-1)^g;else

f = exp(x) + cos(x)*(-1)^g;end



Matrix Sign Function

Let A ∈ Cn×n have no pure imaginary eigenvalues and letA = ZJZ−1 be a Jordan canonical form with

J =

[ p q

p J1 0q 0 J2

], Λ(J1) ∈ LHP, Λ(J2) ∈ RHP.

sign(A) = Z[−Ip 00 Iq

]Z−1.

sign(A) = A(A2)−1/2 .

sign(A) =2π

A∫ ∞

0(t2I + A2)−1 dt .



Newton’s Method for Sign

Xk+1 = 12(Xk + X−1

k ), X0 = A.

Convergence

Let S := sign(A), G := (A− S)(A + S)−1. Then

Xk = (I −G2k)−1(I + G2k

)S,

Ei’vals of G are (λi − sign(λi))/(λi + sign(λi)).Hence ρ(G) < 1 and Gk → 0 .Easy to show

‖Xk+1 − S‖ ≤ 12‖X−1

k ‖‖Xk − S‖2.



Newton’s Method for Square Root

Newton’s method: X0 given,

Solve XkEk + EkXk = A− X 2k

Xk+1 = Xk + Ek

k = 0,1,2, . . .

Assume AX0 = X0A. Then can show

Xk+1 = 12(Xk + X−1

k A). (∗)

For nonsingular A, local quadratic cgce of full Newtonto a primary square root.To which square root do the iterations converge?(∗) can converge when full Newton breaks down.Lack of symmetry in (∗).



Convergence (Jordan)

Assume X0 = p(A) for some poly p. Let Z−1AZ = J beJordan canonical form and set Z−1XkZ = Yk . Then

Yk+1 = 12(Yk + Y−1

k J), Y0 = J.

Convergence of diagonal of Yk reduces to scalar case:

Heron: yk+1 = 12

(yk +

λ

yk

), y0 = λ.

Can show that off-diagonal converges.Problem: analysis does not generalize to X0A = AX0!X0 not necessarily a polynomial in A.



Convergence (via Sign)

TheoremLet A ∈ Cn×n have no e’vals on R−. The Newton squareroot iterates Xk with X0A = AX0 are related to the Newtonsign iterates

Sk+1 =12(Sk + S−1

k ), S0 = A−1/2X0

by Xk ≡ A1/2Sk . Hence, provided A−1/2X0 has no pure

imag e’vals, Xk are defined and Xk → A1/2sign(S0)

quadratically.

⇒: Xk → A1/2 if Λ(A−1/2X0) ⊆ RHP, e.g., if X0 = A .



History of Newton Sqrt Instability

Xk+1 = 12(Xk + X−1

k A).

Instability of Newton noted by Laasonen (1958):“Newton’s method if carried out indefinitely, isnot stable whenever the ratio of the largest tothe smallest eigenvalue of A exceeds thevalue 9.”

Described informally by Blackwell (1985) inMathematical People: Profiles and Interviews.Analyzed by H (1986) for diagonalizable A by deriving“error amplification factors”.



Stability

DefinitionThe iteration Xk+1 = g(Xk) is stable in a nbhd of a fixedpoint X if Fréchet derivative dgX has bounded powers.

For scalar, superlinearly cgt g, dgX ≡ g′(x) = 0.

Let X0 = X + E0, Ek := Xk − X . Then

Xk+1 = g(Xk) = g(X + Ek) = g(X ) + dgX (Ek) + o(‖Ek‖).So, since g(X ) = X ,

Ek+1 = dgX (Ek) + o(‖Ek‖).If ‖dg i

X (E)‖ ≤ c, then recurring leads to

‖Ek‖ ≤ c‖E0‖+ kc · o(‖E0‖).



Stability of Newton Square Root

g(X ) = 12(X + X−1A).

dgX (E) = 12(E − X−1EX−1A).

Relevant fixed point: X = A1/2.dgA1/2(E) = 1

2(E − A−1/2EA1/2).

Ei’vals of dgA1/2 are

12(1− λ−1/2

i λ1/2j ), i , j = 1 : n.

For stability we need

maxi,j12

∣∣∣1− λ−1/2i λ

1/2j

∣∣∣ < 1.

For hpd A, need κ2(A) < 9.



Advantages

Uses only Fréchet derivative of g.No additional assumptions on A.Perturbation analysis is all in the definition.General, unifying approach.Facilitates analysis of families of iterations.



Stability of Sign Iterations (H)

Theorem

Let Xk+1 = g(Xk) be any superlinearly convergentiteration for S = sign(X0).Then dgS(E) = LS(E) = 1

2(E − SES) , where LS is theFréchet derivative of the matrix sign function at S.Hence dgS is idempotent (dgS dgS = dgS) and theiteration is stable.

“All” sign iterations are automatically stable.



More Iterations for Sign Function. . .

Start with Newton for the sign function:

xk+1 =12(xk + x−1

k ).

Invert:xk+1 =

2xk

x2k + 1

.

Halley:

xk+1 =xk(3 + x2

k )

1 + 3x2k.

Newton–Schulz:

xk+1 =xk

2(3− x2

k ).



Outline





The f (A)b Problem

Givenmatrix function f : Cn×n → Cn×n,A ∈ Cn×n, b ∈ Cn,

compute f (A)b without first computing f (A).

Most important casesf (x) = x−1

f (x) = ex .f (x) = log(x).f (x) = x1/2.f (x) = sign(x).



A1/2 b via Contour Integration

f (A)b =1

2πi

∫Γ

f (z)(zI − A)−1b dz.

A 5× 5 Pascal matrix: λ(A) ∈ [0.01,92.3], f (z) = z1/2.Use repeated trapezium rule to integrate around circlecentre (λmin + λmax)/2, radius λmax/2.Hale, H & Trefethen (2008): conformally map

Ω = C \(−∞,0] ∪ [m,M ]

.

to an annulus: [m,M ]→ inner circle, (−∞,0]→ outercircle.

# points 2 digits 13 digitsCircle 32,000 262,000Conformal map 5 35Refined 29



Aα b via Binomial Expansion

Write A = s(I − C).If λi > 0, s = (λmin + λmax)/2 yieldsρmin = (λmax − λmin)/(λmax + λmin).For any A, s = trace(A∗A)/trace(A∗) minimizes ‖C‖F .

(I − C)α =∞∑

j=0

(α

j

)(−C)j , ρ(C) < 1.

So

Aαb = sα∞∑

j=0

(α

j

)(−C)jb.

For M-matrices, required splitting with C ≥ 0 always exists.



Aα b via ODE IVP

dydt

= α(A− I)[t(A− I) + I

]−1y , y(0) = b

has unique solution

y(t) =[t(A− I) + I

]αb

soy(1) = Aαb.

Used by Allen, Baglama & Boyd (2000) for α = 1/2, spd A.



Formulae for eA, A ∈ Cn×n

Taylor series Limit Scaling and squaring

I + A +A2

2!+

A3

3!+ · · · lim

s→∞(I + A/s)s (eA/2s

)2s

Cauchy integral Jordan form Interpolation

12πi

∫Γ

ez(zI − A)−1 dz Zdiag(eJk )Z−1n∑

i=1

f [λ1, . . . , λi ]

i−1∏j=1

(A − λj I)

Differential system Schur form Padé approximation

Y ′(t) = AY (t), Y (0) = I QeT Q∗ pkm(A)qkm(A)−1

Krylov methods: Arnoldi fact. AQk = QkHk + hk+1,kqk+1eTk

with Hessenberg H: eAb ≈ Qk eHk Q∗k b.



The Sixth Dubious WayMoler & Van Loan (1978, 2003)



Computing eAB

A︸︷︷︸n×n

, B︸︷︷︸n×n0

, n0 n. Exploit, for integer s,

eAB = (es−1A)sB = es−1Aes−1A · · · es−1A︸︷︷︸s times

B.

Choose s so Tm(s−1A) =∑m

j=0(s−1A)j

j!≈ es−1A. Then

Bi+1 = Tm(s−1A)Bi , i = 0 : s − 1, B0 = B

yields Bs ≈ eAB.

How to choose s and m?



Truncation Analysis

hm+1(A) := log(e−ATm(A)) =∞∑

k=m+1

ck Ak .

Then Tm(A) = eA+hm+1(A). Hence

Tm(2−sA)2s= eA+2shm+1(2−sA) =: eA+∆A.

Aim: select s so that

‖∆A‖‖A‖ =

‖hm+1(2−sA)‖‖2−sA‖ ≤ u ≈ 1.1× 10−16.

Moler & Van Loan (1978): a priori bound for h2m+1 forPadé approximant; m = 6, ‖2−sA‖ ≤ 1/2 in MATLAB.H (2005): used symbolic arithmetic and high precisionto optimize (s,m).



Refined Backward Error Analysis

Lemma (Al-Mohy & H, 2009)

Tm(s−1A)sB = eA+∆AB, where ∆A = shm+1(s−1A) andhm+1(x) = log(e−x Tm(x)) =

∑∞k=m+1 ck xk . Moreover,

‖∆A‖ ≤ s∞∑

k=m+1

|ck | αp(s−1A)k

if m + 1 ≥ p(p − 1), where

αp(A) = max(dp,dp+1), dp = ‖Ap‖1/p.



Bounding the Backward Error

Want to bound norm of h2m+1(X ) =∑∞

k=2m+1 ckX k .

Non-normality implies ρ(A) ‖A‖.

Note that

ρ(A) ≤ ‖Ap‖1/p ≤ ‖A‖, p = 1 : ∞.

and limp→∞ ‖Ap‖1/p = ρ(A).

Use ‖Ap‖1/p instead of ‖A‖ in the truncation bounds.


A =

[0.9 5000 −0.5

].

0 5 10 15 2010

0

102

104

106

108

1010

‖A‖k2‖Ak‖2

(‖A5‖1/52 )k

(‖A10‖1/102 )k

‖Ak‖1/k2


Key Ideas

Use the ‖Ap‖1/p in the truncation bounds for a few smallp.Choose optimal m, s for given precision u.Preprocess A by shifting A← A− µI, µ = trace(A)/n.Terminate Taylor series evaluation when converged towithin u



Algorithm for F = etAB

1 µ = trace(A)/n2 A = A− µI3 [m∗, s] = parameters(tA) % Includes norm estimation.4 F = B, η = etµ/s

5 for i = 1: s6 c1 = ‖B‖∞7 for j = 1:m∗8 B = t AB/(sj), c2 = ‖B‖∞9 F = F + B

10 if c1 + c2 ≤ tol‖F‖∞, quit, end11 c1 = c2

12 end13 F = ηF , B = F14 end



Conditioning of eAB

κexp(A,B) ≤ ‖eA‖F‖B‖F

‖eAB‖F(1 + κexp(A)).

Relative forward error due to roundoff bounded by

ue‖A‖2‖B‖2/‖eAB‖F .

A normal implies κexp(A) = ‖A‖2. Then instability ife‖A‖2 ‖eA‖2‖A‖2.A Hermitian implies spectrum of A− n−1trace(A)Ihas λmax = −λmin ⇒ (normwise) stability!



Comparison with the Sixth Dubious Way

Advantages of our method over the one-step ODEintegrator:

Fully exploits the linearity of the ODE.Backward error based; ODE integrator controls local(forward) errors.Overscaling avoided.

x ex − (1 + x) ex − (1 + x/2)2

9.9e-9 2.2e-16 6.7e-168.9e-9 0 6.7e-16



Experiment 1

Trefethen, Weideman & Schmelzer (2006):A ∈ R9801×9801, 2D Laplacian,-2500*gallery(’poisson’,99).

Compute eαAb, tol = ud .

α = 0.02 α = 1speed mv diff speed mv diff

Alg AH 1 1010 1 47702expv 2.8 403 7.7e-15 1.3 8835 4.2e-15phipm 1.1 172 3.1e-15 0.2 2504 4.0e-15rational 3.8 7 3.3e-14 0.1 7 1.2e-12



Experiment 2A = -gallery(’triw’,20,4.1), bi = cos i , tol = ud .

0 20 40 60 80 100

10−10

10−5

100

105

t

‖etAb‖2

Alg AH

phipm

rational

expv



Experiment 3Harwell–Boeing matrices:

orani678, n = 2529, t = 100, b = [1,1, . . . ,1]T ;bcspwr10, n = 5300, t = 10, b = [1,0, . . . ,0,1]T .

2D Laplacian matrix, poisson. tol = 6× 10−8.

Alg AH ode15stime error time error

orani678 0.13 4e-8 136 2e-6bcspwr10 0.021 7e-7 2.92 5e-5poisson 3.76 2e-6 2.48 8e-6

4poisson 15 9e-6 3.24 1e-1



Krylov Methods

Run Krylov process on A to get Arnoldi factorization

AQk = QkHk + hk+1,kqk+1eTk

with Hessenberg H. Approximate

eAb ≈ Qk f (Hk)Q∗kb.



Comparison with Krylov Methods

Alg AH Krylov Methods

Most time spent in matrix–vector products.

Krylov recurrence and eH

can be significant.

“Direct method”, cost pre-dictable.

Iterative method; needsstopping test.

No parameters to estimate. Select Krylov subspace size.

Storage: 2 vectors Storage: Krylov basis

Evaluation of eAt at multiplepoints on interval.

Can handle mult col B. Need block Krylov method.

Cost tends to ⇑ with ‖A‖. ‖A‖1/2-dependence for negdef A.



Outline





MATLAB: “Built-In”

funm: Schur–Parlett alg.expm: scaling and squaring (H, 2005).expmdemo1 (Padé = old expm), expmdemo2 (Taylor),expmdemo3 (eigensystem).logm: inverse scaling and squaring (H, 2008).sqrtm: Schur method (Björck & Hammarling 1983).

Blog post by Clever Moler about expm:


http://blogs.mathworks.com/cleve/2012/07/23/a-balancing-act-for-the-matrix-exponential


MATLAB: Third-party

The Matrix Function Toolbox:http://www.maths.manchester.ac.uk/~higham/mftoolbox(H, 2008)Expokit: eA (scaling and squaring) and eAb(Krylov)—(Sidje, 1998).Expint: linear combination of φ functions(Berland, Skaflestad & Wright, 2007).


http://www.maths.manchester.ac.uk/~higham/mftoolbox


Matrix Functions in the NAG Library

NAG Toolbox for MATLAB (Mark 23, released 2011):

Matrix exponentialGeneral functions of real symmetric or Hermitianmatrices

NAG C Library (Mark 23, released 2012) also contains:Schur-Parlett algorithm for general matrix functionsMatrix logarithm

NAG Fortran Library Mark 24 the NAG Toolbox forMATLAB (released August 2013) also contain:

Action of the matrix exponential on another matrixCondition number estimation for matrix functions



Matrix Functions in the NAG Library (cont.)

Coming in 2014 . . .

Matrix square root.Matrix power As, s ∈ R.Latest algorithms for matrix exponential and logarithm.Latest Fréchet derivative and condition numberalgorithms for matrix exponential, logarithm and power.

Documentation can be found at:http://www.nag.co.uk/support_documentation.asp


http://www.nag.co.uk/support_documentation.asp


NAG Sponsorship of GGSSS

“NAG will provide personal licence (for personal machines)for the NAG Toolbox for MATLAB (and/or NAG Library -flavour of your choice and for any Fortran programmers theNAG Fortran Compiler). We understand the main product ofinterest will be the NAG Toolbox for MATLAB.”

See email from John Holden, NAG.



Octave

expm: Ward’s alg.logm: eigensystem.sqrtm: Schur method (Björck & Hammarling 1983).thfm: trig, hyperbolic and their inverses. Implementstextbook definitions, in terms of expm, logm, sqrtm etc!



Python: SciPy

linalg.expm: scaling and squaring (H, 2005).linalg.logm

linalg.signm

linalg.cosm

linalg.sinm

linalg.tanm

linalg.coshm

linalg.sinhm

linalg.tanhm

linalg.funm: Schur–Parlett (unblocked!).



Python: SciPy in Progress

linalg/_expm_frechet.py

linalg/_onenormest.py

linalg/_expm_multiply.py

linalg/_sqrtm.py (blocked)


http://nickhigham.wordpress.com/2013/03/10/siam-conference-on-computational-science-and-engineering-2013


R

Goulet, Dutang, Maechler, Firth, Shapira & Stadelmann,R package expm: Matrix exponential, Computation ofthe matrix exponential and related [email protected]://cran.r-project.org/web/packages/expm/index.html

expm (default is expm.Higham08)expmCond

expmFrechet

logm

sqrtm


http://cran.r-project.org/web/packages/expm/index.html


C++

J. Niesen, Matrix functions module for Eigen C++template library for linear algebra.http://eigen.tuxfamily.org/dox-devel//unsupported/group__MatrixFunctions__Module.html

cos coshexp logpow matrixFunctionsin sinhsqrt


http://eigen.tuxfamily.org/dox-devel//unsupported/group__MatrixFunctions__Module.html




Outline





Case Study:Schur Method for Matrix Square Root

For more details of the following, including analysis of bestapproach for parallel computation, see Deadman, Higham& Ralha (2013).



Schur Method: Derivation

Compute Schur decomp A = QTQ∗.

Expand U2 = T , where U upper triangular for primarysquare root:

u2ii = tii ,

uiiuij + uijujj = tij −j−1∑

k=i+1

uikukj .

U is found either a column or a superdiagonal at a time.√

A = QUQ∗



The Schur Method: Properties

Cost: 2813n3 flops.

Real arithmetic version uses real Schur decomp (upperquasi-triang T )

Computed square root U satisfies U2 = T +∆T , where

|∆T | ≤ γn|U|2,

or normwise in real arithmetic.

No use of level 3 BLAS: slow!

Now focus on the triangular phase of the algorithm.



Random Complex Triangular

0 1000 2000 3000 4000 5000 6000 7000 80000

50

100

150

200

250

300

350

400

450

n

time

(s)

point



The Blocked Schur Method

The Uij and Tij are now taken to be blocks:

U2ii = Tii , (1)

UiiUij + UijUjj = Tij −j−1∑

k=i+1

UikUkj . (2)

Solve (1) by point method.Solve Sylvester eqn (2) by, e.g., xTRSYL in LAPACK.

Error bounds unchanged.Over 90% of run time spent in GEMM calls and 8% inSylvester equation solution.Insensitive to block size and choice of kernel routines.



The Blocked Schur Method

The Uij and Tij are now taken to be blocks:

U2ii = Tii , (1)

UiiUij + UijUjj = Tij −j−1∑

k=i+1

UikUkj . (2)

Solve (1) by point method.Solve Sylvester eqn (2) by, e.g., xTRSYL in LAPACK.Error bounds unchanged.Over 90% of run time spent in GEMM calls and 8% inSylvester equation solution.Insensitive to block size and choice of kernel routines.



Random Complex Triangular

0 1000 2000 3000 4000 5000 6000 7000 80000

50

100

150

200

250

300

350

400

450

n

time

(s)

pointblock



The Recursive Blocked Schur Method

Recursive solution of the triangular phase:[U11 U12

0 U22

]2

=

[T11 T12

0 T22

].

U211 = T11 and U2

22 = T22 solved recursively.

Solve U11U12 + U12U22 = T12 using the recursivemethod of Jonsson & Kågström (2002)

Point algs are used when the recursion once thresholdreached (e.g. n = 64).

Same error bound as point alg.



Random Complex Triangular Matrices

0 1000 2000 3000 4000 5000 6000 7000 80000

50

100

150

200

250

300

350

400

450

n

time

(s)

pointblockrecursion



Full Complex Matrices

0 200 400 600 800 1000 1200 1400 1600 1800 20000

50

100

150

200

250

n

time

(s)

sqrtmfort_pointfort_recurse



Full Real Matrices

0 200 400 600 800 1000 1200 1400 1600 1800 20000

50

100

150

200

250

n

time

(s)

sqrtm_realsqrtmfort_point_realfort_recurse_real


http://blog.nag.com/2012/07/matrix-square-root-blocking-and.html

http://blog.nag.com/2013/01/matrix-functions-in-parallel.html


Outline




Liu Hui

Carl Friedrich Gauss Myrick Hascall Doolittle

André-Louis Cholesky Prescott Durand Crout John von Neumann

Liu Hui Carl Friedrich Gauss

Myrick Hascall Doolittle


Liu Hui Carl Friedrich Gauss Myrick Hascall Doolittle



André-Louis Cholesky

Prescott Durand Crout John von Neumann


André-Louis Cholesky Prescott Durand Crout

John von Neumann

Paul Sumner Dwyer

John Todd Alan Turing

Cornelius Lanczos George Forsythe

Paul Sumner Dwyer John Todd

Alan Turing


Paul Sumner Dwyer John Todd Alan Turing



Cornelius Lanczos

George Forsythe

Peter Lancaster

Olga Taussky-Todd English Electric Lightning

Peter Lancaster

Olga Taussky-Todd

English Electric Lightning

Peter Lancaster

Olga Taussky-Todd English Electric Lightning


Outline





How to Do ResearchDevelop new metric for solved problem then develop anew alg (or tweak an existing one) so it does best onthat metric. E.g. measure communcation costs as wellas arithmetic.

Find hidden assumptions in existing method andremove them. E.g.: Strassen realized use of innerproducts was not necessary for matrix multiplication.Find simpler proof of existing result.Prove (weaker version of) existing result under weakerassumptions.Develop alg that is faster and more accurate thanexisting ones.Develop alg that can solve problems other algs can’t.E.g.: y ∼ N(µ,C) problem in these slides.



How to Do ResearchDevelop new metric for solved problem then develop anew alg (or tweak an existing one) so it does best onthat metric. E.g. measure communcation costs as wellas arithmetic.Find hidden assumptions in existing method andremove them. E.g.: Strassen realized use of innerproducts was not necessary for matrix multiplication.

Find simpler proof of existing result.Prove (weaker version of) existing result under weakerassumptions.Develop alg that is faster and more accurate thanexisting ones.Develop alg that can solve problems other algs can’t.E.g.: y ∼ N(µ,C) problem in these slides.



How to Do ResearchDevelop new metric for solved problem then develop anew alg (or tweak an existing one) so it does best onthat metric. E.g. measure communcation costs as wellas arithmetic.Find hidden assumptions in existing method andremove them. E.g.: Strassen realized use of innerproducts was not necessary for matrix multiplication.Find simpler proof of existing result.

Prove (weaker version of) existing result under weakerassumptions.Develop alg that is faster and more accurate thanexisting ones.Develop alg that can solve problems other algs can’t.E.g.: y ∼ N(µ,C) problem in these slides.



How to Do ResearchDevelop new metric for solved problem then develop anew alg (or tweak an existing one) so it does best onthat metric. E.g. measure communcation costs as wellas arithmetic.Find hidden assumptions in existing method andremove them. E.g.: Strassen realized use of innerproducts was not necessary for matrix multiplication.Find simpler proof of existing result.Prove (weaker version of) existing result under weakerassumptions.

Develop alg that is faster and more accurate thanexisting ones.Develop alg that can solve problems other algs can’t.E.g.: y ∼ N(µ,C) problem in these slides.



How to Do ResearchDevelop new metric for solved problem then develop anew alg (or tweak an existing one) so it does best onthat metric. E.g. measure communcation costs as wellas arithmetic.Find hidden assumptions in existing method andremove them. E.g.: Strassen realized use of innerproducts was not necessary for matrix multiplication.Find simpler proof of existing result.Prove (weaker version of) existing result under weakerassumptions.Develop alg that is faster and more accurate thanexisting ones.

Develop alg that can solve problems other algs can’t.E.g.: y ∼ N(µ,C) problem in these slides.



How to Do ResearchDevelop new metric for solved problem then develop anew alg (or tweak an existing one) so it does best onthat metric. E.g. measure communcation costs as wellas arithmetic.Find hidden assumptions in existing method andremove them. E.g.: Strassen realized use of innerproducts was not necessary for matrix multiplication.Find simpler proof of existing result.Prove (weaker version of) existing result under weakerassumptions.Develop alg that is faster and more accurate thanexisting ones.Develop alg that can solve problems other algs can’t.E.g.: y ∼ N(µ,C) problem in these slides.



Reproducible Research

Wikipediahttp://en.wikipedia.org/wiki/Reproducibility

“Reproducibility . . . refers to the ability of an entireexperiment or study to be reproduced, or by someoneelse working independently.”“The term reproducible research was first proposedby Jon Claerbout at Stanford University and refers tothe idea that the ultimate product of research is thepaper along with the full computational environmentused to produce the results in the paper such as thecode, data, etc. necessary for reproduction of theresults and building upon the research.”


http://en.wikipedia.org/wiki/Reproducibility


Reproducible Research: Writing

Irreproducible research [NJH 1980s]:paper.tex, paper.tex, paper.tex, . . .

Reproducible but error-prone research [NJH 1990s]:paper1.tex, paper2.tex, paper3.tex, . . . ,

Reproducible research [NJH since 2010]: paper.texin a version controlled repository.

Use of repository also benefits collaboration with co-authors(or just use Dropbox with personal repos).

See the ICERM 2012 workshophttp://icerm.brown.edu/tw12-5-rcemparticularly the introduction by Randy LeVeque.


http://icerm.brown.edu/tw12-5-rcem


Git

Git is a widely used distributed version control system.Command line, wysywig interfaces, Emacs interface,. . .gitinfo.sty enables version information to automaticallybe placed into a LATEX document.Github is a popular website to host and browse gitrepositories.


http://git-scm.com

https://github.com/


Other Toolslatexdiff is a Perl script:

latexdiff paper_old.tex paper_new.tex ...> paper_diff.tex

“Need to Know” page for theManchester Numerical LinearAlgebra Research group, withinformation about “LaTeX -BibTeX - Beamer - VersionControl - Emacs - MATLAB -Computing - Refereeing - NewsSources - Societies - Dropbox-Personal Web Page - Travelsupport - CV”.


http://www.ctan.org/tex-archive/support/latexdiff

http://www.maths.manchester.ac.uk/~higham/misc/need2know.php


Literate Programming and RR

JSS Journal of Statistical SoftwareJanuary 2012, Volume 46, Issue 3. http://www.jstatsoft.org/

A Multi-Language Computing Environment for

Literate Programming and Reproducible Research

Eric SchulteUniversity of New Mexico

Dan DavisonCounsyl

Thomas DyeUniversity of Hawai‘i

Carsten DominikUniversity of Amsterdam

Abstract

We present a new computing environment for authoring mixed natural and com-puter language documents. In this environment a single hierarchically-organized plaintext source file may contain a variety of elements such as code in arbitrary program-ming languages, raw data, links to external resources, project management data, workingnotes, and text for publication. Code fragments may be executed in situ with graphical,numerical and textual output captured or linked in the file. Export to LATEX, HTML,LATEX beamer, DocBook and other formats permits working reports, presentations andmanuscripts for publication to be generated from the file. In addition, functioning purecode files can be automatically extracted from the file. This environment is implementedas an extension to the Emacs text editor and provides a rich set of features for authoringboth prose and code, as well as sophisticated project management capabilities.

Keywords: literate programming, reproducible research, compendium, WEB, Emacs.

1. Introduction

There are a variety of settings in which it is desirable to mix prose, code, data, and compu-tational results in a single document.

Scientific research increasingly involves the use of computational tools. Successful com-munication and verification of research results requires that this code is distributedtogether with results and explanatory prose.

In software development the exchange of ideas is accomplished through both code and



Emacs + Org-mode + Python in RR

Presentation by John Kitchin at SciPy 2013.


http://www.youtube.com/watch?v=1-dUkyn_fZA&feature=youtu.be


Professional Use of Social Media

SIAM Annual Meeting Minisymposium, San Diego, 2013

Tammy Kolda: how to make your publications easilyavailable online and various ways to maintain such alist.

David Gleich: survey of social media tools.

NJH: how and why to blog and tweet.

Karthika Muthukumaraswamy: importance ofblogging for scientific communication.


http://nickhigham.wordpress.com/2013/06/30/siam-annual-meeting-minisymposium-on-professional-use-of-social-media


Journal Publication Delays

Median time to first review, 2009–present:SISC 3.7 monthsSIMAX 3.4 months

Average months from submission to acceptance:SISC 9.2 monthsSIMAX 11.1 months

Don’t be afraid to contact SIAM to ask for update onpaper’s status. Use “Send ManuscriptCorrespondence” option once logged in to the journal’sweb site.



Householder Symposium

The Householder Symposium XIX on Numerical LinearAlgebra will be held in Spa, Belgium, 8-13 June 2014.

Web page: http://sites.uclouvain.be/HHXIX/Application deadline: 31 October 2013Notification of acceptance: 31 January 2014

Attendance at the meeting is by invitation only.Applications are welcome from researchers innumerical linear algebra, matrix theory, andrelated areas such as optimization, differentialequations, signal processing, and control....The Symposium is very informal, with the interminglingof young and established researchers a priority.


References I

A. H. Al-Mohy and N. J. Higham.A new scaling and squaring algorithm for the matrixexponential.SIAM J. Matrix Anal. Appl., 31(3):970–989, 2009.

A. H. Al-Mohy and N. J. Higham.The complex step approximation to the Fréchetderivative of a matrix function.Numer. Algorithms, 53(1):133–148, 2010.

A. H. Al-Mohy and N. J. Higham.Computing the action of the matrix exponential, with anapplication to exponential integrators.SIAM J. Sci. Comput., 33(2):488–511, 2011.


References II

E. J. Allen, J. Baglama, and S. K. Boyd.Numerical approximation of the product of the squareroot of a matrix with a vector.Linear Algebra Appl., 310:167–181, 2000.

F. L. Bauer.Decrypted Secrets: Methods and Maxims ofCryptology.Springer-Verlag, Berlin, third edition, 2002.ISBN 3-540-42674-4.xii+474 pp.


References III

H. Berland, B. Skaflestad, and W. Wright.EXPINT—A MATLAB package for exponentialintegrators.ACM Trans. Math. Software, 33(1):Article 4, 2007.

Å. Björck and S. Hammarling.A Schur method for the square root of a matrix.Linear Algebra Appl., 52/53:127–140, 1983.

G. Boyd, C. A. Micchelli, G. Strang, and D.-X. Zhou.Binomial matrices.Adv. in Comput. Math., 14:379–391, 2001.


References IV

A. Cayley.A memoir on the theory of matrices.Philos. Trans. Roy. Soc. London, 148:17–37, 1858.

T. Charitos, P. R. de Waal, and L. C. van der Gaag.Computing short-interval transition matrices of adiscrete-time Markov chain from partially observeddata.Statistics in Medicine, 27:905–921, 2008.

J. Chen, M. Anitescu, and Y. Saad.Computing f (A)b via least squares polynomialapproximations.SIAM J. Sci. Comput., 33(1):195–222, 2011.


References V

T. Crilly.Cayley’s anticipation of a generalised Cayley–Hamiltontheorem.Historia Mathematica, 5:211–219, 1978.

T. Crilly.Arthur Cayley: Mathematician Laureate of the VictorianAge.Johns Hopkins University Press, Baltimore, MD, USA,2006.ISBN 0-8018-8011-4.xxi+610 pp.


References VI

P. I. Davies and N. J. Higham.A Schur–Parlett algorithm for computing matrixfunctions.SIAM J. Matrix Anal. Appl., 25(2):464–485, 2003.

C. Davis.Explicit functional calculus.Linear Algebra Appl., 6:193–199, 1973.


References VII

E. Deadman, N. J. Higham, and R. Ralha.Blocked Schur algorithms for computing the matrixsquare root.In P. Manninen and P. Öster, editors, Applied Paralleland Scientific Computing: 11th InternationalConference, PARA 2012, Helsinki, Finland, volume7782 of Lecture Notes in Computer Science, pages171–182. Springer-Verlag, Berlin, 2013.

J. Descloux.Bounds for the spectral norm of functions of matrices.Numer. Math., 15:185–190, 1963.


References VIII

E. Estrada and D. J. Higham.Network properties revealed through matrix functions.SIAM Rev., 52(4):696–714, 2010.

G. E. Forsythe.Pitfalls in computation, or why a math book isn’tenough.Amer. Math. Monthly, 77(9):931–956, 1970.


References IX

R. A. Frazer, W. J. Duncan, and A. R. Collar.Elementary Matrices and Some Applications toDynamics and Differential Equations.Cambridge University Press, Cambridge, UK, 1938.xviii+416 pp.1963 printing.

J. F. Grcar.How ordinary elimination became Gaussian elimination.Historia Mathematica, 38(2):163–218, 2011.

J. F. Grcar.Mathematics of Gaussian elimination.Notices Amer. Math. Soc., 58(8):782–792, 2011.


References X

D. A. Grier.When Computers Were Human.Princeton University Press, Princeton, NJ, USA, 2005.ISBN 0-691-09157-9.viii+411 pp.

N. Hale, N. J. Higham, and L. N. Trefethen.Computing Aα, log(A), and related matrix functions bycontour integrals.SIAM J. Numer. Anal., 46(5):2505–2523, 2008.

W. F. Harris.The average eye.Opthal. Physiol. Opt., 24:580–585, 2005.


References XI

N. J. Higham.Newton’s method for the matrix square root.Math. Comp., 46(174):537–549, Apr. 1986.

N. J. Higham.The scaling and squaring method for the matrixexponential revisited.SIAM J. Matrix Anal. Appl., 26(4):1179–1193, 2005.


References XII

N. J. Higham.Functions of Matrices: Theory and Computation.Society for Industrial and Applied Mathematics,Philadelphia, PA, USA, 2008.ISBN 978-0-898716-46-7.xx+425 pp.

N. J. Higham and A. H. Al-Mohy.Computing matrix functions.Acta Numerica, 19:159–208, 2010.

N. J. Higham and L. Lin.On pth roots of stochastic matrices.Linear Algebra Appl., 435(3):448–463, 2011.


References XIII

N. J. Higham and F. Tisseur.A block algorithm for matrix 1-norm estimation, with anapplication to 1-norm pseudospectra.SIAM J. Matrix Anal. Appl., 21(4):1185–1201, 2000.

I. Jonsson and B. Kågström.Recursive blocked algorithms for solving triangularsystems—Part I: One-sided and coupled Sylvester-typematrix equations.ACM Trans. Math. Software, 28(4):392–415, 2002.


References XIV

P. Laasonen.On the iterative solution of the matrix equationAX 2 − I = 0.Math. Tables Aids Comput., 12(62):109–116, 1958.

J. D. Lawson.Generalized Runge-Kutta processes for stable systemswith large Lipschitz constants.SIAM J. Numer. Anal., 4(3):372–380, Sept. 1967.


References XV

L. Lin.Roots of Stochastic Matrices and Fractional MatrixPowers.PhD thesis, The University of Manchester, Manchester,UK, 2010.117 pp.MIMS EPrint 2011.9, Manchester Institute forMathematical Sciences.

C. B. Moler and C. F. Van Loan.Nineteen dubious ways to compute the exponential of amatrix, twenty-five years later.SIAM Rev., 45(1):3–49, 2003.


References XVI

K. H. Parshall.James Joseph Sylvester. Jewish Mathematician in aVictorian World.Johns Hopkins University Press, Baltimore, MD, USA,2006.ISBN 0-8018-8291-5.xiii+461 pp.

M. Pusa and J. Leppänen.Computing the matrix exponential in burnupcalculations.Nuclear Science and Engineering, 164:140–150, 2010.


References XVII

Y. Saad.Analysis of some Krylov subspace approximations tothe matrix exponential operator.SIAM J. Numer. Anal., 29(1):209–228, Feb. 1992.

E. Schulte, D. Davison, T. Dye, and C. Dominik.A multi-language computing environment for literateprogramming and reproducible research.Journal of Statistical Software, 46(3):1–24, 2012.


References XVIII

D. She, Y. Liu, K. Wang, G. Yu, B. Forget, P. K. Romano,and K. Smith.Development of burnup methods and capabilities inMonte Carlo code RMC.Annals of Nuclear Energy, 51:289–294, 2013.

R. B. Sidje.Expokit: A software package for computing matrixexponentials.ACM Trans. Math. Software, 24(1):130–156, 1998.


References XIX

B. Singer and S. Spilerman.The representation of social processes by Markovmodels.Amer. J. Sociology, 82(1):1–54, 1976.

M. I. Smith.A Schur algorithm for computing matrix pth roots.SIAM J. Matrix Anal. Appl., 24(4):971–989, 2003.

L. N. Trefethen, J. A. C. Weideman, and T. Schmelzer.Talbot quadratures and rational approximations.BIT, 46(3):653–670, 2006.


References XX

H. W. Turnbull.The matrix square and cube roots of unity.J. London Math. Soc., 2(8):242–244, 1927.

C. F. Van Loan.A study of the matrix exponential.Numerical Analysis Report No. 10, University ofManchester, Manchester, UK, Aug. 1975.Reissued as MIMS EPrint 2006.397, ManchesterInstitute for Mathematical Sciences, The University ofManchester, UK, November 2006.


functions of matrices - department of mathematicshigham/misc/ggsss13/fun.pdf · in the interview....

Documents