pierre-antoine absil (dept. of mathematical engineering ... · optimization algorithms on matrix...
TRANSCRIPT
![Page 1: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/1.jpg)
Optimization and curve fitting on manifolds
Pierre-Antoine Absil(Dept. of Mathematical Engineering, UCLouvain)
Journee conjointe des GDR MIA et ISIS
Optimisation Geometrique sur les Varietes
21st November 2014
1
![Page 2: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/2.jpg)
Optimization on Manifolds in one picture
Mf
R
2
![Page 3: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/3.jpg)
Optimization on Manifolds in one picture
Mf
R
x
3
![Page 4: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/4.jpg)
A bookhttp://press.princeton.edu/titles/8586.html
Optimization Algorithms on Matrix ManifoldsP.-A. Absil, R. Mahony, R. SepulchrePrinceton University Press, January 2008
1. Introduction2. Motivation and applications3. Matrix manifolds: first-order geometry4. Line-search algorithms5. Matrix manifolds: second-order geometry6. Newton’s method7. Trust-region methods8. A constellation of superlinear algorithms
4
![Page 5: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/5.jpg)
A toolboxhttp://www.manopt.org/
Ref: Nicolas Boumal et al, Manopt, a Matlab toolbox for optimization onmanifolds, JMLR 15(Apr) 1455-1459, 2014.
5
![Page 6: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/6.jpg)
Optimization on manifolds: an introduction Motivation and problem formulation
Optimization on Manifolds in one picture
Mf
R
x
6
![Page 7: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/7.jpg)
Optimization on manifolds: an introduction Motivation and problem formulation
Why general manifolds? – Motivating examples
Given A = AT ∈ Rn×n Given A = AT ∈ R
n×n,and N = diag(p, p − 1, . . . , 1),
min f (X ) = − trace(XTAXN) min f (Y ) = − trace(
(Y TY )−1(Y TAY ))
subj. to X ∈ Rn×p : XTX = I subj. to Y ∈ R
n×p∗ (i.e., Y full rank)
f
R
Y
R
f
YM
f (YM) = f (Y )
Feasible set: St(p, n) Feasible set: Gr(p, n)
= {X ∈ Rn×p : XTX = I} =
{
{YM : M ∈ Rp×p∗ } : Y ∈ Rn×p
∗
}
Embedded submanifold Quotient manifold
7
![Page 8: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/8.jpg)
Optimization on manifolds: an introduction Specific manifolds
Optimization on Manifolds in one picture
Mf
R
x
8
![Page 9: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/9.jpg)
Optimization on manifolds: an introduction Specific manifolds
Specific manifolds, and where they appear
◮ Stiefel manifold St(p, n) and orthogonal group Op = St(n, n)
St(p, n) = {X ∈ Rn×p : XTX = Ip}
Applications: computer vision; principal component analysis;independent component analysis...
◮ Grassmann manifold Gr(p, n)
Set of all p-dimensional subspaces of Rn
Applications: various dimension reduction problems...
◮ Set of fixed-rank PSD matrices S+(p, n). A quotient representation:
X ∼ Y ⇔ ∃Q ∈ Op : Y = XQ
Applications: Low-rank approximation of symmetric matrices;algorithms for (large-scale) semidefinite programming...
9
![Page 10: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/10.jpg)
Optimization on manifolds: an introduction Specific manifolds
Specific manifolds, and where they appear
◮ Low-rank manifold Rm×nrkp
Rm×nrkp = {M ∈ R
m×n : rk(M) = p}
Applications: dimensionality reduction; model for matrixcompletion...
◮ Shape manifold On\Rn×p∗
Y ∼ X ⇔ ∃U ∈ On : Y = UX
Applications: shape analysis
◮ Oblique manifold Rn×p∗ /Sdiag+
Rn×p∗ /Sdiag+ ≃ {Y ∈ R
n×p∗ : diag(Y TY ) = Ip}
Applications: blind source separation; factor analysis (obliqueProcrustes problem)...
10
![Page 11: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/11.jpg)
Optimization on manifolds: an introduction Mathematical background
Smooth optimization problems on general manifolds
Mf
R
x
f ∈ C∞(x)?
ϕ(U)
Rd
ϕ
Yes ifff ◦ ϕ−1 ∈ C∞(ϕ(x))
ψ
UV
ψ(V)ϕ(U ∩ V) ψ(U ∩ V)
ψ ◦ ϕ−1
ϕ ◦ ψ−1
C∞
Rd
11
![Page 12: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/12.jpg)
Optimization on manifolds: an introduction Mathematical background
Optimization on manifolds in its most abstract formulation
Mf
R
x
f ∈ C∞(x)?
ϕ(U)
Rd
ϕ
Yes ifff ◦ ϕ−1 ∈ C∞(ϕ(x))
ψ
UV
ψ(V)ϕ(U ∩ V) ψ(U ∩ V)
ψ ◦ ϕ−1
ϕ ◦ ψ−1
C∞
Rd
Given:
◮ A set M endowed (explicitly or implicitly) with a manifold structure(i.e., a collection of compatible charts).
◮ A function f : M → R, smooth in the sense of the manifoldstructure.
Task: Compute a local minimizer of f .
12
![Page 13: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/13.jpg)
Optimization on manifolds: an introduction Algorithms on abstract manifolds
Algorithms formulated on abstract manifolds
◮ Steepest-descentNeeds: Riemannian structure and retraction
◮ NewtonNeeds: affine connection and retraction
◮ Conjugate GradientsNeeds: Riemannian structure, retraction, and vector transport
◮ BFGSNeeds: needs Riemannian structre, retraction, and vector transport
◮ Trust RegionNeeds: Riemannian structure and retraction
13
![Page 14: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/14.jpg)
Optimization on manifolds: an introduction Algorithms on abstract manifolds
Steepest descent on abstract manifolds
Required: Riemannian manifold M; retraction R on M.Iteration xk ∈ M 7→ xk+1 ∈ M defined by
1. Compute steepest-descent direction in TxkM:
ηk = − grad f (xk).
2. Setxk+1 := Rxk (tkηk)
where tk is chosen using a line-search method.R
fx
x+
grad f (x)
14
![Page 15: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/15.jpg)
Optimization on manifolds: an introduction Algorithms on abstract manifolds
Newton on abstract manifolds
Required: Riemannian manifold M; retraction R on M; affineconnection ∇ on M; real-valued function f on M.Iteration xk ∈ M 7→ xk+1 ∈ M defined by
1. Solve the Newton equation
Hess f (xk)ηk = − grad f (xk)
for the unknown ηk ∈ TxkM, where
Hess f (xk)ηk := ∇ηk grad f .
2. Setxk+1 := Rxk (ηk).
15
![Page 16: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/16.jpg)
Optimization on manifolds: an introduction Algorithms on abstract manifolds
Newton on submanifolds of Rn
Required: Riemannian submanifold M of Rn; retraction R on M;real-valued function f on M.Iteration xk ∈ M 7→ xk+1 ∈ M defined by
1. Solve the Newton equation
Hess f (xk)ηk = − grad f (xk)
for the unknown ηk ∈ TxkM, where
Hess f (xk)ηk := PTxkMDgrad f (xk)[ηk ].
2. Setxk+1 := Rxk (ηk).
16
![Page 17: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/17.jpg)
Optimization on manifolds: an introduction Algorithms on abstract manifolds
Newton on the unit sphere Sn−1
Required: real-valued function f on Sn−1.Iteration xk ∈ Sn−1 7→ xk+1 ∈ Sn−1 defined by
1. Solve the Newton equation
{
Pxk D(grad f )(xk)[ηk ] = − grad f (xk)
xTηk = 0,
for the unknown ηk ∈ Rn, where
Pxk = (I − xkxTk ).
2. Set
xk+1 :=xk + ηk‖xk + ηk‖
.
17
![Page 18: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/18.jpg)
Optimization on manifolds: an introduction Algorithms on abstract manifolds
Newton for Rayleigh quotient optimization on unit sphere
Iteration xk ∈ Sn−1 7→ xk+1 ∈ Sn−1 defined by
1. Solve the Newton equation
{
Pxk APxk ηk − ηkxTk Axk = −Pxk Axk ,
xTk ηk = 0,
for the unknown ηk ∈ Rn, where
Pxk = (I − xkxTk ).
2. Set
xk+1 :=xk + ηk‖xk + ηk‖
.
18
![Page 19: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/19.jpg)
Optimization on manifolds: an introduction Algorithms on abstract manifolds
Conjugate Gradients on abstract manifolds
Require: Riemannian manifold M; vector transport T on M withassociated retraction R ; real-valued function f on M; initial iteratex0 ∈ M.
1: Set η0 = − grad f (x0).2: for k = 0, 1, 2, . . . do
3:Compute a step size αk and set
xk+1 = Rxk (αkηk). (1)
4:Compute βk+1 and set
ηk+1 = − grad f (xk+1) + βk+1Tαkηk (ηk). (2)
5: end for
Fletcher-Reeves: βk+1 =〈grad f (xk+1),grad f (xk+1)〉
〈grad f (xk),grad f (xk )〉.
Polak-Ribiere: βk+1 =〈grad f (xk+1),grad f (xk+1)−Tαkηk
(grad f (xk ))〉
〈grad f (xk),grad f (xk)〉.
Ref: PAA et al [AMS08, §8.3], Sato & Iwai [SI13].19
![Page 20: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/20.jpg)
Optimization on manifolds: an introduction Algorithms on abstract manifolds
BFGS on abstract manifolds
1: Given: Riemannian manifold M with Riemannian metric g ; vectortransport T on M with associated retraction R ; smooth real-valuedfunction f on M; initial iterate x0 ∈ M; initial Hessian approximationB0.
2: for k = 0, 1, 2, . . . do3: Obtain ηk ∈ TxkM by solving Bkηk = − grad f (xk).4: Compute step size αk and set xk+1 = Rxk (αkηk).5: Define sk = Tαηk (αηk) and yk = grad f (xk+1)− Tαηk (grad f (xk)).6: Define the linear operator Bk+1 : Txk+1
M → Txk+1M by
Bk+1p = Bkp−g(sk , Bkp)
g(sk , Bksk)Bksk+
g(yk , p)
g(yk , sk)yk for all p ∈ Txk+1
M,
(3)with
Bk = Tαηk ◦ Bk ◦ (Tαηk )−1. (4)
7: end for
Ref: Qi et al [QGA10], Ring & Wirth [RW12].20
![Page 21: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/21.jpg)
Optimization on manifolds: an introduction Algorithms on abstract manifolds
Trust region on abstract manifolds
y
v1
M
TyM
my
ηy+
Refs: PAA et al [ABG07], Huang et al [HAG14].
21
![Page 22: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/22.jpg)
Optimization on manifolds: an introduction A brief history
Optimization on Manifolds in one picture
Mf
R
x
22
![Page 23: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/23.jpg)
Optimization on manifolds: an introduction A brief history
Some classics on Optimization On Manifolds (I)
R
f
x
x+
Luenberger (1973), Introduction to linear and nonlinear programming.Luenberger mentions the idea of performing line search along geodesics,“which we would use if it were computationally feasible (which itdefinitely is not)”.
23
![Page 24: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/24.jpg)
Optimization on manifolds: an introduction A brief history
Some classics on Optimization On Manifolds (II)
Gabay (1982), Minimizing a differentiable function over a differentialmanifold. Stepest descent along geodesics; Newton’s method alonggeodesics; Quasi-Newton methods along geodesics.
Smith (1994), Optimization techniques on Riemannian manifolds.Levi-Civita connection ∇; Riemannian exponential; parallel translation.But Remark 4.9: If Algorithm 4.7 (Newton’s iteration on the sphere forthe Rayleigh quotient) is simplified by replacing the exponential updatewith the update
xk+1 =xk + ηk‖xk + ηk‖
then we obtain the Rayleigh quotient iteration.
24
![Page 25: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/25.jpg)
Optimization on manifolds: an introduction A brief history
Some classics on Optimization On Manifolds (III)
Manton (2002), Optimization algorithms exploiting unitary constraints“The present paper breaks with tradition by not moving alonggeodesics”. The geodesic update Expx η is replaced by a projectiveupdate π(x + η), the projection of the point x + η onto the manifold.
Adler, Dedieu, Shub, et al. (2002), Newton’s method on Riemannianmanifolds and a geometric model for the human spine. The exponentialupdate is relaxed to the general notion of retraction. The geodesic canbe replaced by any (smoothly prescribed) curve tangent to the searchdirection.
25
![Page 26: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/26.jpg)
Optimization on manifolds: an introduction A brief history
Optimization on Manifolds in one picture
Mf
R
x
26
![Page 27: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/27.jpg)
Application: curve fitting
An Application:Curve Fitting
27
![Page 28: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/28.jpg)
Application: curve fitting
Sources
◮ Nonparametric curve fitting on manifolds:◮ Chafik Samir, PAA, Anuj Srivastava, Eric Klassen, A gradient-descent
method for curve fitting on Riemannian manifolds, Foundations ofComputational Mathematics, 12(1), pp. 49-73, 2012.
◮ Nicolas Boumal, PAA, Discrete regression methods on the cone ofpositive-definite matrices, ICASSP 2011.
◮ Nicolas Boumal, PAA, A discrete regression method on manifolds andits application to data on SO(n), IFAC World Congress 2011.
◮ Parametric curve fitting on manifolds (see Pierre-Yves’s talk):◮ C. Samir, P. Van Dooren, D. Laurent, K. A. Gallivan, PAA, Elastic
morphing of 2D and 3D objects on a shape manifold, Lecture Notesin Computer Science, Volume 5627/2009, pp. 563-572, 2009
◮ Pierre-Yves Gousenbourger, Chafik Samir, PAA, Piecewise-Bezier C 1
interpolation on Riemannian manifolds with application to 2D shapemorphing, ICPR 2014
◮ Antoine Arnould, Pierre-Yves Gousenbourger, Chafik Samir, PAA,Fitting Smooth Paths on Riemannian manifolds: Endometrial SurfaceReconstruction and Preoperative MRI-Based Navigation, submitted.
28
![Page 29: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/29.jpg)
Application: curve fitting Motivation and problem formulation
Curve fitting on manifolds
Γ
R
E
M
p0
p1
p2
γ(t0)
γ(t1)γ(t2)
γ
29
![Page 30: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/30.jpg)
Application: curve fitting Motivation and problem formulation
Curve fitting on manifolds: application to morphing
Γ
R
E
Shape manifold
γ
30
![Page 31: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/31.jpg)
Application: curve fitting Motivation and problem formulation
Curve fitting on manifolds: possible applications
Γ
R
E
M
p0
p1
p2
γ(t0)
γ(t1)γ(t2)
γ
Applications in noise reduction, resampling, and trajectory generation.
◮ Evolution of the paleomagnetic north pole, as in Jupp andKent [JK87]: M = S2, the sphere.
◮ Rigid body motion: M = SE(3), the special Euclidean group.◮ Diffusion-Tensor MRI: M = S++
3 , the set of all 3× 3 symmetricpositive-definite matrices.
◮ Morphing: M is a shape manifold.◮ ...
31
![Page 32: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/32.jpg)
Application: curve fitting Motivation and problem formulation
Curve fitting on manifolds: problem considered
Γ
R
E
M
p0
p1
p2
γ(t0)
γ(t1)γ(t2)
γ
Given: Riemannian manifold M; p0, . . . , pN ∈ M;0 = t0 < · · · < tN = 1.Goal: find the curve γ : [0, 1] 7→ M that minimizes
E2 : Γ2 → R : E2(γ) = Ed(γ) + λEs,2(γ)
=1
2
N∑
i=0
d2(γ(ti ), pi ) +λ
2
∫ 1
0〈D2γ
dt2,D2γ
dt2〉 dt,
where Γ2 is the Sobolev space H2([0, 1],M).32
![Page 33: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/33.jpg)
Application: curve fitting Previous work
Previous work
Machado and Silva Leite [ML06, Mac06] consider
E2 : Γ2 → R : E2(γ) =1
2
N∑
i=0
d2(γ(ti ), pi ) +λ
2
∫ 1
0〈D2γ
dt2,D2γ
dt2〉 dt,
and obtain the Euler-Lagrange equations (stationarity conditions):On each subinterval,
D4γ
dt4+ R
(
D2γ
dt2, γ
)
γ = 0,
and at the knot points,
Dkγ
dtk(t+i )−
Dkγ
dtk(t−i ) =
0, k = 0, 1, (i = 1, . . . ,N − 1)
0, k = 2, (i = 0, . . . ,N)1λExp−1
γ(ti )(pi ), k = 3, (i = 0, . . . ,N)
,
withD2γ
dt2(t−0 ) =
D3γ
dt3(t−0 ) =
D2γ
dt2(t+N ) =
D3γ
dt3(t+N ) = 0.
33
![Page 34: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/34.jpg)
Application: curve fitting Previous work
Gradient-descent for discretized E2
Objective: E2(γ) =12
∑Ni=0 d
2(γ(ti ), pi ) +λ2
∫ 10 〈
D2γdt2
, D2γ
dt2〉 dt.
◮ Finite differences in Rn:
x0 =2
∆tf +∆tb
1
∆tf∆tb[∆tb(xf − x0) + ∆tf(xb − x0)] +O(∆t)
(5)
34
![Page 35: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/35.jpg)
Application: curve fitting Previous work
Gradient-descent for discretized E2
Objective: E2(γ) =12
∑Ni=0 d
2(γ(ti ), pi ) +λ2
∫ 10 〈
D2γdt2
, D2γ
dt2〉 dt.
◮ Finite differences in Rn:
x0 =2
∆tf +∆tb
1
∆tf∆tb[∆tb(xf − x0) + ∆tf(xb − x0)] +O(∆t)
(5)
◮ Finite differences on a manifold:
x0 ≈2
∆tf +∆tb
1
∆tf∆tb
[
∆tb Logx0 (xf) + ∆tf Logx0 (xb)]
(6)
35
![Page 36: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/36.jpg)
Application: curve fitting Previous work
Gradient-descent for discretized E2
Objective: E2(γ) =12
∑Ni=0 d
2(γ(ti ), pi ) +λ2
∫ 10 〈
D2γdt2
, D2γ
dt2〉 dt.
◮ Finite differences in Rn:
x0 =2
∆tf +∆tb
1
∆tf∆tb[∆tb(xf − x0) + ∆tf(xb − x0)] +O(∆t)
(5)
◮ Finite differences on a manifold:
x0 ≈2
∆tf +∆tb
1
∆tf∆tb
[
∆tb Logx0 (xf) + ∆tf Logx0 (xb)]
(6)
◮ Discretized E2:
E2 : MNd → R : E2(γ) =
1
2
N∑
i=0
d2(pi , γsi ) +λ
2
Nd∑
i=1
βi‖ai‖2γi
36
![Page 37: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/37.jpg)
Application: curve fitting Previous work
Illustrations on the sphere
Objective: E2(γ) =12
∑Ni=0 d
2(γ(ti ), pi ) +λ2
∫ 10 〈
D2γdt2
, D2γ
dt2〉 dt.
λ = 10−4 λ = 10−3 λ = 100
37
![Page 38: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/38.jpg)
Application: curve fitting Parametric curve fitting
Curve fitting on manifolds
Γ
R
E
M
p0
p1
p2
γ(t0)
γ(t1)γ(t2)
γ
38
![Page 39: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/39.jpg)
Application: curve fitting Parametric curve fitting
Curve fitting on manifolds: application to morphing
Γ
R
E
Shape manifold
γ
39
![Page 40: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/40.jpg)
Application: curve fitting Parametric curve fitting
Polynomial interpolation on manifolds
◮ Polynomial interpolation reminder: Given (t0, x0), . . . , (tn, xn) in Rd ,
there is one and only one polynomial pn of degree at most n suchthat p(tk) = xk , k = 0, . . . , n.
40
![Page 41: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/41.jpg)
Application: curve fitting Parametric curve fitting
Polynomial interpolation on manifolds
◮ Polynomial interpolation reminder: Given (t0, x0), . . . , (tn, xn) in Rd ,
there is one and only one polynomial pn of degree at most n suchthat p(tk) = xk , k = 0, . . . , n.
◮ pn(t) can be computed with Neville’s algorithm, based on theformula
Pi ,j(t) = Pi ,j−1(t) +t − titj − ti
(Pi+1,j(t)− Pi ,j−1(t)) , (7)
where Pi ,j stands for the polynomial of degree at most j − i thatinterpolates (ti , xi ), . . . , (tj , xj). We have pn(t) = P0,n(t).
41
![Page 42: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/42.jpg)
Application: curve fitting Parametric curve fitting
Polynomial interpolation on manifolds
◮ Polynomial interpolation reminder: Given (t0, x0), . . . , (tn, xn) in Rd ,
there is one and only one polynomial pn of degree at most n suchthat p(tk) = xk , k = 0, . . . , n.
◮ pn(t) can be computed with Neville’s algorithm, based on theformula
Pi ,j(t) = Pi ,j−1(t) +t − titj − ti
(Pi+1,j(t)− Pi ,j−1(t)) , (7)
where Pi ,j stands for the polynomial of degree at most j − i thatinterpolates (ti , xi ), . . . , (tj , xj). We have pn(t) = P0,n(t).
◮ When x0, . . . , xn are on a manifold, (7) readily generalizes to
Pi ,j(t) = ExpPi,j−1(t)
(
t − titj − ti
LogPi,j−1(t) Pi+1,j(t)
)
.
42
![Page 43: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/43.jpg)
Application: curve fitting Parametric curve fitting
Piecewise-polynomial interpolation on manifolds
◮ Polynomial interpolation on manifolds is prone to the Rungephenomenon.
43
![Page 44: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/44.jpg)
Application: curve fitting Parametric curve fitting
Piecewise-polynomial interpolation on manifolds
◮ Polynomial interpolation on manifolds is prone to the Rungephenomenon.
◮ Polynomial interpolation on manifolds is also prone to a Runge-likeeffect!
44
![Page 45: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/45.jpg)
Application: curve fitting Parametric curve fitting
Piecewise-polynomial interpolation on manifolds
◮ Polynomial interpolation on manifolds is prone to the Rungephenomenon.
◮ Polynomial interpolation on manifolds is also prone to a Runge-likeeffect!
◮ Remedy: Piecewise-polynomial interpolation on manifolds.
45
![Page 46: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/46.jpg)
Application: curve fitting Parametric curve fitting
Piecewise-polynomial interpolation on manifolds
◮ Polynomial interpolation on manifolds is prone to the Rungephenomenon.
◮ Polynomial interpolation on manifolds is also prone to a Runge-likeeffect!
◮ Remedy: Piecewise-polynomial interpolation on manifolds.
◮ See Pierre-Yves Gousenbourger’s talk later today.
46
![Page 47: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/47.jpg)
Conclusion
Optimization on Manifolds in one picture
Mf
R
x
47
![Page 48: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/48.jpg)
Conclusion
P.-A. Absil, C. G. Baker, and K. A. Gallivan, Trust-region methodson Riemannian manifolds, Found. Comput. Math. 7 (2007), no. 3,303–330.
P.-A. Absil, R. Mahony, and R. Sepulchre, Optimization algorithmson matrix manifolds, Princeton University Press, Princeton, NJ,2008.
Wen Huang, P.-A. Absil, and K. A. Gallivan, A riemanniansymmetric rank-one trust-region method, MathematicalProgramming (2014), accepted for publication.
Peter E. Jupp and John T. Kent, Fitting smooth paths to sphericaldata, J. Roy. Statist. Soc. Ser. C 36 (1987), no. 1, 34–46. MRMR887825 (88f:62066)
Luıs Miguel Faustino Machado, Least squares problems onRiemannian manifolds, Ph.D. thesis, Department of Mathematics,University of Coimbra, 2006.
48
![Page 49: Pierre-Antoine Absil (Dept. of Mathematical Engineering ... · Optimization Algorithms on Matrix Manifolds P.-A. Absil, R. Mahony, R. Sepulchre Princeton University Press, January](https://reader033.vdocuments.net/reader033/viewer/2022043020/5f3c8d9ca555160b773dcc0d/html5/thumbnails/49.jpg)
Conclusion
Luıs Machado and F. Silva Leite, Fitting smooth paths onRiemannian manifolds, Int. J. Appl. Math. Stat. 4 (2006), no. J06,25–53.
Chunhong Qi, Kyle A. Gallivan, and P.-A. Absil, An efficient bfgsalgorithm for riemannian optimization, Proceedings of the 19thInternational Symposium on Mathematical Theory of Networks andSystems (MTNS 2010), 2010, pp. 2221–2227.
Wolfgang Ring and Benedikt Wirth, Optimization methods onRiemannian manifolds and their application to shape space, SIAM J.Optim. 22 (2012), no. 2, 596–627.
Hiroyuki Sato and Toshihiro Iwai, Convergence analysis for theriemannian conjugate gradient method, 2013, arXiv:1302.0125.
49