1 numerical geometry of non-rigid shapes multidimensional scaling multidimensional scaling ©...

1Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

Multidimensional scaling

© Alexander & Michael Bronstein, 2006-2009© Michael Bronstein, 2010tosca.cs.technion.ac.il/book

048921 Advanced topics in visionProcessing and Analysis of Geometric Shapes

EE Technion, Spring 2010

2Numerical Geometry of Non-Rigid Shapes Multidimensional scalingIf it doesn’t fit, you must acquit!If it doesn’t fit, you must acquit!

Image: Associated Press


Metric model

Euclidean metric

Invariant to rigid

motion

EXTRINSIC GEOMETRY

Similarity = isometry w.r.t.

Shape = metric space

Extrinsic geometry is not invariant

under inelastic deformations


Metric model

Geodesic metric

Invariant to inelastic

deformation

INTRINSIC GEOMETRY

Euclidean metric

Invariant to rigid

motion

EXTRINSIC GEOMETRY

Similarity = isometry w.r.t.

Shape = metric space


Intrinsic vs. extrinsic similarity

INTRINSIC SIMILARITYEXTRINSIC SIMILARITY

Part of the same metric space Two different metric spaces

SOLUTION: Find a representation of and

in a common metric space


Canonical forms

Isometric embedding


Canonical form distance

Compute canonical formsEXTRINSIC SIMILARITY OF CANONICAL FORMS

INTRINSIC SIMILARITY

= INTRINSIC SIMILARITY


Mapmaker’s problem


Mapmaker’s problem

A sphere has non-zero curvature, therefore, it

is not isometric to the plane

(a consequence of Theorema egregium)

Karl Friedrich Gauss (1777-1855)


Linial’s exampleA

C D

1 1

2

AC D

1 1

1 1

2C D

B

A

B

C

D

B

A

C D

1 1

B

1

Conclusion: generally, isometric embedding does not exist!

No contradiction to Nash embedding theorem: Nash guarantees that any

Riemannian structure can be realized as a length metric induced by

Euclidean metric. We try to realize it using restricted Euclidean metric


Minimum distortion embedding

Minimum distortion embedding


Multidimensional scaling

The problem is called multidimensional scaling (MDS)

Find an embedding into that distorts the distances the least by

solving the optimization problem

The function measuring the distortion of distances is called stress

where are the coordinates of the canonical form


Stress

L2-stress

The stress is a function of the canonical form coordinates and

the distances

Lp-stress

L-stress (distortion)


- an matrix of canonical form coordinates (each row

corresponds to a point)

Matrix expression of L2-stress

Some notation:

Shorthand notation for Euclidean distances

Write the stress as

1 2


Term 1


Term 1

where is and matrix with elements

Matrices commute

under trace operator


Term 2

For and

zero otherwise


Term 2

where is and matrix-valued function with elements


variables

Non-linear

Non-convex (thus liable to local convergence)

Optimum defined up to Euclidean transformation

LS-MDS

Minimization of L2-stress is called least squares MDS (LS-MDS)


Gradient of L2-stress

Recall exercise

in optimization

Quadratic in Nonlinear in Constant in


Stress computation complexity:

Complexity

Steepest descent

with line search may be prohibitively expensive (requires multiple

evaluations of the stress and the gradient)

Stress gradient computation complexity:


Observation

for all and

By Cauchy-Schwarz inequality

Equality is achieved for

Write it differently:


Observation

Consider the nonlinear term in the stress


Majorizing inequality

[de Leeuw, 1977]

Jan de Leeuw

Equality is achieved for

We have a quadratic majorizing function to

use in iterative majorization algorithm!


Iterative majorization

Construct a majorizing function satisfying

.

Majorizing inequality: for all

is convex



Start with some

Find such that

Update iterate

Increment iteration counter

Solution

Until

convergence



The majorizing function

is quadratic!

Analytic expression for the minimim

Analytic expression for

pseudoinverse:


SMACOF algorithm

Start with some

Update iterate


Solution

Until

convergence

The algorithm is called SMACOF (Scaling by Minimizing A COnvex

Function)


SMACOF is a constant-step gradient descent!

SMACOF algorithm vs steepest descent

Majorization guarantees monotonically decreasing sequence of

stress

values (untypical behavior for constant-step steepest descent)

No guarantee of global convergence

Iteration cost:


MATLAB® intermezzoSMACOF algorithm

Canonical forms


Examples of canonical forms

Canonical forms

Near-isometric deformations of a shape


Examples of canonical forms

Visualization of intrinsic similarity


Weighted stress

where are some fixed weights

where is and matrix with elements

Matrix expression

SMACOF algorithm can be used to minimize weighted stress!


Variations on the stress theme

Generic form of the stress

where is some norm

For example, gives the Lp-stress

The necessary condition for to be the minimizer of is


Variations on the stress theme

Idea: minimize weighted stress instead of the generic stress and choose

the weights such that the two are equivalent

If the weights are selected as

the minimizers of and will coincide

Since is unknown in advance, iterate!


Iteratively reweighted LS

Start with some and

Find

using as an initialization

Update weights


Until

convergence

The algorithm is called Iteratively Reweighted Least Squares (IRLS)


L-stress

Optimization problem

can be written equivalently as a constrained problem

is called an artificial variable


Complexity, bis

N=1000 points

MDS

N=200 points

MDS

x5 less pointsx25 lower complexity


Multiresolution MDS

Bottom-up approach: solve coarse level MDS problem to initialize fine

level problem

Reduce complexity (less fine-level iterations)

Reduce the risk of local convergence (good initialization)

Can be performed on multiple resolution levels


Solve coarse level problem

Interpolate

Two-resolution MDS

Solve finelevel problem

Grid

Data

Interpolation operator to transfer solution

from

coarse level to fine level

Can be represented as an sparse matrix


Solve coarsest level problem

Interpolate

Multiresolution MDS

Solve -st level problem

Interpolate

Solve finestlevel problem


Multiresolution MDS algorithm

Hierarchy of grids

Hierarchy of data

Interpolation operators

Start with coarsest-level initialization

Solve the l-th level MDS problem

using as initialization

Interpolate to next level

For

Solve finest level problem



Top-down approach

Start with a fine-level initialization

Decimate fine-level initialization to the coarse grid

Solve the coarse-level MDS problem


Improve the fine-level solution propagating the coarse-level error

Typically


A problem

Fine level

Coarse level


Correction

The fine- and the coarse-level minimizers do not coincide!

is called a residual

Force consistency of the coarse- and fine-level problems by introducing a

correction term into the stress

which gives the gradient


Correction

Fine level

Coarse level


Correction

In order to guarantee consistency, must satisfy

at

which gives the correction term

Verify: is the minimizer of coarse-level problem with correction,

Chain rule


Modified stress

Another problem: the coarse grid problem

is unbounded for

Fix the translation ambiguity by adding a quadratic penalty to the stress

Can grow arbitrarily

is called the modified stress [Bronstein et al., 05]

The modified coarse grid problem is bounded


Solve corrected coarse level problem

Two-grid MDS

Correct fine level solution

Decimate Interpolate

Iteratively improve the fine-level solution using coarse grid residual

External iteration is called cycle


Solve corrected coarse level problem

Two-grid MDS

Correct fine level solution

Decimate Interpolate

Perform a few optimization iterations at fine level between cycles

(relaxation)

Relax


Two-grid MDS algorithm

Start with fine-level initialization

Produce an improved fine-level solution by

making a few iterations on initialized with

Decimate fine-level solution

Correction:

Solve the coarse-level corrected MDS problem


Transfer residual:

Update iterate

Solution

Until

convergence


Solve coarsest level problem

Interpolatecorrect

Multigrid MDS

Relax

Decimate

Relax

Decimate

Interpolatecorrect

Relax

Relax

Apply the two-grid algorithm recursively

Different cycles possible, e.g. V-cycle


MATLAB® intermezzoMultigrid MDS


Convergence example

Time (sec)

Str

ess

Convergence of SMACOF and Multigrid MDS (N=2145)


How to accelerate convergence?

Let be a sequence of iterates produced by some

optimization algorithms converging to

Denote by the remainder

Construct a new sequence converging faster to

in the sense


Vector extrapolation

Construct the new sequence as a transformation of iterates

Particular choice: linear combination

Question: how to choose the coefficients ?

Ideally, hence


Reduced rank extrapolation (RRE)

Problem: depends on unknown

Eliminate this dependence by considering the difference

Result: solve the constrained linear system

which ideally should vanish.

of equations in variables

Typically hence the system is over-determined and must be

solved approximately using least-squares


Start with some and

Generate iterates

using optimization on initialized by

Extrapolate from

Update


Until

convergence

Optimization with vector extrapolation


Start with some and

Generate iterates

using optimization on initialized by

Extrapolate from

If

else


Until

convergence

Safeguarded optimization with vector extrapolation


MATLAB® intermezzoRRE MDS

1 numerical geometry of non-rigid shapes multidimensional scaling multidimensional scaling ©...

Documents

multidimensional scaling

restricted euclidean

stress distortion

length metric

euclidean distances

common metric space

canonical form coordinates

matrix of canonical