multidimensional scaling - techniontosca.cs.technion.ac.il/book/handouts/technionee2010_mds.pdf ·...

1

1Processing & Analysis of Geometric Shapes Introduction to Geometry

Multidimensional scaling

© Alexander & Michael Bronstein, 2006-2009© Michael Bronstein, 2010tosca.cs.technion.ac.il/book

048921 Advanced topics in visionProcessing and Analysis of Geometric Shapes

EE Technion, Spring 2010

2Numerical Geometry of Non-Rigid Shapes Multidimensional scaling

If it doesn’t fit, you must acquit!If it doesn’t fit, you must acquit!

Image: Associated Press


Metric model

� Euclidean metric

� Invariant to rigid motion

EXTRINSIC GEOMETRY

Similarity = isometry w.r.t.Shape = metric space

Extrinsic geometry is not invariant

under inelastic deformations


Metric model

� Geodesic metric

� Invariant to inelastic deformation

INTRINSIC GEOMETRY

� Euclidean metric

� Invariant to rigid motion

EXTRINSIC GEOMETRY

Similarity = isometry w.r.t.Shape = metric space


Intrinsic vs. extrinsic similarity

INTRINSIC SIMILARITYEXTRINSIC SIMILARITY

Part of the same metric space Two different metric spaces

SOLUTION: Find a representation of and

in a common metric space


Canonical forms

Isometric embedding

2


Canonical form distance

Compute canonical formsEXTRINSIC SIMILARITY OF CANONICAL FORMS

INTRINSIC SIMILARITY

= INTRINSIC SIMILARITY


Mapmaker’s problem


Mapmaker’s problem

A sphere has non-zero curvature, therefore, it

is not isometric to the plane

(a consequence of Theorema egregium)

Karl Friedrich Gauss (1777-1855)


Linial’s exampleA

C D

1 1

2

AC D

1 1

1 1

2C D

B

A

B

C

D

B

A

C D

1 1

B

1

Conclusion: generally, isometric embedding does not exist!

No contradiction to Nash embedding theorem: Nash guarantees that any

Riemannian structure can be realized as a length metric induced by

Euclidean metric. We try to realize it using restricted Euclidean metric


Minimum distortion embedding

Minimum distortion embedding


Multidimensional scaling

� The problem is called multidimensional scaling (MDS)

Find an embedding into that distorts the distances the least by

solving the optimization problem

� The function measuring the distortion of distances is called stress

where are the coordinates of the canonical form

3


Stress

� L2-stress

The stress is a function of the canonical form coordinates and

the distances

� Lp-stress

� L∞∞∞∞-stress (distortion)


� - an matrix of canonical form coordinates (each row

corresponds to a point)

Matrix expression of L 2-stress

Some notation:

� Shorthand notation for Euclidean distances

Write the stress as

1 2


Term 1


Term 1

where is and matrix with elements

Matrices commute

under trace operator


Term 2

For and

zero otherwise


Term 2

where is and matrix-valued function with elements

4


� variables

� Non-linear

� Non-convex (thus liable to local convergence )

� Optimum defined up to Euclidean transformation

LS-MDS

Minimization of L2-stress is called least squares MDS (LS-MDS)


Gradient of L 2-stress

Recall exercise

in optimization

Quadratic in Nonlinear in Constant in


� Stress computation complexity:

Complexity

� Steepest descent

with line search may be prohibitively expensive (requires multiple

evaluations of the stress and the gradient)

� Stress gradient computation complexity:


Observation

for all and

By Cauchy-Schwarz inequality

Equality is achieved for

Write it differently:


Observation

Consider the nonlinear term in the stress


Majorizing inequality

[de Leeuw, 1977]

Jan de Leeuw

Equality is achieved for

We have a quadratic majorizing function to

use in iterative majorization algorithm!

5


Iterative majorization

Construct a majorizing function satisfying

� .

� Majorizing inequality: for all

� is convex



� Start with some

� Find such that

� Update iterate

� Increment iteration counter

� Solution

Until

convergence



The majorizing function

is quadratic !

Analytic expression for the minimim

Analytic expression for

pseudoinverse:


SMACOF algorithm

� Start with some

� Update iterate


� Solution

Until

convergence

The algorithm is called SMACOF (Scaling by Minimizing A COnvex

Function)


SMACOF is a constant-step gradient descent!

SMACOF algorithm vs steepest descent

� Majorization guarantees monotonically decreasing sequence of stress

values (untypical behavior for constant-step steepest descent)

� No guarantee of global convergence

� Iteration cost:


MATLAB ® intermezzoSMACOF algorithm

Canonical forms

6


Examples of canonical forms

Canonical forms

Near-isometric deformations of a shape


Examples of canonical forms

Visualization of intrinsic similarity


Weighted stress

where are some fixed weights

where is and matrix with elements

Matrix expression

SMACOF algorithm can be used to minimize weighted stress!


Variations on the stress theme

Generic form of the stress

where is some norm

For example, gives the Lp-stress

The necessary condition for to be the minimizer of is


Variations on the stress theme

Idea: minimize weighted stress instead of the generic stress and choose

the weights such that the two are equivalent

If the weights are selected as

the minimizers of and will coincide

Since is unknown in advance, iterate!


Iteratively reweighted LS

� Start with some and

� Find

using as an initialization

� Update weights


Until

convergence

The algorithm is called Iteratively Reweighted Least Squares (IRLS)

7


L∞∞∞∞-stress

Optimization problem

can be written equivalently as a constrained problem

is called an artificial variable


Complexity, bis

N=1000 points

MDS

N=200 points

MDS

x5 less pointsx25 lower complexity


Multiresolution MDS

� Bottom-up approach: solve coarse level MDS problem to initialize fine

level problem

� Reduce complexity (less fine-level iterations)

� Reduce the risk of local convergence (good initialization )

� Can be performed on multiple resolution levels


Solve coarse level problem

Interpolate

Two-resolution MDS

Solve finelevel problem

� Grid

� Data

� Interpolation operator to transfer solution from

coarse level to fine level

Can be represented as an sparse matrix


Solve coarsest level problem

Interpolate

Multiresolution MDS

Solve -st level problem

Interpolate

Solve finestlevel problem


Multiresolution MDS algorithm

� Hierarchy of grids

� Hierarchy of data

� Interpolation operators

� Start with coarsest-level initialization

� Solve the l-th level MDS problem

using as initialization

� Interpolate to next level

For

� Solve finest level problem


8


Top-down approach

� Start with a fine-level initialization

� Decimate fine-level initialization to the coarse grid

� Solve the coarse-level MDS problem


� Improve the fine-level solution propagating the coarse-level error

Typically


A problem

Fine level

Coarse level


Correction

The fine- and the coarse-level minimizers do not coincide!

is called a residual

Force consistency of the coarse- and fine-level problems by introducing a

correction term into the stress

which gives the gradient


Correction

Fine level

Coarse level


Correction

In order to guarantee consistency, must satisfy

at

which gives the correction term

Verify: is the minimizer of coarse-level problem with correction,

Chain rule


Modified stress

Another problem: the coarse grid problem

is unbounded for

Fix the translation ambiguity by adding a quadratic penalty to the stress

Can grow arbitrarily

is called the modified stress [Bronstein et al., 05]

The modified coarse grid problem is bounded

9


Solve corrected coarse level problem

Two-grid MDS

Correct fine level solution

Decimate Interpolate

� Iteratively improve the fine-level solution using coarse grid residual

� External iteration is called cycle


Solve corrected coarse level problem

Two-grid MDS

Correct fine level solution

Decimate Interpolate

� Perform a few optimization iterations at fine level between cycles

(relaxation )

Relax


Two-grid MDS algorithm

� Start with fine-level initialization

� Produce an improved fine-level solution by

making a few iterations on initialized with

� Decimate fine-level solution

� Correction:

� Solve the coarse-level corrected MDS problem


� Transfer residual:

� Update iterate

� Solution

Until

convergence


Solve coarsest level problem

Interpolatecorrect

Multigrid MDS

Relax

Decimate

Relax

Decimate

Interpolatecorrect

Relax

Relax

� Apply the two-grid algorithm recursively

� Different cycles possible, e.g. V-cycle


MATLAB ® intermezzoMultigrid MDS


Convergence example

Time (sec)

Str

ess

Convergence of SMACOF and Multigrid MDS (N=2145)

10


How to accelerate convergence?

Let be a sequence of iterates produced by some

optimization algorithms converging to

Denote by the remainder

Construct a new sequence converging faster to

in the sense


Vector extrapolation

Construct the new sequence as a transformation of iterates

Particular choice: linear combination

Question: how to choose the coefficients ?

Ideally, hence


Reduced rank extrapolation (RRE)

Problem: depends on unknown

Eliminate this dependence by considering the difference

Result: solve the constrained linear system

which ideally should vanish.

of equations in variables

Typically hence the system is over-determined and must be

solved approximately using least-squares



� Generate iterates

using optimization on initialized by

� Extrapolate from

� Update


Until

convergence

Optimization with vector extrapolation



� Generate iterates

using optimization on initialized by

� Extrapolate from

� If

� else


Until

convergence

Safeguarded optimization with vector extrapolation


MATLAB ® intermezzoRRE MDS

multidimensional scaling - techniontosca.cs.technion.ac.il/book/handouts/technionee2010_mds.pdf ·...

Documents