Recent developments in nonlinear dimensionality

reduction

Josh Tenenbaum

MIT

Collaborators

• Vin de Silva

• John Langford

• Mira Bernstein

• Mark Steyvers

• Eric Berger

Outline

• The problem of nonlinear dimensionality reduction

• The Isomap algorithm

• Development #1: Curved manifolds

• Development #2: Sparse approximations

Learning an appearance map

• Given input: . . .

• Desired output:– Intrinsic dimensionality: 3– Low-dimensional

representation:

Linear dimensionality reduction: PCA, MDS

• PCA dimensionality of faces:

• First two

PCs:

• Linear manifold: PCA

• Nonlinear manifold: ?

Previous approaches to nonlinear dimensionality reduction

• Local methods seek a set of low-dimensional models, each valid over a limited range of data:– Local PCA

– Mixture of factor analyzers

• Global methods seek a single low-dimensional model valid over the whole data set:– Autoencoder neural networks– Self-organizing map– Elastic net– Principal curves & surfaces– Generative topographic mapping

A generative model

• Latent space Y Rd

• Latent data {yi} Y generated from p(Y)

• Mapping f: YRN for some N > d

• Observed data {xi = f (yi)} RN

Goal: given {xi}, recover f and {yi}.

Chicken-and-egg problem

• We know {xi} . . .

• . . . and if we knew{yi}, could estimate f.

• . . . or if we knew f, could estimate {yi}.

• So use EM, right? Wrong.

The problem of local minima

GTM SOM

• Global nonlinear dimensionality reduction + local optimization = severe local minima

A different approach

• Attempt to infer {yi} directly from {xi}, without explicit reference to f.

• Closed-form, non-iterative, globally optimal solution for {yi}.

• Then can approximate f with a suitable interpolation algorithm (RBFs, local linear, ...).

• In other words, finding f becomes a supervised learning problem on pairs {yi ,xi}.

When does this work?

• Only given some assumptions on the nature of f and the distribution of the {yi}.

• The trick: exploit some invariant of f, a property of the {yi} that is preserved in the {xi}, and that allows the {yi} to be read off uniquely*.

* up to some isomorphism (e.g., rotation).

The assumptions behind three algorithms

No free lunch: weaker assumptions on f stronger assumptions on p(Y).

Distribution: p(Y) Mapping: f Algorithm

ii) convex, dense isometric Isomap

iii) convex, uniformly dense conformal C-Isomap

i) ii) iii)

i) arbitrary linear isometric Classical MDS

The assumptions behind three algorithms

Distribution: p(Y) Mapping: f Algorithm

ii) convex, dense isometric Isomap

iii) convex, uniformly dense conformal C-Isomap

i)

i) arbitrary linear isometric Classical MDS

Classical MDS

• Invariant: Euclidean distance • Algorithm:

– Calculate Euclidean distance matrix D– Convert D to canonical inner product matrix B by

“double centering”:

– Compute {yi} from eigenvectors of B.

ijij

jij

iijijij d

nd

nd

ndb 2

2222 111

2

1

The assumptions behind three algorithms

Distribution: p(Y) Mapping: f Algorithm

ii) convex, dense isometric Isomap

iii) convex, uniformly dense conformal C-Isomap

ii)

i) arbitrary linear isometric Classical MDS

Isomap

• Invariant: geodesic distance

The Isomap algorithm• Construct neighborhood graph G.

– method– K method

• Compute shortest paths in G, with edge ij weighted by the Euclidean distance |xi - xj|.

– Floyd – Dijkstra (+ Fibonacci heaps)

• Reconstruct low-dimensional latent data {yi}.

– Classical MDS on graph distances– Sparse MDS with landmarks

Illustration on swiss roll

Discovering the dimensionality

• Measure residual variance in geodesic distances . . .

• . . . and find the elbow.

MDS / PCA

Isomap

Theoretical analysis of asymptotic convergence

• Conditions for PAC-style asymptotic convergence– Geometric:

• Mapping f is isometric to a subset of Euclidean space (i.e., zero intrinsic curvature).

– Statistical: • Latent data {yi} are a “representative” sample* from

a convex domain.

* Minimum distance from any point on the manifold to a sample point < e.g., variable density Poisson process).

Theoretical results on the rate of convergence

• Upper bound on the number of data points required.

• Rate of convergence depends on several geometric parameters of the manifold: – Intrinsic:

• dimensionality

– Embedding-dependent: • minimal radius of curvature

• minimal branch separation

Face under varying pose and illumination

• Dimensionality

• pictureMDS / PCA

Isomap

Hand under nonrigid articulation

• Dimensionality

• pictureMDS / PCA

Isomap

Apparent motion

Digits

• Dimensionality

• picture. MDS / PCA

Isomap

Summary of Isomap

A framework for global nonlinear dimensionality reduction that preserves the crucial features of PCA and classical MDS:

• A noniterative, polynomial-time algorithm.• Guaranteed to construct a globally optimal Euclidean

embedding. • Guaranteed to converge asymptotically for an important class

of nonlinear manifolds.

Plus, good results on real and nontrivial synthetic data sets.

Outline

• The problem of nonlinear dimensionality reduction

• The Isomap algorithm

• Development #1: Curved manifolds

• Development #2: Sparse approximations

Locally Linear Embedding (LLE)

• Roweis and Saul (2000)

Comparing LLE and Isomap

• Both start with only local metric information.• Isomap first estimates global metric structure, then

finds an embedding that optimally preserves global structure.

• LLE finds an embedding that optimally preserves only local structure.

• LLE may be more efficient, but may also introduce unpredictable global distortions.

• No asymptotic convergence results for LLE.

LLE Isomap

Outline

• The problem of nonlinear dimensionality reduction

• The Isomap algorithm

• Development #1: Curved manifolds

• Development #2: Sparse approximations

The assumptions behind three algorithms

Distribution: p(Y) Mapping: f Algorithm

ii) convex, dense isometric Isomap

iii) convex, uniformly dense conformal C-Isomap

iii)

i) arbitrary linear isometric Classical MDS

Isometric vs. conformal mapping

• Isometric map: preserves the Euclidean metric at each point y.

• Conformal map: preserves the Euclidean metric at each point y, up to an arbitrary scale factor (y) > 0.

• Properties of conformal maps: – Angle-preserving.– Any subset topologically equivalent to a disk can be

conformally mapped onto a disk.

)()()( iYX yiMiM

C-Isomap

• Invariant: ,

,

f

ijjiX xxiM

||)(ijjiY yyiM

||)(

independent of i

Y

X

The Isomap algorithm• Construct neighborhood graph G.

– method– K method

• Compute shortest paths in G, with edge ij weighted by the Euclidean distance |xi - xj|.

– Floyd – Dijkstra (+ Fibonacci heaps)

• Reconstruct low-dimensional latent data {yi}.

– Classical MDS on graph distances– Sparse MDS with landmarks

The C-Isomap algorithm• Construct neighborhood graph G.

– method– K method

• Compute shortest paths in G, with edge ij weighted by rescaled distance – Floyd – Dijkstra (+ Fibonacci heaps)

• Reconstruct low-dimensional latent data {yi}.

– Classical MDS on graph distances– Sparse MDS with landmarks

)()(|| jMiMxx XXji

Conformal fishbowl

Data MDS Isomap

C-Isomap LLE GTM

Uniform fishbowl

Data MDS Isomap

C-Isomap LLE GTM

Conformal fishbowl, Gaussian density

Latent data C-Isomap LLE

Conformal fishbowl, offset Gaussian density

Latent data C-Isomap LLE

Wavelet

Data MDS Isomap

C-Isomap LLE GTM

Images of Tom’s face

• Two intrinsic degrees of freedom:– Translation: left/right– Zoom: in/out

• Scale variables (e.g., zoom) introduce conformal distortion.

. . .

Face under translation and zoom

Data MDS Isomap

C-Isomap LLE GTM

Curvature in LLE vs. Isomap

• LLE: +/- Approach: look only at local structure, ignoring global structure.

- Asymptotics: unknown.

+ Nonconformal maps: good for some, but not all.

• Isomap: +/- Approach: explicitly estimate, and factor out, local metric distortion (assuming uniform density).

+ Asymptotics: succeeds for all conformal mappings.

+ Nonconformal maps: good for some, but not all.