dimensionality reduction part 2: nonlinear methods

CS685 : Special Topics in Data Mining, UKY

The UNIVERSITY of KENTUCKY

Dimensionality ReductionPart 2: Nonlinear Methods

Comp 790-090Spring 2007


Why Dimensionality Reduction

• Two approaches to reduce number of features– Feature selection: select the salient features by

some criteria– Feature extraction: obtain a reduced set of

features by a transformation of all features

• Data visualization and exploratory data analysis also need to reduce dimension– Usually reduce to 2D or 3D


Deficiencies of Linear Methods

• Data may not be best summarized by linear combination of features– Example: PCA cannot discover 1D structure of a

helix

-1-0.5

00.5

1

-1

-0.5

0

0.5

10

5

10

15

20


Intuition: how does your brain store these pictures?


Brain Representation


Brain Representation

• Every pixel?• Or perceptually meaningful

structure?– Up-down pose– Left-right pose– Lighting directionSo, your brain successfully

reduced the high-dimensional inputs to an intrinsically 3-dimensional manifold!


Manifold Learning

• Discover low dimensional representations (smooth manifold) for data in high dimension.

• Linear approaches(PCA, MDS)

• Non-linear approaches (ISOMAP, LLE, others)

d

i Ry Y

XN

i Rx

latent

observed


Linear Approach- PCA

• PCA Finds subspace linear projections of input data.


Linear Method

• Linear Methods for Dimensionality Reduction– PCA: rotate data so that principal axes lie in direction

of maximum variance– MDS: find coordinates that best preserve pairwise

distances

PCA


Motivation

• Linear Dimensionality Reduction doesn’t always work

• Data violates underlying “linear”assumptions– Data is not accurately modeled by “affine”

combinations of measurements– Structure of data, while apparent, is not

simple– In the end, linear methods do nothing

more than “globally transform” (rate, translate, and scale) all of the data, sometime what’s needed is to “unwrap” the data first


What does PCA Really Model?

• Principle Component Analysis assumptions• Mean-centered distribution

– What if the mean, itself is atypical?• Eigenvectors of

Covariance– Basis vectors aligned

with successive directionsof greatest variance

• Classic 1st Orderstatistical model– Distribution is characterized

by its mean and variance (Gaussian Hyperspheres)


Nonlinear Approaches- Isomap

• Constructing neighbourhood graph G• For each pair of points in G, Computing shortest path distances ----

geodesic distances.• Use Classical MDS with geodesic distances. Euclidean distance Geodesic distance

Josh. Tenenbaum, Vin de Silva, John langford 2000


Sample points with Swiss Roll

• Altogether there are 20,000 points in the “Swiss roll” data set. We sample 1000 out of 20,000.


Construct neighborhood graph GK- nearest neighborhood (K=7)DG is 1000 by 1000 (Euclidean) distance matrix of two neighbors (figure A)


Compute all-points shortest path in G

Now DG is 1000 by 1000 geodesic distance matrix of two arbitrary points along the manifold (figure B)


Find a d-dimensional Euclidean space Y (Figure c) to preserve the pariwise diatances.

Use MDS to embed graph in Rd


Isomap

• Key Observation:

On a manifold distances are measured using geodesic distances rather than Euclidean distances

Small Euclidean distance

Large geodesic distance


Problem: How to Get Geodesics

• Without knowledge of the manifold it is difficult to compute the geodesic distance between points

• It is even difficult if you know the manifold• Solution

– Use a discrete geodesic approximation– Apply a graph algorithm to approximate the

geodesic distances


Dijkstra’s Algorithm• Efficient Solution to all-points-shortest path

problem• Greedy breath-first algorithm


Dijkstra’s Algorithm

• Efficient Solution to all-points-shortest path problem

• Greedy breath-first algorithm


Isomap algorithm– Compute fully-connected

neighborhood of points for each item

• Can be k nearest neighbors or ε-ball

• Neighborhoods must be symmetric

• Test that resulting graph is fully-connected, if not increase either K or

– Calculate pairwise Euclidean distances within each neighborhood

– Use Dijkstra’s Algorithm to compute shortest path from each point to non-neighboring points

– Run MDS on resulting distance matrix


Isomap Results• Find a 2D embedding of the 3D S-curve

(also shown for LLE)• Isomap does a good job of preserving metric structure

(not surprising)• The affine structure is also well preserved


Residual Fitting Error


Neighborhood Graph


More Isomap Results


Isomap Failures• Isomap also has problems on closed manifolds of

arbitrary topology


Local Linear Embeddings

• First Insight– Locally, at a fine enough scale, everything looks

linear


Local Linear Embeddings

• First Insight– Find an affine combination the “neighborhood”

about a point that best approximates it


Finding a Good Neighborhood

• This is the remaining “Art” aspect of nonlinear methods

• Common choices• -ball: find all items that lie within an epsilon ball of

the target item as measured under some metric– Best if density of items is high and every point has a

sufficient number of neighbors• K-nearest neighbors: find the k-closest neighbors to

a point under some metric– Guarantees all items are similarly represented, limits

dimension to K-1


Characterictics of a Manifold

Locally it is a linear patch

Key: how to combine all localpatches together?

M

x1

x2R2

Rn


LLE: Intuition• Assumption: manifold is approximately “linear”

when viewed locally, that is, in a small neighborhood– Approximation error, e(W), can be made small

• Local neighborhood is effected by the constraint Wij=0 if zi is not a neighbor of zj

• A good projection should preserve this local geometric property as much as possible


LLE: IntuitionWe expect each data point and its neighbors to lie on or close to a locally linear patch of themanifold.

Each point can be written as a linear combination of its neighbors.The weights chosen tominimize the reconstructionError.


LLE: Intuition

• The weights that minimize the reconstruction errors are invariant to rotation, rescaling and translation of the data points.

– Invariance to translation is enforced by adding the constraint that the weights sum to one.

– The weights characterize the intrinsic geometric properties of each neighborhood.

• The same weights that reconstruct the data points in D dimensions should reconstruct it in the manifold in d dimensions.– Local geometry is preserved


LLE: Intuition

Use the same weights from the original space

Low-dimensional embedding

the i-th row of W


Local Linear Embedding (LLE)• Assumption: manifold is approximately “linear” when viewed locally, that is, in a

small neighborhood

• Approximation error, (W), can be made small

• Meaning of W: a linear representation of every data point by its neighbors– This is an intrinsic geometrical property of the manifold

• A good projection should preserve this geometric property as much as possible


Constrained Least Square problemCompute the optimal weight for each point individually:

Neightbors of x

Zero for all non-neighbors of x


Finding a Map to a Lower Dimensional Space

• Yi in Rk: projected vector for Xi

• The geometrical property is best preserved if the error below is small

• Y is given by the eigenvectors of the lowest d non-zero eigenvalues of the matrix

Use the same weightscomputed above


Numerical Issues

• Numerical problems can arise in computing LLEs• The least-squared covariance matrix that arises

in the computation of the weighting matrix, W, solution can be ill-conditioned– Regularization (rescale the measurements by adding

a small multiple of the Identity to covariance matrix)• Finding small singular (eigen) values is not as

well conditioned as finding large ones. The small ones are subject to numerical precision errors, and to get mixed– Good (but slow) solvers exist, you have to use them


Results

• The resulting parameter vector, yi, gives the coordinates associated with the item xi

• The dth embedding coordinate is formed from the orthogonal vector associated with thedst singular value of A.


Reprojection

• Often, for data analysis, a parameterization is enough• For interpolation and compression we might want to

map points from the parameter space back to the “original” space

• No perfect solution, but a few approximations– Delauney triangulate the points in the embedding space, find

the triangle that the desired parameter setting falls into, and compute the baricenric coordinates of it, and use them as weights

– Interpolate by using a radially symmetric kernel centered about the desired parameter setting

– Works, but mappings might not be one-to-one


LLE Example• 3-D S-Curve manifold with points color-coded• Compute a 2-D embedding

– The local affine structure is well maintained– The metric structure is okay locally, but can drift slowly over the

domain (this causes the manifold to taper)


More LLE Examples


LLE Failures• Does not work on to closed manifolds• Cannot recognize Topology


Summary• Non-Linear Dimensionality Reduction Methods

– These methods are considerably more powerful and temperamental than linear method

– Applications of these methods are a hot area of research

• Comparisons– LLE is generally faster, but more brittle than Isomaps– Isomaps tends to work better on smaller data sets

(i.e. less dense sampling)– Isomaps tends to be less sensitive to noise

(perturbation of the input vectors)

• Issues– Neither method handles closed manifolds and topological variations well

dimensionality reduction part 2: nonlinear methods

Documents