an applied introduction - aminer...spectral graph theory an applied introduction in lectures on...
TRANSCRIPT
Spectral graph theory
An applied introduction In Lectures on Structural Pattern Recognition @ PRIP
Ernst Schwartz 05/07/2008
Overview Theory 1: Adjacency, Laplacian,
Eigendecomposition Applications 1: Embeddings, object retrieval
Theory 2: Normalized Laplacian Applications 2: Segmentation, Manifold learning
Theory 3: Random Walks Applications 3: Manifold learning, PageRank
Conclusions
Theory 1: Adjacency, Laplacian, Graph spectrum
Many ways to represent a graph Many ways to describe a graph (number of
edges, degree, regularity, ...) How to gather the most discriminative
information?
Basic notions about graphs Pairwise affinity
some distance measure for weighting the edges of the graph based on an embedding of the nodes.
For example Node volume
Volume of a cluster
Cut between 2 clusters
Adjacency matrix
G is a simple graph, meaning there are no self edges (E) and at most one edge between two nodes (V)
The corresponding (weighted) adjacency matrix is defined as
Laplacian matrix
with D as the diagonal matrix with D(i,i) = di:
Where di is equal to the degree of the node (eg. the number of edges at that node)
if all weights are 1 (eg. the graph is unweighted) or equal to the volume of the node as seen before
Laplacian matrix
Describes gradient on the graph
Strong connection to differential geometry: discrete version of the Laplace-Beltrami operator on manifolds
A (very) simple example
# Adjacency
Laplacian
1 2 3 4 5 6 7 8 9 10 11 12 1 0 1 0 0 0 0 0 0 0 0 0 0 2 1 0 1 0 0 0 0 0 0 0 0 0 3 0 1 0 1 0 0 0 0 0 0 0 0 4 0 0 1 0 1 0 0 0 0 0 0 0 5 0 0 0 1 0 1 0 0 0 0 0 0 6 0 0 0 0 1 0 1 0 0 0 0 0 7 0 0 0 0 0 1 0 1 0 0 0 0 8 0 0 0 0 0 0 1 0 1 0 0 0 9 0 0 0 0 0 0 0 1 0 1 0 0 10 0 0 0 0 0 0 0 0 1 0 1 0 11 0 0 0 0 0 0 0 0 0 1 0 1 12 0 0 0 0 0 0 0 0 0 0 1 0
1 2 3 4 5 6 7 8 9 10 11 12 1 1 -1 0 0 0 0 0 0 0 0 0 0 2 -1 2 -1 0 0 0 0 0 0 0 0 0 3 0 -1 2 -1 0 0 0 0 0 0 0 0 4 0 0 -1 2 -1 0 0 0 0 0 0 0 5 0 0 0 -1 2 -1 0 0 0 0 0 0 6 0 0 0 0 -1 2 -1 0 0 0 0 0 7 0 0 0 0 0 -1 2 -1 0 0 0 0 8 0 0 0 0 0 0 -1 2 -1 0 0 0 9 0 0 0 0 0 0 0 -1 2 -1 0 0 10 0 0 0 0 0 0 0 0 -1 2 -1 0 11 0 0 0 0 0 0 0 0 0 -1 2 -1 12 0 0 0 0 0 0 0 0 0 0 -1 1
(vertices are numbered from 1 to 12)
Eigendecomposition
Computing eigenvalues and eigenvectors of a matrix.
Corresponds to factorizing the matrix into a canonical form.
scalars (eigenvalues)
(eigen)vectors
The following is imposed on the eigenvectors:
A (very) simple example
First four and the last
(eg. ordered by their corresponding eigenvalues)
eigenvector (values of every dimension)
(no vertex information...)
Recap 1
Laplacian strongly connected to gradient
Continuous analogue: Laplace-Beltrami (heat) operator
Application: spectral embedding
The eigenvectors of the laplacian can be used to compute a (allmost) planar embedding of a mesh.
No information on the vertices is needed, the embedding is based solely on the connectivity of the mesh.
Original mesh
gplot(A, V)
Spectral embedding
L = diag(sum(A)) - A; [v d] = eigs(L); gpolt(A, v(:,[1 2]));
Application: Object retrieval
(Deduce mesh from binary image: shock graphs)
Use spectral information for similarity measures between datasets
Usefull for image retrieval, classification, ...
Object retrieval
© Jain & Zhang
Application: Brain surface flattening
© Angenent & Haker
Theory 2: Normalized Laplacian
Laplacian is more informative than the adjacency matrix
Sometimes cumbersome calculations
Solution: normalized laplacian !
Usefull mathematical properties ...
Definition
Properties of the normalized laplacian
Cheegers inequality
Conductance of a vertex set:
Volume of a set of edges joining two subsets:
Conductance of a graph:
Recap 2
is a very expressive value
Approximation using Cheeger‘s inequality
Eigenvalues contain global information about the mesh
Application: Image segmentation
Generate a graph representation from an image (see last week‘s lecture)
Find „best“ cuts in that graph
Spectral interpretation of a „good“ cut: normalized cuts.
Normalized cuts uses the quantity
called the „Normalized Cut“
Normalized Cuts
Define the volume of a set of vertices S as
Normalized Cuts
Compute eigendecomposition of Laplacian
Set V = [v1, v2, v3 ... , vk]
View each row of V as a point in Rk ; cluster into k clusters using k-means ( )
Normalized Cuts
Normalized Cuts
Normalized Cuts
Normalized Cuts
Application: Manifold learning
New framework for dimensionality reduction Based on the assumption of an underlying
manifold Model this manifold implicitly using graph
techniques Use spectral methods to measure the quality
of the embedding
Manifold learning
?
Manifold learning
?
Manifold learning
?
(semi-supervised learning)
Manifold learning
What is needed is a way to compute the (geodesic) distance on the data
For this, we need to be able to describe the structure of the manifold
Use eigendecomposition of the laplacian !
Represents geodesic distance !
Manifold learning
How to measure distances ? „swiss roll“ dataset With vertices in 3 dimensions
Manifold learning
Build graph based on k-nns !
Manifold learning
Use spectral embedding using first eigenvectors
First eigenvector
Sec
ond
eige
nvec
tor
Theory 3: Random Walks on graphs
Use graph structure as a model for a statistical process
Vertices are states in the system Edges are transition probabilities for going
from one state to another Only a small change to the Laplacian
formulation Good statistical tools available
Definitions
Walk matrix
Taking one step in the walk
Normalized walk matrix
Interesting values of a random walk
Access/hitting time: expected number of steps before node j is visited
Commute time expected number of steps in a random walk starting at i, going to j and then back to i
Cover time expected number of steps to reach every node
Mixing rate how fast the random walk converges to it‘s limiting distribution
Access time:
Commute time:
Mixing rate:
Spectral computation of graph properties
Recap
Random walks are a tool for statistical modelling
Properties can easily be deduced from eigendecomposition
Application: Manifold Learning 2
Concept: Diffusion on a graph Propagate a state according to the
connectivity Discrete time steps equivalent to multiple
resolution analysis More resistant to noise Usefull for dimensionality reduction,
segmentation, classification, ...
Diffusion maps
© Coifman & Lafon
Idea: map graph distances to euclidean distances
Use random walks for noise robustness
Very general framework; can be interpreted as
Stochastic differential equations, Graph theoretical methods Physical processes (heat kernel) ...
Connection to kernel methods (!)
Diffusion maps Based on random walk Definition of a diffusion distance Describes distance between points in terms of their
connectivity Use eigendecomposition to find embedding in euclidean
space (= diffusion maps) Probability to be in state x at time k can be approximated
using eigendecomposition More robust to noise due because more robust to
outliers Extensions: diffusion wavelets using different kernel
sizes
PageRank
© Page & Brim
Recap: Operators on graphs
Operator Spectrum
Adjacency: A
Laplacian: D - A
Normalized Laplacian: I - D-1/2AD-1/2 Random Walk: D-1A
Note: why consider the adjacency as an operator? A1 => nodes reachable from another in one step An => nodes reachable from another in n steps => can be interpreted as an operator on the graph
Conclusions Tools for graph processing: laplacian & it‘s
eigendecomposition, random walks, diffusions
All data is structured (even if we don‘t know how)! Use graphs to represent the underlying structure Use spectral analysis to find modes of the data.
Note: classical uses in network analysis... not covered here
Bibliography Videolectures.net (!!)
http://videolectures.net/mlcued08_azran_mcl/ Spectral clustering (50min)
http://videolectures.net/sicgt07_belkin_gmagod/ Graph methods and geometry of data (55min)
http://videolectures.net/gbr07_hancock_gsis/ Spectral image smoothing (27min)
http://videolectures.net/mlss05us_nadler_dmscr/ Diffusion maps (55min)
http://videolectures.net/aop05_hancock_pagt/ Pattern analysis with graphs and trees (120min)
Bibliography Fan R. Chung; Spectral Graph Theory
(chapters 1-4 online at http://www.math.ucsd.edu/~fan/research/revised.html )
Daniel Spielman; Spectral Graph Theory and its Applications (online with Matlab code at http://www.cs.yale.edu/homes/spielman/sgta/)
L. Lovász; Random Walks on Graphs; A Survey Mauro Maggioni; diffusion maps (matlab code)
http://www.math.duke.edu/~mauro/
Bibliography Angenent & Haker; On the Laplacian-Beltrami
Operator for Brain Surface flattening Shi & Malik; Normalized Cuts and Image
Segmentation Jain & Zhang; A Spectral Approach to Shape-Based
Retrieval of Articulated 3D Models Cayton; Algorithms for Manifold Learning Coifmann & Lafon; Diffusion Maps Page & Brim; The PageRank Citation Ranking:
Bringing Order to the Web