approximate nearest subspace search with applications to pattern recognition ronen basri, tal...
Post on 20-Dec-2015
220 views
TRANSCRIPT
Approximate Nearest Subspace Search with Applications to Pattern Recognition
Ronen Basri, Tal Hassner, Lihi Zelnik-Manor
presented by Andrew Guillory and Ian Simon
The Problem
• Given n linear subspaces Si:
• And a query point q:
• Find the subspace Si that minimizes dist(Si,q).
0T xZ i
Approach
• Solve by reduction to nearest neighbor.– point-to-point distances
• In higher-dimensional space.not actual reduction
Point-Subspace Distance
• Use squared distance.
2
2
2
2
22
1
12
11
21
22212
11211
dd
d
d
dddd
d
d
a
a
aa
a
a
aaa
aaa
aaa
h
TT
TT
TT
TT
2T2
2
,dist
ZZhxxh
ZZxx
xZZx
xZxZ
xZSx
Point-Subspace Distance
• Use squared distance.
• Squared point-subspace distancecan be represented as a dot product.
2
2
2
2
22
1
12
11
21
22212
11211
dd
d
d
dddd
d
d
a
a
aa
a
a
aaa
aaa
aaa
h
TT
TT
TT
TT
2T2
2
,dist
ZZhxxh
ZZxx
xZZx
xZxZ
xZSx
The Reduction
• Let:
• Then:
Remember:
222
222
,dist
2,dist
vuSx
vvuuvu
TT2 2,dist ZZhxxhSx
TT
xxhv
ZZhu
The Reduction
kd
ZZ
ZZ
ZZZZ
ZZhZZh
ZZhu
2
1
Tr2
1
Tr2
1
Tr2
1
T
T
TT
TT
2T2
T
T
xxhv
ZZhu
ZTZ = I
2222 ,dist,dist vuSxvu ? constant over query
The Reduction
kd
ZZ
ZZ
ZZZZ
ZZhZZh
ZZhu
2
1
Tr2
1
Tr2
1
Tr2
1
T
T
TT
TT
2T2
T
T
xxhv
ZZhu
ZTZ = I
Z is d-by-(d-k), columns orthonormal.
2222 ,dist,dist vuSxvu ? constant over query
The Reduction
kd
ZZ
ZZ
ZZZZ
ZZhZZh
ZZhu
2
1
Tr2
1
Tr2
1
Tr2
1
T
T
TT
TT
2T2
T
T
xxhv
ZZhu
ZTZ = I
Z is d-by-(d-k), columns orthonormal.
2222 ,dist,dist vuSxvu ? constant over query
The Reduction
• For query point q:
• Can we decrease the additive constant?
422
2
1,dist,dist qkdSqvu
Observation 1
• All data points lie on a hyperplane.
• Let:
• Now the hyperplane contains the origin.
kdZZ TTr
Id
kdZZhu T
Id
qqqhv
2
T
Observation 2
• After hyperplane projection:
• All data points lie on a hypersphere.
1
2
T2
d
kdk
Id
kdZZhu
Observation 2
• After hyperplane projection:
• All data points lie on a hypersphere.
• Let:
• Now the query point lies on the hypersphere.
1
2
T2
d
kdk
Id
kdZZhu
Id
qqqh
d
kdk
qv
2
T2 1
1
Observation 2
• After hyperplane projection:
• All data points lie on a hypersphere.
• Let:
• Now the query point lies on the hypersphere.
1
2
T2
d
kdk
Id
kdZZhu
Id
qqqh
d
kdk
qv
2
T2 1
1
Finally
• Additive constant depends only on dimension of points and subspaces.
• This applies to linear subspaces, all of the same dimension.
11
1
1
,dist,dist
2
22
d
kdkk
d
k
d
kdk
q
Sqvu
Extensions
• subspaces of different dimension– lines and planes, e.g.– Not all data points have the same norm.• Add extra dimension to fix this.
Extensions
• subspaces of different dimension– lines and planes, e.g.– Not all data points have the same norm.• Add extra dimension to fix this.
• affine subspaces
– Again, not all data pointshave the same norm.
bxZ i T
Approximate Nearest Neighbor Search
• Find point x with distance d(x, q) <= (1 + ε) mini d(xi,q)
• Tree based approaches: KD-trees, metric / ball trees, cover trees
• Locality sensitive hashing• This paper uses multiple KD-Trees with
(different) random projections
Random Projections
• Multiply data with a random matrix X with X(i,j) drawn from N(0,1)
• Several different justifications– Johnson-Lindenstrauss (data set that is small
compared to dimensionality)– Compressed Sensing (data set that is sparse in
some linear basis)– RP-Trees (data set that has small doubling
dimension)
Results
• Two goals– show their method is fast – show nearest subspace is useful
• Four experiments– Synthetic Experiments– Image Approximation– Yale Faces– Yale Patches
Questions / Issues
• Should random projections be applied before or after the reduction?
• Why does the effective distance error go down with the ambient dimensionality?
• The reduction tends to make query points far away from the points in the database. Are there better approximate nearest neighbor algorithms in this case?