“dimension versus distortion” and “intrinsic dimension of ...anupamg/adfocs/gupta-lec3.pdf ·...
TRANSCRIPT
![Page 1: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/1.jpg)
“Dimension versus Distortion”
and
“Intrinsic Dimension of
Metric Spaces”Metric Spaces”
Anupam Gupta
Carnegie Mellon University
![Page 2: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/2.jpg)
Metric space M = (V, d)
set V of points
y
z
distances d(x,y)
triangle inequality
d(x,y) ≤ d(x,z) + d(z,y)x
![Page 3: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/3.jpg)
in the previous two lectures…
We saw:
every metric is (almost) a (random) tree
every metric is (almost) an Euclidean metric
Today:
low-dimensional embeddings
low-dimensional metric spaces
![Page 4: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/4.jpg)
why dimension reduction?
2-dimensional Euclidean space is simple
3-dim space is not so bad
4-dim ??
.
harder for algorithms as well!
.
.
100000-dim space may be a bit much
![Page 5: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/5.jpg)
the “curse of dimensionality”
many (geometric) algorithms have running times that are
exponential in the dimension.
so can we reduce the dimensions without changing the so can we reduce the dimensions without changing the
distances?
how about only changing the distances by a small amount?
![Page 6: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/6.jpg)
hasn’t this been solved?
There are many approaches to doing dimension reduction.
Many based on singular-value decompositions
e.g., PCA and many variants, ICA, factor analysis
also, multidimensional scaling…also, multidimensional scaling…
However, they try to minimize some other notion of goodness
does not work for distortion
could stretch/shrink distances by large factors.
![Page 7: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/7.jpg)
central question
How do we reduce the dimension
so that every distance is distorted byso that every distance is distorted by
only a small factor?
natural tension between dimension and distortion
![Page 8: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/8.jpg)
exercise #2, problem 5
![Page 9: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/9.jpg)
the Johnson Lindenstrauss lemma
Theorem [1984]
Given any n-point subset V of Euclidean space Rm, one can
map into O(log n/ǫ2)-dimensional space while incurring a
distortion at most (1+ǫ).distortion at most (1+ǫ).
There exists such a map that is a linear map.
![Page 10: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/10.jpg)
What is this map?
Let k = O(log n/ǫ2)
• Choose a uniformly random k-dimensional subspace
and project the points down onto it.
The distance between the projections is expected to be
about sqrtk/n of their original distances.
One can prove that the distances are within (1+ǫ) of
this expected value with high probability.
[JL, FM, DG]
![Page 11: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/11.jpg)
Easier to deal with
Let k = O(log n/ǫ2)
• Take a k ×m matrix A
each entry is an independent N(0,1) random value.
• the linear map is F(x) = Ax• the linear map is F(x) = Ax
• For any vector x and row a of A, a.x∼∼∼∼ N(0,|x|2)
• For any vector x, E[|Ax|2] = k|x|2
• Pr[ |Ax|2 in (1 ± ǫ) k|x|2] ≥
![Page 12: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/12.jpg)
![Page 13: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/13.jpg)
![Page 14: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/14.jpg)
![Page 15: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/15.jpg)
Easier to deal with
Let k = O(log n/ǫ2)
• Take a k ×m matrix A
each entry is an independent N(0,1) random value.
• the linear map is F(x) = Ax• the linear map is F(x) = Ax
• For any vector x and row a of A, a.x∼∼∼∼ N(0,|x|2)
• For any vector x, E[|Ax|2] = k|x|2
• Pr[ |Ax|2 in (1 ± ǫ) k|x|2] ≥
• So can take union bound over all n2 difference vectors.
[IM]
1 – exp-Ω(ǫ2k)
![Page 16: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/16.jpg)
proof
![Page 17: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/17.jpg)
embedding ℓ2 into ℓ1
![Page 18: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/18.jpg)
Even easier to do
Database-Friendly Projections [A ’01]
Again, let A be a k ×m matrix
now fill it with uniformly random ±1 values.
(lower density: 0 w.p. 2/3, uniformly random ±1 otherwise.)(lower density: 0 w.p. 2/3, uniformly random ±1 otherwise.)
Again F(x) = Ax
Proof along similar general lines, but more involved.
Question: can we reduce the support of A any further?
![Page 19: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/19.jpg)
faster maps
The time to compute the map was O(mk) = O(m log n).
“Fast JL Transform” [AC]
Time O(m log m + ǫ-2 log3 n)
Basic idea: uncertainty principle.
density of JL/DFP matrices can be reduced if x is “spread out”
support of x, Hx cannot both be small (H = Hadamard matrix)
so map F(x) = AHx
(actually AHDx, D is random diagonal sign matrix).
![Page 20: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/20.jpg)
and faster still
The time to compute the map was O(mk) = O(m log n).
“Fast JL Transform” [AC]
Time O(m log m + k3)
[AL] O(m log k)
[LAS] O(m) --- for “well-spread-out” x.
![Page 21: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/21.jpg)
dimension-distortion tradeoffs
What if we have a dimension k in mind?
exercise showed:
Ω(n1/k) distortion
best results:best results:
O(n2/d log3/2 n) distortion
Ω(n1/ (k+1)/2 )
very interesting new results:
NP-hard to nc/d-approximate distortion for all constant d.
![Page 22: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/22.jpg)
tighter lower bound
Theorem [1984]
Given any n-point subset V of Euclidean space, one can
map into O(log n/ǫ2)-dimensional space while incurring a
distortion at most (1+ǫ).
We’ve already seen a lower bound of Ω(log n/ǫ).
Theorem (Alon):
Lower bound of Ω(log n/(ǫ2 log ǫ -1))
again for the “uniform” metric Un
![Page 23: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/23.jpg)
so what do we do now?
Note that we’re again proving uniform results.
(“All n-point Euclidean metrics…”)
Can we give a better “per-instance” guarantee?
If the metric contains a copy of Ut
we get a lower bound of Ω(log t) dimensions if we want O(1) distortion
But this is somewhat fragile (what if there’s an almost uniform metric?)…
![Page 24: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/24.jpg)
the doubling dimension
convenient way of capturing the property that
there are no large near-uniform metrics
gives us a notion of “intrinsic dimension” of a point setgives us a notion of “intrinsic dimension” of a point set
in Euclidean space
in fact, doubling dimension is defined for any metric space,
not just for point sets in Euclidean space.
![Page 25: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/25.jpg)
Dimension dimD(M) is the smallest k such that
every set S with diameter DS
can be covered by 2k sets of diameter ½DS
the doubling dimension
D
![Page 26: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/26.jpg)
Dimension dimD(M) is the smallest k such that
every set S with diameter DS
can be covered by 2k sets of diameter ½DS
the doubling dimension
D
![Page 27: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/27.jpg)
doubling generalizes geometric dimension
Take k-dim Euclidean space Rk
Claim: dimD(Rk) ≈ Θ(k)Claim: dimD(R ) ≈ Θ(k)
Easy to see for boxes
Argument for spheres a bit more involved.23 boxes to cover
larger box in R3
![Page 28: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/28.jpg)
Dimension at most k if
every set S with diameter DS can be covered by 2k sets of
diameter ½DS
doubling metrics
A family of metric spaces is called “doubling”
if there exists a constant k such that the doubling dimension
of these metrics is bounded by k.
![Page 29: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/29.jpg)
what is not a doubling metric?
The uniform metric Ut on t points have dimension Ω(log t)
![Page 30: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/30.jpg)
δδδδ
∆∆∆∆ this 2-dim set
has (∆/d)2 points
small near-uniform metrics
δδδδ
Suppose a metric (X,d) has doubling dimension k.
If any subset S ⊆ X of points has
all inter-point distances lying between δ and ∆
there are at most (∆/δ)O(k) points in S.
![Page 31: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/31.jpg)
the (simple) proof
![Page 32: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/32.jpg)
advantages of this fact
Thm: Doubling metrics admit O(dim(M))-padded decompositions
⇒doubling metrics embed into ℓ2 with distortion sqrtlog n.
Q: Do all doubling metrics embed into ℓ2 with distortion O(1)?
![Page 33: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/33.jpg)
Many geometric algorithms can be extended to doubling spaces…
Near neighbor search
Compact routing
Distance labeling
Network triangulation
Sensor placements
Small-world networks
Traveling Salesman
Sparse Spanners
Approx. inference
Network Design
Clustering problems
Well-separated pair
decomposition
Data structures
Learnability
![Page 34: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/34.jpg)
example application
Assign labels L(x) to each host x in a metric space
Looking just at L(x) and L(y), can infer distance d(x,y)
Results
labels with (O(1)/ε)dim × log n bitsy010001010001labels with (O(1)/ε)dim × log n bits
estimates within (1 + ε) factor
Contrast with
lower bound of n bit labels
in general for any factor < 2x
y010001
110001
f( , )
110001
010001
≈ d(x,y)
![Page 35: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/35.jpg)
For example:
[Arora 95] showed that TSP on Rk was (1+ǫ)-approximable in time
another example application
[Talwar 04] extended this result to metrics with doubling dimension k
![Page 36: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/36.jpg)
example in action:
sparse spanners for doubling metrics[Chan G. Maggs Zhou]
![Page 37: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/37.jpg)
r-nets
![Page 38: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/38.jpg)
the construction
![Page 39: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/39.jpg)
the sparsity
![Page 40: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/40.jpg)
the stretch
![Page 41: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/41.jpg)
Back to dimensionality reduction
![Page 42: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/42.jpg)
useful for distortion-dimension
Theorem: [Chan G. Talwar]
Any metric with doubling dimension k embeds into
Euclidean space with T dimensions with distortion
(where T ∈ [ k log log n, log n])
Can get very similar tradeoffs using results of [ABN’08]
![Page 43: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/43.jpg)
For the case of
![Page 44: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/44.jpg)
other notions of metric dimension
strong doubling dimension
correlation fractal dimensioncorrelation fractal dimension
negative-curvature
![Page 45: “Dimension versus Distortion” and “Intrinsic Dimension of ...anupamg/adfocs/Gupta-lec3.pdf · There exists such a map that is a linear map. What is this map? Let k = O(log n](https://reader037.vdocuments.net/reader037/viewer/2022090608/605dfdbb117ab76e4256af75/html5/thumbnails/45.jpg)
thank you!