segmentation and low-level grouping. · 2005-04-14 · the generic, unavoidable problem with...
TRANSCRIPT
![Page 1: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/1.jpg)
Segmentation and low-level grouping.
Bill Freeman, MIT
6.869 April 14, 2005
![Page 2: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/2.jpg)
Readings: Mean shift paper and background segmentation paper.
• Mean shift IEEE PAMI paper by Comanici and Meer, http://www.caip.rutgers.edu/~comanici/Papers/MsRobustApproach.pdf
• Forsyth&Ponce, Ch. 14, 15.1, 15.2.• Wallflower: Principles and Practice of
Background Maintenance, by Kentaro Toyama, John Krumm, Barry Brumitt, Brian Meyers. http://research.microsoft.com/users/jckrumm/Publications%202000/Wall%20Flower.pdf
![Page 3: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/3.jpg)
The generic, unavoidable problem with low-level segmentation and grouping
• It makes a hard decision too soon. We want to think that simple low-level processing can identify high-level object boundaries, but any implementation reveals special cases where the low-level information is ambiguous.
• So we should learn the low-level grouping algorithms, but maintain ambiguity and pass along a selection of candidate groupings to higher processing levels.
![Page 4: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/4.jpg)
Segmentation methods
• Segment foreground from background• K-means clustering• Mean-shift segmentation• Normalized cuts
![Page 5: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/5.jpg)
A simple segmentation technique: Background Subtraction
• If we know what the background looks like, it is easy to identify “interesting bits”
• Applications– Person in an office– Tracking cars on a road– surveillance
• Approach:– use a moving average
to estimate background image
– subtract from current frame
– large absolute values are interesting pixels
• trick: use morphological operations to clean up pixels
![Page 6: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/6.jpg)
Movie frames from which we want to extract the foreground subject(the textbook author’s child)
![Page 7: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/7.jpg)
2 different background removal modelsBackground estimate Foreground estimate Foreground estimate
low thresh high thresh
EM
Average over frames
EM background estimate
![Page 8: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/8.jpg)
Static Background Modeling Examples
[MIT Media Lab Pfinder / ALIVE System]
![Page 9: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/9.jpg)
Static Background Modeling Examples
[MIT Media Lab Pfinder / ALIVE System]
![Page 10: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/10.jpg)
Static Background Modeling Examples
[MIT Media Lab Pfinder / ALIVE System]
![Page 11: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/11.jpg)
Dynamic Background
BG Pixel distribution is non-stationary:
[MIT AI Lab VSAM]
![Page 12: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/12.jpg)
Mixture of Gaussian BG model
Staufer and Grimson tracker:Fit per-pixel mixture model to observed distrubution.
[MIT AI Lab VSAM]
![Page 13: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/13.jpg)
http://research.microsoft.com/users/toyama/wallflower.pd
![Page 14: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/14.jpg)
Background removal issues
http://research.microsoft.com/users/toyama/wallflower.pd
![Page 15: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/15.jpg)
Background Subtraction PrinciplesWallflower: Principles and Practice of Background Maintenance, by KentaroToyama, John Krumm, Barry Brumitt, Brian Meyers.
P1:
P2:
P3:
P4:
P5:
![Page 16: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/16.jpg)
Background Techniques Compared
From
the
Wal
lflow
er P
aper
![Page 17: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/17.jpg)
Segmentation as clustering
• Cluster together (pixels, tokens, etc.) that belong together…
• Agglomerative clustering– attach closest to cluster it is closest to– repeat
• Divisive clustering– split cluster along best boundary– repeat
• Dendrograms– yield a picture of output as clustering process continues
![Page 18: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/18.jpg)
Greedy Clustering Algorithms
![Page 19: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/19.jpg)
Data set Dendrogram formed by agglomerative clustering using single-link clustering.
![Page 20: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/20.jpg)
Segmentation methods
• Segment foreground from background• K-means clustering• Mean-shift segmentation• Normalized cuts
![Page 21: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/21.jpg)
K-Means
• Choose a fixed number of clusters
• Choose cluster centers and point-cluster allocations to minimize error
• can’t do this by search, because there are too many possible allocations.
• Algorithm– fix cluster centers; allocate
points to closest cluster– fix allocation; compute best
cluster centers
• x could be any set of features for which we can compute a distance (careful about scaling)
x j − µ i
2
j∈elements of i'th cluster∑
⎧ ⎨ ⎩
⎫ ⎬ ⎭ i∈clusters
∑
![Page 22: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/22.jpg)
K-Means
![Page 23: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/23.jpg)
Matlab k-means clustering demo
![Page 24: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/24.jpg)
Image Clusters on intensity (K=5) Clusters on color (K=5)
K-means clustering using intensity alone and color alone
![Page 25: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/25.jpg)
Image Clusters on color
K-means using color alone, 11 segments
![Page 26: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/26.jpg)
K-means usingcolor alone,11 segments.
Color aloneoften will not yeild salient segments!
![Page 27: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/27.jpg)
Ways to include spatial relationships
(a) Define a Markov Random Field (MRF), where the state to be estimated includes the segment index. Solve by graph cuts or BP.
(b) Augment data to be clustered with spatial coordinates.
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
=
yxvuY
zcolor coordinates
spatial coordinates
![Page 28: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/28.jpg)
K-means using colour andposition, 20 segments
Still misses goal of perceptuallypleasing segmentation!
Hard to pick K…
![Page 29: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/29.jpg)
Segmentation methods
• Segment foreground from background• K-means clustering• Mean-shift segmentation• Normalized cuts
![Page 30: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/30.jpg)
Mean Shift Segmentation
http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html
![Page 31: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/31.jpg)
Mean Shift AlgorithmMean Shift Algorithm
1. Choose a search window size.2. Choose the initial location of the search window.3. Compute the mean location (centroid of the data) in the search window.4. Center the search window at the mean location computed in Step 3.5. Repeat Steps 3 and 4 until convergence.
The mean shift algorithm seeks the “mode” or point of highest density of a data distribution:
![Page 32: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/32.jpg)
Mean Shift Segmentation Algorithm1. Convert the image into tokens (via color, gradients, texture measures etc).2. Choose initial search window locations uniformly in the data.3. Compute the mean shift window location for each initial position.4. Merge windows that end up on the same “peak” or mode.5. The data these merged windows traversed are clustered together.
*Image From: Dorin Comaniciu and Peter Meer, Distribution Free Decomposition of Multivariate Data, Pattern Analysis & Applications (1999)2:22–30
Mean Shift Segmentation
![Page 33: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/33.jpg)
• For your homework, you will do a mean shift algorithm just in the color domain. In the slides that follow, however, both spatial and color information are used in a mean shift segmentation.
![Page 34: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/34.jpg)
Comaniciu and Meer, IEEE PAMI vol. 24, no. 5, 2002
![Page 35: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/35.jpg)
Apply mean shift jointly in the image (left col.) and range (right col.) domains
1
Window in image domain
0 12
Intensities of pixels within image domain window
0 13
Window in range domain
5
4
Center of mass of pixels within both image and range domain windows
0 1 70 1
6
Center of mass of pixels within both image and range domain windows
![Page 36: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/36.jpg)
Mean Shift color&spatial Segmentation Results:
http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html
![Page 37: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/37.jpg)
Mean Shift color&spatial Segmentation Results:
![Page 38: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/38.jpg)
Segmentation methods
• Segment foreground from background• K-means clustering• Mean-shift segmentation• Normalized cuts
![Page 39: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/39.jpg)
Graph-Theoretic Image Segmentation
Build a weighted graph G=(V,E) from image
V: image pixels
E: connections between pairs of nearby pixels
region same the tobelong
j& iy that probabilit :ijW
![Page 40: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/40.jpg)
Graphs Representations
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
0110110000100000000110010a
d
b
ce
Adjacency Matrix
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 41: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/41.jpg)
Weighted Graphs and Their Representations
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
∞∞∞
∞∞∞
0172106760432401
310a
e
d
c
b
6
Weight Matrix
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 42: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/42.jpg)
Boundaries of image regions defined by a number of attributes
– Brightness/color– Texture– Motion– Stereoscopic depth– Familiar configuration
[Malik]
![Page 43: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/43.jpg)
Measuring AffinityIntensity
aff x, y( )= exp − 12σ i
2⎛ ⎝
⎞ ⎠ I x( )− I y( ) 2( )⎧
⎨ ⎩
⎫ ⎬ ⎭
Distance
aff x, y( )= exp − 12σ d
2⎛ ⎝
⎞ ⎠ x − y 2( )⎧
⎨ ⎩
⎫ ⎬ ⎭
Color
aff x, y( )= exp − 12σ t
2⎛ ⎝
⎞ ⎠ c x( )− c y( ) 2( )⎧
⎨ ⎩
⎫ ⎬ ⎭
![Page 44: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/44.jpg)
Eigenvectors and affinity clusters• Simplest idea: we want a
vector a giving the association between each element and a cluster
• We want elements within this cluster to, on the whole, have strong affinity with one another
• We could maximize
• But need the constraint
• This is an eigenvalueproblem (p. 321 of Forsyth&Ponce)
• - choose the eigenvector of A with largest eigenvalue
aT Aa
aTa = 1
![Page 45: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/45.jpg)
Example eigenvector
points
eigenvector
matrix
![Page 46: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/46.jpg)
Example eigenvector
points
eigenvector
matrix
![Page 47: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/47.jpg)
Scale affects affinity
σ=.2
σ=.1 σ=.2 σ=1
![Page 48: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/48.jpg)
Some Terminology for Graph Partitioning
• How do we bipartition a graph:
∅=∩
∈∈∑=
BAwith
BA,
),,W(B)A,(vu
vucut
disjointy necessarilnot A' andA
A'A,
),(W)A'A,( ∑∈∈
=vu
vuassoc
[Malik]
![Page 49: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/49.jpg)
Minimum CutA cut of a graph G is the set of edges S such that removal of S from G disconnects G.
Minimum cut is the cut of minimum weight, where weight of cut <A,B> is given as
( ) ( )∑ ∈∈=
ByAxyxwBAw
,,,
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 50: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/50.jpg)
Minimum Cut and Clustering
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 51: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/51.jpg)
Drawbacks of Minimum Cut
• Weight of cut is directly proportional to the number of edges in the cut.
Cuts with lesser weightthan the ideal cut
Ideal Cut
* Slide from Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 52: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/52.jpg)
Normalized cuts
• First eigenvector of affinity matrix captures within cluster similarity, but not across cluster difference
• Min-cut can find degenerate clusters
• Instead, we’d like to maximize the within cluster similarity compared to the across cluster difference
• Write graph as V, one cluster as A and the other as B
• Minimize
where cut(A,B) is sum of weights with one end in A and one end in B; assoc(A,V) is sum of all edges with one end in A.
I.e. construct A, B such that their within cluster similarity is high compared to their association with the rest of the graph
cut(A,B)assoc(A,V)
cut(A,B)assoc(B,V)
+
![Page 53: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/53.jpg)
Solving the Normalized Cut problem
• Exact discrete solution to Ncut is NP-complete even on regular grid,– [Papadimitriou’97]
• Drawing on spectral graph theory, good approximation can be obtained by solving a generalized eigenvalue problem.
[Malik]
![Page 54: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/54.jpg)
Normalized Cut As Generalized Eigenvalue problem
after simplification, Shi and Malik derive
...
),(
),( ;
11)1()1)(()1(
11)1)(()1(
)VB,()BA,(
)VA,(B)A,(B)A,(
0
=
=−
−−−+
+−+=
+=
∑∑ >
i
xT
T
T
T
iiD
iiDk
DkxWDx
DkxWDx
assoccut
assoccutNcut
i
.01},,1{ with ,)(),( =−∈−
= DybyDyy
yWDyBANcut TiT
T
[Malik]
∑=j
ijii AD
![Page 55: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/55.jpg)
Normalized cuts
• Instead, solve the generalized eigenvalue problem
• which gives
• They show that the 2nd smallest eigenvector solution y is a good real-valued appox to the original normalized cuts problem. Then you look for a quantization threshold that maximizes the criterion --- i.e all components of y above that threshold go to one, all below go to -b
maxy yT D − W( )y( ) subject to yT Dy = 1( )
D − W( )y = λDy
http://www.cs.berkeley.edu/~malik/papers/SM-ncut.pdf
![Page 56: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/56.jpg)
Brightness Image Segmentation
http://www.cs.berkeley.edu/~malik/papers/SM-ncut.pdf
![Page 57: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/57.jpg)
Brightness Image Segmentation
http://www.cs.berkeley.edu/~malik/papers/SM-ncut.pdf
![Page 58: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/58.jpg)
http://www.cs.berkeley.edu/~malik/papers/SM-ncut.pdf
![Page 59: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/59.jpg)
Results on color segmentation
http://www.cs.berkeley.edu/~malik/papers/SM-ncut.pdf
![Page 60: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/60.jpg)
Nice web page on grouping from Malik’s group.
![Page 61: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/61.jpg)
Contains a large dataset of images with human “ground truth”labeling.
![Page 62: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/62.jpg)
Of course, the human labelingsdiffer one from another.
![Page 63: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/63.jpg)
Line Fitting
• Hough transform• Iterative fitting
![Page 64: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/64.jpg)
Fitting
• Choose a parametric object/some objects to represent a set of tokens
• Most interesting case is when criterion is not local– can’t tell whether a set of
points lies on a line by looking only at each point and the next.
• Three main questions:– what object represents this
set of tokens best?– which of several objects
gets which token?– how many objects are
there?
(you could read line for object here, or circle, or ellipse or...)
![Page 65: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/65.jpg)
Fitting and the Hough Transform• Purports to answer all three
questions– in practice, answer isn’t
usually all that much help• We do for lines only• A line is the set of points (x, y)
such that
• Different choices of θ, d>0 give different lines
• For any (x, y) there is a one parameter family of lines through this point, given by
• Each point gets to vote for each line in the family; if there is a line that has lots of votes, that should be the line passing through the points
sinθ( )x + cosθ( )y + d = 0sinθ( )x + cosθ( )y + d = 0
![Page 66: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/66.jpg)
d
θtokens
Votes for parameter values satisfyingat each token
sinθ( )x + cosθ( )y + d = 0
![Page 67: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/67.jpg)
Mechanics of the Hough transform
• Construct an array representing θ, d
• For each point, render the curve (θ, d) into this array, adding one at each cell
• Difficulties– how big should the cells be?
(too big, and we cannot distinguish between quite different lines; too small, and noise causes lines to be missed)
• How many lines?– count the peaks in the
Hough array
• Who belongs to which line?– tag the votes
• Problems with noise and cell size can defeat it
![Page 68: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/68.jpg)
tokens votes
![Page 69: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/69.jpg)
![Page 70: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/70.jpg)
![Page 71: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/71.jpg)
![Page 72: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/72.jpg)
Rules of thumb for getting Hough transform to work well
• Can work for finding lines in a set of edge points.
• Ensure minimum number of irrelevant tokens by tuning the edge detector.
• Choose the quantization grid carefully by trial and error.
![Page 73: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/73.jpg)
Line fitting
What criteria to optimize when fitting a line to a set of points?
![Page 74: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/74.jpg)
“Least Squares”
Line fitting can be max.likelihood - but choice ofmodel is important
“Total Least Squares”
![Page 75: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/75.jpg)
Who came from which line?
• Assume we know how many lines there are - but which lines are they?– easy, if we know who came from which line
• Three strategies– Incremental line fitting– K-means (described in book)– Probabilistic (in book, and in earlier lecture
notes)
![Page 76: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/76.jpg)
![Page 77: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/77.jpg)
Incremental line fitting
![Page 78: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/78.jpg)
Incremental line fitting
![Page 79: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/79.jpg)
Incremental line fitting
![Page 80: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/80.jpg)
Incremental line fitting
![Page 81: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/81.jpg)
Incremental line fitting
![Page 82: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/82.jpg)
Fitting contours
• Two common techniques:– Snakes (Terzopolous, Witkin, and Kass)– Dynamic programming methods
![Page 83: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/83.jpg)
http://www.cs.huji.ac.il/~shashua/papers/saliency.pdf
![Page 84: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/84.jpg)
http://people.csail.mit.edu/people/billf/freemanThesis.pdf
![Page 85: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/85.jpg)
http://people.csail.mit.edu/people/billf/freemanThesis.pdf
![Page 86: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/86.jpg)
http
://pe
ople
.csa
il.m
it.ed
u/pe
ople
/bill
f/fre
eman
Thes
is.p
df
![Page 87: Segmentation and low-level grouping. · 2005-04-14 · The generic, unavoidable problem with low-level segmentation and grouping • It makes a hard decision too soon. We want to](https://reader035.vdocuments.net/reader035/viewer/2022070916/5fb679c6752d172c733ffb08/html5/thumbnails/87.jpg)
http
://w
ww
.cs.h
uji.a
c.il/
~sha
shua
/pap
ers/
salie
ncy.