top-down & bottom-up segmentation presented by: joseph djugash is this a building or a horse? do...
TRANSCRIPT
Top-Down & Bottom-Up Segmentation
Presented By:
Joseph Djugash
Is this a Building or a
Horse?
Do these edges and contours
represent anything?
What’s wrong with Segmentation from Image Statistics?
Is this an object boundary?
Slides from Eitan Sharon, ”Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.
Not all Image Statistics are Helpful!
Slides from Eitan Sharon, ”Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.
How can Class Information help?
Where is the object boundary?
Slides from Eitan Sharon, ”Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.
The Class can help resolve ambiguities!
Slides from Eitan Sharon, ”Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.
Motivation
Bottom-Up segmentation Capture image properties Segmentation based on similarities between
image regions
How can we capture prior knowledge of a specific object (class)? Answer: Top-Down Segmentation
Class-Specific, Top-Down Segmentation
Eran Borenstein and Shimon Ullman
MethodInput Fragments
Matching Cover
Method Outline
Fragment Extraction Figure Ground Label Reliability Value
Fragment Matching Individual Correspondences Consistency Reliability
Segmentation Optimal Cover
Fragment Extraction
Want to find fragments that: Generalize well Are specific to the class Add information that other fragments haven’t already given
us Fragment Size varies from 1/50 to 1/7 of object size
Slides from David Bradley, ” Object Recognition with Informative Features and Linear Classification”.
Fragment Extraction
Figure-ground label Manual labeling Learned from relative motion or grey level
variability Reliability Value – Class Specific
Hit rate: A fixed level of false alarms is achieved by
the criterion: Select the k best fragments according to the Hit
rate
Strength of Response –Maximal normalized correlation of a fragment i with each image I in C and NC
x y
x y
vyuxI
vyuxIyxTvuC
),(
),(),(),(
2
Method Outline
Fragment Extraction Figure Ground Label Reliability Value
Fragment Matching Individual Correspondences Consistency Reliability
Segmentation Optimal Cover
Fragment Matching – Individual Correspondences Measuring Similarity
Region Correlation Normalized Correlation Restrict to pixels with the “figure” label
Edge Information Derived from the boundary of the
figure-ground label
The Similarity Measure:
Fragment Matching – Special Requirements The Difference
Entire Template
Figure Part Only Figure & Edge Similarity
Input Template
Fragment Matching – Consistency To acquire a good global cover of the shape,
each local match needs to satisfy the consistency measure.
Consistency Measure:
Fragment Matching – Reliability Using more “reliable” (anchor) fragments is likely
to increase the chance of finding the optimal cover
A fragment’s reliability is evaluated by the likelihood ratio between the hit/detection rate and the false alarm rate
Reliable fragments used first to guide the covering process
Fragment Matching – Reliability
Easy to identify and more commonly seen fragments used first
Problem Areas –Do not exactly follow image discontinuities
Method Outline
Fragment Extraction Figure Ground Label Reliability Value
Fragment Matching Individual Correspondences Consistency Reliability
Segmentation Optimal Cover
Segmentation – The Cover Algorithm The best cover should maximize individual
match quality, consistency and reliability Thus the cover score is written:
Penalizes for inconsistent overlapping fragments
Rewards for match quality and reliability
Constant that determines the magnitude of the penalty for insufficient consistency
Zero for non-overlapping pairs
Segmentation – The Cover Algorithm Initialize with a sub-window that has the maximal
concentration of reliable fragments Similarity of all the reliable fragments is examined at 5
scales at all possible locations Iterative Algorithm:
Select a small number (M=15) of good candidate fragments
Add to cover a subset of the M fragments that maximally improve the score
Remove existing fragments inconsistent with new cover (fragments with cumulative negative score)
Guaranteed to converge to a local max – score is bounded and increases each iteration
Results I
Results I (cont.)
Results I (cont.)
Learning to Segment
Eran Borenstein and Shimon Ullman
Method – OldInput Fragments
Matching Cover
Method – Updated
Learning Figure-Ground Segmentation – Degree of Cover Start with over-segmented fragments –
each fragment now contains many regions Degree of Cover (ri)
Calculated by counting the average number of fragments (from C) overlapping the region Ri
The fragment selection method extracts most fragments from the figure region Higher ri higher likelihood to be “figure”
Lower ri lower likelihood to be “background”
Learning Figure-Ground Segmentation – Degree of Cover
By thresholding the degree of cover, ri, we can choose the figure part to be:
Most likely figure region
Learning Figure-Ground Segmentation – Border Consistency A fragment often contains multiple edges Determine the boundary that optimally
separates figure from the background Fragment hit (Hj={1,n}) – image patches
where fragment Fi is detected Border Consistency:
Learning Figure-Ground Segmentation – Border Consistency
This approach emphasizes consistent edges (border and interior edges) while diffusing noise edges (background features).
Learning Figure-Ground Segmentation Combining degree of cover and boarder
consistency we get the figure part (P)
Maximized when P contains the most of the consistent edges
Maximized when the boundary between the figure and ground are supported by the consistent edges
Fragments detected in an image applies its figure-ground “vote” for all the pixels it covers
Li(x,y) = +1 – vote for figure label
Li(x,y) = –1 – vote for background label
i w(i) Li(x,y) – total votes for pixel (x,y)
Reliability of fragment i
Improving Figure-Ground Labeling
Fragments that are not consistent with the cover (S) is removed and a new cover (S') is generated
Further Refinements: Modify the degree of cover to be the average number
of times its pixels cover figure parts With a more accurate degree of cover, individual
pixels can be substituted for the sub-regions This new degree of cover can them produce an
improved cover This iterative approach converges within 3 iterations
Results II
Results II (cont.)
Results II (cont.)
Results II (cont.)
Bottom-Up Segmentation“Segmentation and Boundary Detection Using Multiscale Intensity Measurements”
Eitan Sharon, Achi Brandt, and Ronen Basri
Segmentation by Weighted Aggregation Normalized-cuts measure in graphs
Detect segments that optimize a NCut measure Hierarchical Structure
Recursively coarsen a graph reflecting similarities between intensities of neighboring points
Aggregates of pixels of increasing size are gradually collected to form segments
Modify the graph to reflect the coarse scale measurements based on computed properties of the aggregates Use multiscale measures of intensity, texture, shape,
and boundary integrity
Normalized-Cut Measure2( ) ( )ij i j
i j
E S w u u
Si
Siui 0
1
( ) ij i jN S w u u
( )( )
( )
E SS
N S
Minimize:
Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.
Segmentation by Weighted Aggregation Normalized-cuts measure in graphs
Detect segments that optimize a NCut measure Hierarchical Structure
Recursively coarsen a graph reflecting similarities between intensities of neighboring points
Aggregates of pixels of increasing size are gradually collected to form segments
Modify the graph to reflect the coarse scale measurements based on computed properties of the aggregates Use multiscale measures of intensity, texture, shape,
and boundary integrity
Bottom-Up Segmentation
Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.
Segmentation by Weighted Aggregation Normalized-cuts measure in graphs
Detect segments that optimize a NCut measure Hierarchical Structure
Recursively coarsen a graph reflecting similarities between intensities of neighboring points
Aggregates of pixels of increasing size are gradually collected to form segments
Modify the graph to reflect the coarse scale measurements based on computed properties of the aggregates Use multiscale measures of intensity, texture, shape,
and boundary integrity
Full Texture – Lion Cub
Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.
Full Texture – Polar Bear
Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.
Full Texture - Zebra
Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.
Benefits of the Hierarchical Structure
Able to detect regions that differ by fine as well as coarse properties
Accurate detection of individual object boundaries
Able to detect regions separated by weak, yet consistent edges By combining intensity difference with
measures of boundary integrity across neighboring aggregates
Combining Top-Down and Bottom-Up Segmentation
Eran Borenstein, Eitan Sharon and Shimon Ullman
Another step towards the middle
Bottom-Up
Top-Down
Some Definitions & Constraints
Measure of saliency h(i), hi є [0,1)
A configuration vector s contains labels si (1/-1) of all the segments (Si) in the tree
The label si can be different from its parent’s label s i
–
Cost function for a given s
Top-down term Bottom-up termDefines the weighted edge between Si & Si
–
Classification Costs
The terminal segments of the tree determine the final classification
The top-down term is defined as:
The saliency of a segment should restrict its label (based on its parent’s label)
The bottom-up term is defined as:
Minimizing the Costs – Information Exchange in a Tree Bottom-up message:
Top-down message:
Min-cost Label:
Cost of si = –1and s = x
Message from si = –1Cost of si = +1
and s = xMessage
from si = +1
Computed at each node – minimal of the values is the selected label of node s in s
Minimal Cost if the region was classified as background
Minimal Cost if the region was classified as figure
Confidence Map Evaluating the confidence of a region:
Causes of Uncertainty of Classification Bottom-up uncertainty – regions where there is no
salient bottom-up segment matching the top-down classification
Top-down uncertainty – regions where the top-down classification is ambiguous (highly variable shape regions)
The type of uncertainty and the confidence values can be used to select appropriate additional processing to improve segmentation
Results III
Results III (cont.)
Results III (cont.)
Results III (cont.)
Results III (cont.)
Results III (cont.)
Questions?
– Appendix – Why Fragments?
Image fragments make good features especially when training data is limited
Image fragments contain more information than wavelets allows for simpler classifiers
Information theory framework for feature selection
Vs.
Slides from David Bradley, ” Object Recognition with Informative Features and Linear Classification”.
– Appendix – Intermediate complexity
Slides from David Bradley, ” Object Recognition with Informative Features and Linear Classification”.