![Page 1: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/1.jpg)
Yuri Boykov, UWO
1: Basics of optimization-based segmentation
- continuous and discrete approaches
2 : Exact and approximate techniques
- non-submodular and high-order problems
3: Multi-region segmentation (Milan)
- high-dimensional applications
Tutorial on Medical Image Segmentation: Beyond Level-SetsMICCAI, 2014 Western University
Canada
www.csd.uwo.ca/faculty/yuri/miccai14_MIS
![Page 2: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/2.jpg)
Yuri Boykov, UWO
Introduction to Image Segmentation implicit/explicit representation of boundaries
• active contours, level-sets, graph cut, etc. Basic low-order objective functions (energies)
• physics, geometry, statistics, information theory
Set functions, submodularity• Exact methods
Approximation methods• Higher-order and non-submodular objectives• Comparison to gradient descent (level-sets)
Part
1Pa
rt 2
![Page 3: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/3.jpg)
Yuri Boykov, UWO
Thresholding
T
![Page 4: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/4.jpg)
Yuri Boykov, UWO
Thresholding
S={ p : Ip < T }
T
![Page 5: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/5.jpg)
Yuri Boykov, UWO
Background Subtraction
?
Thresholding
- =I= Iobj - Ibkg
Threshold intensities above T
better segmentation?
![Page 6: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/6.jpg)
Yuri Boykov, UWO
Good segmentation S ?
Objective function must be specified
Quality function
Cost function
Loss function E(S) : 2P “Energy”
Regularization functional
Segmentation becomes an optimization problem: S = arg min E(S)
![Page 7: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/7.jpg)
Yuri Boykov, UWO
Good segmentation S ?
combining different constraints
e.g. on region and boundary
Objective function must be specified
Quality function
Cost function
Loss function E(S) = E1(S)+…+ En(S)“Energy”
Regularization functional
Segmentation becomes an optimization problem: S = arg min E(S)
![Page 8: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/8.jpg)
Yuri Boykov, UWO
Beyond linear combination of terms
Ratios are also used• Normalized cuts [Shi, Malik, 2000]
• Minimum Ratio cycles [Jarmin Ishkawa, 2001]
• Ratio regions [Cox et al, 1996]
• Parametric max-flow applications [Kolmogorov et al 2007]
)()(
)(SE
SESE
2
1
Not in this tutorial
![Page 9: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/9.jpg)
Yuri Boykov, UWO
Segmentation principles
Boundary seeds• Livewire (intelligent scissors)
Region seeds• Graph cuts (intelligent paint)• Distance (Voronoi-like cells)
Bounding box• Grabcut [Rother et al]
Center seeds• Star shape [Veksler]
Many other options…
Normalized cuts [Shi Malik] Mean-shift [Comaniciu] MDL [Zhu&Yuille] Entropy of appearance
interactive vs. unsupervised
Add enough constraints: Saliency Shape Known appearance Texture
![Page 10: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/10.jpg)
Yuri Boykov, UWO
Boundary seeds• Livewire (intelligent scissors)
Region seeds• Graph cuts (intelligent paint)• Distance (Voronoi-like cells)
Bounding box• Grabcut [Rother et al]
Center seeds• Star shape [Veksler]
Many other options…
Normalized cuts [Shi Malik] Mean-shift [Comaniciu] MDL [Zhu&Yuille] Entropy of appearance
interactive vs. unsupervised
Add enough constraints: Saliency Shape Known appearance Texture
1. We won’t cover everything…2. We will not emphasize the differences between interactive and unsupervised (too easy to convert one into the other)
![Page 11: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/11.jpg)
Yuri Boykov, UWO
optimization-based
Common segmentation techniques
region-growing
intelligent scissors(live-wire)
active contours(snakes)
watersheds
boundary-based region-based both region & boundary
thresholding geodesic
active contours(e.g. level-sets)
MRF
(e.g. graph-cuts)
random walker
![Page 12: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/12.jpg)
Yuri Boykov, UWO
Common surface representations
mesh level-setsgraph
labelingon complex
on grid
point cloudlabeling
ps
continuous optimization
mixed optimization
Zps p{0,1}ps
combinatorial optimization
![Page 13: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/13.jpg)
Yuri Boykov, UWO
Active contours (e.g. snakes)
[Kass, Witkin, Terzopoulos 1987]
Given: initial contour (model) near desirable object
Goal: evolve the contour to fit exact object boundary
![Page 14: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/14.jpg)
Yuri Boykov, UWO
5-14
Tracking via active contours
Tracking Heart Ventricles
![Page 15: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/15.jpg)
Yuri Boykov, UWO
5-15
Active contours - snakes Parametric Curve Representation (continuous case)
A curve can be represented by 2 functions
open curve closed curve
1s0sy,sxs ))()(()(ν
]}1,0[|)({ ssC ν
parameter
![Page 16: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/16.jpg)
Yuri Boykov, UWO
5-16
Snake Energy
)C(E)C(E)C(E exin
internal energy encourages smoothness or any particular
shapeexternal energy encourages curve onto image structures (e.g. image
edges)
![Page 17: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/17.jpg)
Yuri Boykov, UWO
5-17
Active contours - snakes (continuous case)
internal energy (physics of elastic band)
external energy (from image)
elasticity / stretching stiffness / bending
1
02
221
0
2
in dssd
ddsdsd)C(E νν
1
0
2ex ds|s(I|)C(E ))(v
proximity to image edges
![Page 18: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/18.jpg)
Yuri Boykov, UWO
Active contours – snakes (discrete case)
5v4v
3v
2v
1v6v
7v
8v
10v
9v
elastic energy(elasticity)
211
iiv
ds
d
bending energy
(stiffness)1ii1i1iii1i2
2
2)()(ds
d
)( iii y,xν
2n)( 1n210 ,....,,, ννννC
![Page 19: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/19.jpg)
Yuri Boykov, UWO
5-19
Basic Elastic Snake
continuous case
discrete case
1
0
21
0
2 ds|))s(v(I|ds|ds
dv|E
1n
0i
2i
1n
0i
2i1i |)v(I||vv|E
elastic smoothness term(interior energy)
image data term(exterior energy)
]}1,0[s|)s({ νC
}ni0|{ i νC
![Page 20: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/20.jpg)
Yuri Boykov, UWO
5-20
Snakes - gradient descentsimple elastic snake energy
tE' CCupdate equation for the whole snake
t...
yx
...y
x
'y'x
...'y
'x
1n
1n
0
0
yE
xE
yE
xE
1n
1n
0
0
1n
1n
0
0
C
21
1
0
21 )()( ii
n
iii yyxx
2iiy
1n
0i
2iix1n01n0 |)y,x(I||)y,x(I|)y,,y,x,,x(E
here, energy is a function of 2n variables
C
![Page 21: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/21.jpg)
Yuri Boykov, UWO
5-21
Snakes - gradient descentsimple elastic snake energy
i
i
yE
xE
iF
iF tF' iii
νν
update equation for each node
Ct...
yx
...y
x
'y'x
...'y
'x
1n
1n
0
0
yE
xE
yE
xE
1n
1n
0
0
1n
1n
0
0
21
1
0
21 )()( ii
n
iii yyxx
2iiy
1n
0i
2iix1n01n0 |)y,x(I||)y,x(I|)y,,y,x,,x(E
here, energy is a function of 2n variables
C
![Page 22: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/22.jpg)
Yuri Boykov, UWO
5-22
Snakes - gradient descent
energy function E(C) for contours C E(C)
0c
EtCC i1i
gradient descent steps
1c2c
local minima for E(C)
c
n2C
second derivative of image intensities
step size could be tricky
![Page 23: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/23.jpg)
Yuri Boykov, UWO
Implicit (region-based) surface representation via level-sets
),( yxuz
}0),(:),{( yxuyxC(implicit contour representation)
[Dervieux, Thomasset, 79, 81] [Osher, Sethian, 89]
![Page 24: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/24.jpg)
Yuri Boykov, UWO
Implicit (region-based) surface representation via level-sets
|| udu NCd
[Dervieux, Thomasset, 79, 81] [Osher, Sethian, 89]
The scaling by is easily verified in one dimension
dC
dCudu ||
x
)(xu || u
p
Normal contour motion can be represented by evolution of level-set function u
Note 2: - level sets can not represent tangential motion of contour points ???
Note 1: - commonly used for gradient descent evolution dtEdC
![Page 25: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/25.jpg)
Yuri Boykov, UWO
Tangential vs. normal motion of contour points
- normal motion of a contour point visibly changes shape (geometry)- tangential motion generates no “visible” shape change
A simple example of tangential motion of contour points
(rotation)
A simple example of normal motion
of contour points(expansion)
Comments: - geometric “energy” of a contour measures “visible” shape properties
(length, curvature, area, e.t.c.). Thus, gradient descent w.r.t. geometric objective generates only ”visible” (normal) motion. - level sets can represent contour gradient descent evolution
only for geometric “energies” E(C) (s.t. E is collinear with contour normal )
EdtCd
N
Level sets(implicit contour representation)
Geodesic active contours
![Page 26: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/26.jpg)
Yuri Boykov, UWO
pEpN
Tangential vs. normal motion of contour points
- normal motion of a contour point visibly changes shape (geometry)- tangential motion generates no “visible” shape change
Parametric contours(explicit contour representation)
Physics-based active contours
Comments: - gradient descent for physics-based “energy” of a contour (e.g. elasticity)
may produce geometrically “invisible” tangential motion of contour points - physics-based energy of a contour depends on its parameterization,
while geometrically it could be the same contour (compare two shapes above)
Q: in what medical applications tangential motion of
segment boundary matters?
![Page 27: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/27.jpg)
Yuri Boykov, UWO
mesh level-sets
continuous optimization
Physics vs. Geometry
snakes, balloons, active contours
• explicit or parametric contour representation• physics-based objectives (typically)• gradient descent• could use dynamic programming in 2D
geodesic active contours
• implicit or non-parametric representation • geometry-based objectives• gradient descent• can use convex formulations (TV-based)
[Amini, Weymouth, Jain, 1990] [Chan, Esidoglu, Nikolova 2006]
![Page 28: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/28.jpg)
Yuri Boykov, UWO
weighted length C
dsgCE )(
Functional E( C )
,~ Ngg
weighted area )int(
)(C
dafCE ~ f
flux C
dsNCE ,)(
v )div(~ v
NdC
Most common geometric functionals for segmentation with level-sets
(boundary alignment to intensity edges)
(region alignment to appearance model)
(oriented boundary alignment)
|| udu or
![Page 29: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/29.jpg)
Yuri Boykov, UWO
weighted lengthgSCE ||)(
Functional E( C )
,~ Ngg
weighted area )int(
)(C
dafCE ~ f
flux C
dsNCE ,)(
v )div(~ v
NdC
Most common geometric functionals for segmentation with level-sets
(boundary alignment to intensity edges)
(region alignment to appearance model)
(oriented boundary alignment)
|| udu or
![Page 30: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/30.jpg)
Yuri Boykov, UWO
Towards discrete geometry:weighted boundary length on a graph
[Barrett and Mortensen 1996]
| I|
)( |I|w
|I|
image-based edge weightspixels
A
B
shortest path algorithm (Dijkstra)
“Live wire” or “intelligent scissors”
![Page 31: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/31.jpg)
Yuri Boykov, UWO
Shortest pathsapproach
shortest path on a 2D graph graph cut
Example: find the shortest
closed contour in a given domain of a
graph
Compute the shortest path p ->p for a point
p.
p
Graph Cutsapproach
Compute the minimum cut that
separates red region from blue
region
Repeat for all points on the gray line. Then choose the optimal
contour.
![Page 32: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/32.jpg)
Yuri Boykov, UWO
graph cuts vs. shortest paths
On 2D grids graph cuts and shortest paths give optimal 1D contours.
A Cut separates regions
A
B
A Path connects points
Shortest paths still give optimal 1-D contours on N-D grids
Min-cuts give optimal hyper-surfaces on N-D grids
![Page 33: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/33.jpg)
Yuri Boykov, UWO
Graph cut
n-links
s
t a cuthard constraint
hard constraint
Minimum cost cut can be computed in polynomial
time(max-flow/min-cut algorithms)
22exp
pq
pq
Iw
pqI
[Boykov and Jolly 2001]
![Page 34: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/34.jpg)
Yuri Boykov, UWO
Minimum s-t cuts algorithms
Augmenting paths [Ford & Fulkerson, 1962] - heuristically tuned to grids [Boykov&Kolmogorov 2003]
Push-relabel [Goldberg-Tarjan, 1986] - good choice for denser grids, e.g. in 3D
Preflow [Hochbaum, 2003] - also competitive
![Page 35: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/35.jpg)
Yuri Boykov, UWO
Optimal boundary in 2D
“max-flow = min-cut”
![Page 36: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/36.jpg)
Yuri Boykov, UWO
Optimal boundary in 3D
3D bone segmentation (real time screen capture, year 2000)
![Page 37: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/37.jpg)
Yuri Boykov, UWO
‘Smoothness’ of segmentation boundary
- snakes (physics-based contours)
- geodesic contours (geometry)
- graph cuts
NOTE: many distance-to-seed methods optimize segmentation boundary only indirectly,
they compute some analogue of optimum Voronoi cells[fuzzy connectivity, random walker, geodesic Voronoi cells, etc.]
(discrete geometry)
![Page 38: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/38.jpg)
Yuri Boykov, UWO
Discrete vs. continuous boundary cost
Geodesic contours
S
s dswSE )(
Both incorporate segmentation boundary smoothness and
alignment to image edges
qp
qppq SSwSE,
][)(
Graph cuts
}10{ ,S p
[Caselles, Kimmel, Sapiro, 1997] (level-sets)[Boykov and Jolly 2001]
C
[Chan, Esidoglu, Nikolova, 2006] (convex) [Boykov and Kolmogorov 2003]
![Page 39: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/39.jpg)
Yuri Boykov, UWO
Graph cuts on a grid and boundary of S
Severed n-links can approximate geometric length of contour C [Boykov&Kolmogorov, ICCV 2003]
This result fundamentally relies on ideas of Integral Geometry (also known as Probabilistic Geometry) originally developed in 1930’s.• e.g. Blaschke, Santalo, Gelfand
}10{ ,S p
Ce
|e|SB )(
![Page 40: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/40.jpg)
Yuri Boykov, UWO
Integral geometry approach to length
C 2
0
a set of all lines L
CL
a subset of lines L intersecting contour C
ddnC L21||||Euclidean length of C :
the number of times line L intersects C
Cauchy-Crofton formula
probability that a “randomly drown” line intersects C
![Page 41: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/41.jpg)
Yuri Boykov, UWO
Graph cuts and integral geometry
C
k
kkknC 21||||
Euclidean length
2kk
kw
gcC ||||graph cut cost
for edge weights:the number of edges of family k intersecting C
Edges of any regular neighborhood system
generate families of lines
{ , , , }
Graph nodes are imbeddedin R2 in a grid-like fashion
Length can be estimated without computing any derivatives
![Page 42: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/42.jpg)
Yuri Boykov, UWO
Metrication errors
“standard” 4-neighborhoods
(Manhattan metric)
larger-neighborhoods8-neighborhoods
Euclidean metric
Riemannianmetric
![Page 43: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/43.jpg)
Yuri Boykov, UWO
Metrication errors
4-neighborhood 8-neighborhood
![Page 44: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/44.jpg)
Yuri Boykov, UWO
implicit (region-based) representation of contours
Differential vs. integral approach to geometric boundary length
ddnC L21||||
Cauchy-Crofton formula
1
0|||| dtCC t
Inte
gra
l g
eom
etr
yD
iffere
nti
al
geom
etr
y
Parametric(explicit)contour
representation
dxuC
|||||| Level-setfunction
representation
![Page 45: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/45.jpg)
Yuri Boykov, UWO
A contour may be approximated from u(x,y) with
sub-pixel accuracy
C
-0.8 0.2
0.5
0.70.30.6-0.2
-1.7
-0.6
-0.8
-0.4 -0.5
)()( y,xupu
• Level set function u(p) is normally stored on image pixels• Values of u(p) can be interpreted as distances or heights of image pixels
Implicit (region-based) surface representation via level-sets
![Page 46: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/46.jpg)
Yuri Boykov, UWO
Implicit (region-based) surface representation via graph-cuts
)( y,xSS p
• Graph cuts represent surfaces via binary labeling Sp of each graph node• Binary values of Sp indicate interior or exterior points (e.g. pixel
centers)
There are many contours satisfying
interior/exterior labeling of points
Question: Is this a contour to be reconstructed from binary labeling Sp ? Answer: NO
0 1
1
1 1 1 0
0
0
0
0 0
C
![Page 47: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/47.jpg)
Yuri Boykov, UWO
Contour/surface representations(summary)
Implicit (area-based) Explicit (boundary-based)
Level sets (geodesic active contours)
Graph cuts(minimum cost cuts)
Live-wire(shortest paths on graphs)
Snakes (physics-based band model)
What else besides boundary length |∂S| ?
![Page 48: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/48.jpg)
Yuri Boykov, UWO
From seeds to more general region constraints
pqw
n-links
s
t a cut)(sDp
t-lin
k
)(tDp
t-link
assume are known
“expected” intensities of object and background
ts II and
|II|tD tpp )(
|II|sD spp )(
SS
segmentation
[Boykov and Jolly 2001]
cost of severed t-links
Sp
tp
Sp
sp IIII ||||E(S) = +
cost of severed n-links|| S
![Page 49: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/49.jpg)
Yuri Boykov, UWO
From seeds to more general region constraints
pqw
n-links
s
t a cut)(sDp
t-lin
k
)(tDp
t-link
could be unknown intensities
of object and background
ts II and
|II|tD tpp )(
|II|sD spp )(
SS
segmentation
[Boykov and Jolly 2001]
cost of severed t-links
Sp
tp
Sp
sp IIII ||||E(S, Is,It) = +
cost of severed n-links|| S
Chan-Vese model
re-estimatets II and
Block-Coord.Descent
![Page 50: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/50.jpg)
Yuri Boykov, UWO
Block-coordinate descent for
Minimize over labeling S for fixed I 0, I 1
Minimize over I 0, I 1 for fixed labeling S
1:
21
0:
2010 )()(),,(pp Sp
pSp
p IIIIIISE
Npq
qppq SSw}{
][
),,( 10 IISE
1:
21
0:
2010 )()(),,(pp Sp
pSp
p IIIIIISE
Npq
qppq SSw}{
][fixed for S=const
optimal L can be computed using graph cuts
optimal I 1, I 0 can be computed by minimizing squared errors inside object and background
segments
0:
||10ˆ
pSppS
II
1:
||11ˆ
pSppS II
![Page 51: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/51.jpg)
Yuri Boykov, UWO
Chan-Vese segmentation(binary case )
1:
21
0:
2010 )()(),,(pp Sp
pSp
p IIIIIISE
Npq
qppq SSw}{
][
}10{ ,S p
![Page 52: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/52.jpg)
Yuri Boykov, UWO
Chan-Vese segmentation(could be used for more than 2 labels )
...)()(,...),,(1:
21
0:
2010
pp Sp
pSp
p IIIIIISE
Npq
qppq SSw}{
][
can be used for segmentation, to reduce color-depth,or to create a cartoon
},...2,1,0{pS
![Page 53: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/53.jpg)
Yuri Boykov, UWO
without the smoothing term, this is like “K-means” clustering in the color space
Chan-Vese segmentation(could be used for more than 2 labels )
...)()(,...),,(1:
21
0:
2010
pp Sp
pSp
p IIIIIISE
can be used for segmentation, to reduce color-depth,or to create a cartoon
},...2,1,0{pS
joint optimization
over S and I0, I1,… is NP-hard
![Page 54: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/54.jpg)
Yuri Boykov, UWO
From fixed intensity segmentsto general intensity distributions
pqw
n-links
s
t a cut)(sDp
t-lin
k
)(tDp
t-link
Appearance models can be
given by intensity distributions
of object and background
)()( t|IPrlntD pp
)()( s|IPrlnsD pp
SS
segmentation
[Boykov and Jolly 2001]
cost of severed t-links
Sp
pSp
p tIsI )|Pr(ln)|Pr(lnE(S) = +cost of severed n-links
|| S
![Page 55: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/55.jpg)
Yuri Boykov, UWO
Graph cut (region + boundary)
![Page 56: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/56.jpg)
Yuri Boykov, UWO
Graph cut as energy optimization for S
pqw
n-links
s
t a cut)(sDp
t-lin
k
)(tDp
t-link
segmentation cut
SS
cost of severed t-links
Sp
pSp
p DD )0()1(cost(cut) = +E(S)
}10{ ,S p
unary terms pair-wise terms
cost of severed n-links
Npq
pq SSw ][ qpp
pp SD )(
regional properties of S boundary smoothness for S
[Boykov and Jolly 2001]
![Page 57: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/57.jpg)
Yuri Boykov, UWO
Unary potentials as linear term wrt.
p
pp SD )(unary terms
Sp
pSp
p DD )0()1(
)( pf
Ssfp
pp f, =
pp
pp SDDconst )0()1(
}10{ ,S p
p
ppp SDSD )-(1)0()1( p
Linear region term analogous to
geodesic active contours
S
fSE )(
![Page 58: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/58.jpg)
Yuri Boykov, UWO
p
pp sfSf ,
Examples of potential functions f
unary terms(linear)
2)( cIf pp • Chan-Vese
1pf• Volume Ballooning
patresponsefilterf p • Attention
)( pp If Prln• Log-likelihoods
Unary potentials as linear term wrt.
![Page 59: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/59.jpg)
Yuri Boykov, UWO
In general,…
pS
Npq
pq SSw ][ qp
pair-wise terms
Npq
pq SSSSw qpqp )1()1(
quadratic polynomial wrt.
k-arity potentials are k-th order polynomial
Quadratic term analogous to boundary
length ingeodesic active
contours
S
sw dswSSE ||)(
![Page 60: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/60.jpg)
Yuri Boykov, UWO
{0,1}ps
Examples of discontinuity penalties w
• Euclidean boundary length
second-order terms
][|| qpNpq
pqw sswS
(quadratic)
• contrast-weighted boundary length
2)( qppq IIw exp~
[Boykov&Jolly, 2001]
Basic (quadratic) boundary regularization
||~
pqwpq
1
[Boykov&Kolmogorov, 2003], via integral geometry
![Page 61: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/61.jpg)
Yuri Boykov, UWO
w|S|Sf,E(S)
Basic second-order segmentation energy
Sp
pf
Npq
pq SSw ][ qp
includes linear and quadratic terms
![Page 62: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/62.jpg)
Yuri Boykov, UWO
w|S|Sf,E(S)
Basic second-order segmentation energy
includes linear and quadratic terms
MAIN ADVANTAGE: guaranteed global optimum (t.e. best segmentation w.r.t. objective) (discrete case) via graph cuts
[Boykov&Jolly’01; Boykov&Kolmogorov’03] public code [BK’2004], fast on CPU
(continuous case) via convex TV formulations[Chen,Esidoglu,Nikolova’06; Chambolle,Pock,Cremers’08] public code [C. Nieuwenhuis’2014], comparably fast on GPU
NOTE: this formulation is different from basic level-sets [Osher&Sethian’89]
![Page 63: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/63.jpg)
Yuri Boykov, UWO
Sf,E(S) B(S)Sf,E(S)
Optimization vs Thresholding
I
Fg)|Pr(I Bg)|Pr(I
Sp
pfE(S)
bg)|Pr(I(p)
fg)|Pr(I(p)lnf(p)
S
thresholding e.g. graph cut [BJ, 2001]
![Page 64: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/64.jpg)
Yuri Boykov, UWO
Other examples of usefulglobally optimizable segmentation objectives
Flux [Boykov&Kolmogorov 2005]
Color consistency [“One Cut”, Tang et al. 2014]
Distance ||S-S0|| from template shape• Hamming, L2,… [e.g. Boykov,Cremers,Kolmogorov, 2006]
Star-shape prior [Veksler 2008]
NOT ALLOWED
![Page 65: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/65.jpg)
Yuri Boykov, UWO
Many more example of usefulhard-to-optimize segmentation objectives
Continuous case• Non-convexity
Discrete case• Non-submodularity (more later)• High-order• Density (too many terms)
Typical Problems: Typical Solutions:
gradient descent (linearization)
+ level sets
to be discussed
![Page 66: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/66.jpg)
Yuri Boykov, UWO
Examples of useful higher-order energies
Cardinality potentials (constraints on segment size)
psE(S) 00 VSV ||
E(S)
|| S0V
can not be represented as a sum of simpler (unary or quadratic)
terms
high-orderpotential
![Page 67: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/67.jpg)
Yuri Boykov, UWO
Examples of useful higher-order energies
Cardinality potentials (constraints on segment size)
E(S)
|| S0V
can be represented as a sum of unary and quadratic terms
NOTE: 2nd-orderpotential
still difficult to optimize(completely connected graph)
2
02
0 psVSV ||E(S)
![Page 68: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/68.jpg)
Yuri Boykov, UWO
Examples of useful higher-order energies
Cardinality potentials (constraints on segment size)
m
pm sVSV 00 ||E(S)
E(S)
|| S0V
can not be represented as a sum of simpler (unary or quadratic)
terms
high-orderpotential
![Page 69: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/69.jpg)
Yuri Boykov, UWO
Examples of useful higher-order energies
Cardinality potentials (constraints on segment size)
Curvature of the boundary Shape convexity Segment connectivity Appearance entropy, color consistency Distribution consistency High-order shape moments …
![Page 70: 1: Basics of optimization-based segmentation - continuous and discrete approaches](https://reader035.vdocuments.net/reader035/viewer/2022062321/56813e18550346895da7f9b3/html5/thumbnails/70.jpg)
Yuri Boykov, UWO
Implicit surface representationGlobal optimization is possible
Summary
Thresholding, region growing Snakes, active contours Geodesic contours Graph cuts (binary labeling, MRF)
Covered basics of: Not-Covered: Ratio functionals Normalized cuts Watersheds Random walker Many others…
High-order modelsTo be covered later: