snakes with some math. acknowledgements: university of western ontario, university of manchester,...
TRANSCRIPT
Snakes with Some Math
Acknowledgements: University of Western Ontario, University of Manchester, demos from Visual Dynamics Group
(University of Oxford),
Snakes
Given: initial contour (model) near desirable object
•Snakes, active contours [Kass, Witkin, Terzopoulos 1987]
•In general, deformable models are widely used
Snakes
•Snakes, active contours [Kass, Witkin, Terzopoulos 1987]
•In general, deformable models are widely used
Given: initial contour (model) near desirable object
Goal: evolve the contour to fit exact object boundary
Tracking via deformable models
1. Use final contour/model extracted at frame t as an initial solution for frame t+1
2. Evolve initial contour to fit exact object boundary at frame t+1
3. Repeat steps 1 and 2 for t ‘= t+1
Tracking via deformable models
Tracking Heart Ventricles
“Snakes”
• A smooth 2D curve which matches to image data• Initialized near target, iteratively refined• Can restore missing data
initial intermediate final
Parametric Curve Representation
(continuous case)• A curve can be represented parametrically
open curve closed curve
10))(),(()( ssysxs
Note: in computer vision and medical image analysis communities the term “snake” is normally associated with such parametric representation of contours.
Internal Energy
• The bending energy of a continuous curve is
The more the curve bends, the larger this value
Elasticity Stiffness
sd
ddsd
sssEin 2
2
)()())((
22
1
0
))(( dssEE inin
External energy
• The external energy describes how well the curve matches the image data locally
• Numerous forms can be used, attracting the curve toward different image features
Simple Edge Strength
• Suppose we have an image I(x,y)• Generate gradient images &• External energy based on edge strength at
a point is then
• An external energy term for a snake is thus
),( yxGx),( yxGy
)|))((||))((|())(( 22 sGsGsE yxex (Negative in order that minimizing it forces the curve toward large edges)
1
0
))(( dssEE exex
Snake Energy (continuous form)
• The total energy of the snake is defined as
e.g. bending energy
e.g. total edge strength under curve
exintotal EEE
1
0
))(( dssEE inin
1
0
))(( dssEE exex
Discrete approach
discrete imagediscrete snake representation
discrete optimization(dynamic programming)
Discrete and Continuous approaches
ContinuousModels in
Image Analysis
DiscreteModels in
Image Analysis
Continuous Model of LightElectro-Magnetic Waves
Discrete Model of LightPhotons
The relationship is somewhat analogous to relation between continuous and discrete models in physics,
e.g.
Parametric Curve Representation
(discrete case)• Represent the curve with a set of n points
10),( niyx iii
Discrete Representation
• If the curve is represented by n points
Elasticity Stiffness
10),( niyx iii
211
iiv
ds
d 11112
2
2)()( iiiiiiids
d
1
0
211
21 |2|||
n
iiiiiiinE
Simple Elastic Curve (example)
• For a curve represented as a set of points a simple elastic energy term is
This encourages the closed curve to shrink to a
point (like a very small elastic band)
1
0
2n
iiin LE
21
1
0
21 )()( ii
n
iii yyxx
Encouraging point spacing
• To stop the curve from shrinking to a point, add a term such as [v-v’]^2 where v’ is average distance between snaxels.
Simple Edge Strength
• An external energy term for a (discrete) snake based on image edge
(Negative in order that minimizing it forces the curve toward large edges)
21
0
2 |),(||),(| iiy
n
iiixex yxGyxGE
Simple Elastic Snake
• A simple elastic snake is thus defined by– A set of n points,– An internal elastic energy term– An external edge based energy term
• To use this to locate the outline of an object– Initialize in the vicinity of the object– Modify the points to minimize the total energy
Simple Elastic Snake
• In this case we have
• where
• Optimization problem (2n variables)– note that we can easily compute the derivatives, which
allows efficient optimization (via gradient descent) converging to a local minima
• more robust option: Dynamic Programming
)()()( xExExE exintotal
Tnn yyxxx ),,,,,( 1010
Synthetic example
(1) (2)
(3) (4)
Dealing with missing data
• The smoothness constraint can deal with missing data (sometimes maybe wrong!):
Relative weighting
• Notice that the strength of the internal elastic component can be controlled by a parameter,
• Increasing this increases stiffness of curve
large small medium
1
0
2n
iiin LE
Note: values of energy are normalized from 0 to 1, so if 0.1 times 10 becomes .01, etc.,Then re-scale it. Basically this weight makes it more sensitive to internal energy.
Simple shape prior
• If object is some smooth variation on a known shape, use
• where give points of the basic shape
1
0
2)ˆ(n
iiiinE
}ˆ{ i
Alternative External (image) Energies
• Directed gradient measures
– Where is the unit normal to the boundary at contour point
– This gives a good response when the boundary has the same direction as the edge, but weaker responses when it does not
1
0,, )()(
n
iiyiyixixex GuGuE
),( ,, iyixi uuu i
Open and Closed Curves
• When using an open curve we can impose constraints on the end points (e.g. end points may have fixed position)
open curveclosed curve
n 0 0
1n
Additional Constraints
• Snakes originally developed for interactive image segmentation
• Initial snake result can be nudged where it goes wrong
(correct the incorrect ones)
• Simply modify the external energy term to
– Pull nearby points toward cursor, or
– Push nearby points away from cursor
Interactive (external) forces
• Pull points towards cursor:
Nearby points get pulled hardest
Negative sign gives better energy for
positions near p
1
02
2
||
n
i ipull p
rE
Interactive (external) forces
• Push points from cursor:
Nearby points get pushed
hardest
Positive sign gives better energy for
positions far from p
1
02
2
||
n
i ipush p
rE
Dynamic snakes
• Adding motion parameters into hidden variables (for each snake node)
• Introduce energy terms for motion consistency
• Example: use optic flow to get candidate
points for the snaxels in the next frame.
Discrete Snakes Optimization• At each iteration we compute a new snake position within proximity to the previous snake• New snake energy should be smaller than the previous one• Stop when the energy can not be decreased within local neighborhood of the snake (local energy minima)
Optimization Methods
1. Gradient Descent
2. Dynamic Programming
Gradient Descent
i
i
yE
xE
E
negative gradient at point (x,y) gives direction of the steepest descent towards lower values of
function E
• Example: minimization of functions of 2 variables
),( yxE
),( 00 yx
Gradient Descent
Etpp
• Example: minimization of functions of 2 variables
),( yxE
),( 00 yx
yE
xE
ty
x
y
x
Stop at a local minima where 0
E
Gradient Descent
),( yxE
• Example: minimization of functions of 2 variables
High sensitivity wrt. the initialisation !!
Gradient Descent for Snakessimple elastic snake energy
21
0
21010 |),(||),(|),,,,,( iiy
n
iiixnntotal yxGyxGyyxxE
21
1
0
21 )()( ii
n
iii yyxx
i
i
yE
xE
iF
iF EF
Force is a negative gradient of energy function (2n-
dimentional vector)
energy as a function 2n variables
1
1
0
...
nF
F
F
We can break force/gradient into components
corresponding to individual snake nodes
Discrete Snakes:“Gradient Flow” evolution
1,,0 nitFiii
Update equation for each node
i
Stopping criteria:
iallforEF ii 0|
at a local minima of energy
i i
EdtdC Contour evolution via
“Gradient flow”
Difficulties with Gradient Descent
• Very difficult to obtain accurate estimates of high-order derivatives on images (discretization errors)– E.g., estimating requires computation of
• Gradient descent is not trivial even for one-dimensional functions. Robust numerical performance for 2n-dimensional function is problematic. – Choice of parameter is non-trivial
• Small , the algorithm may be too slow• Large , the algorithm may never converge
– Even if “converged” to a good local minima, the snake is likely to oscillate near it t
xE
2
2
xI
xGx
tt
Alternative solution for 2D snakes:
Dynamic Programming
1
0110 ),(),,(
n
iiiintotal EE
• In most cases, snake energy can be rewritten as a sum of pair-wise interaction potential
More generally, it can be written as a sum of triple-interaction potentials.
1
01110 ),,(),,(
n
iiiiintotal EE
Snake energy: pair-wise interactions
21
0
21010 |),(||),(|),,,,,( iiy
n
iiixnntotal yxGyxGyyxxE
21
1
0
21 )()( ii
n
iii yyxx
Example: simple elastic snake energy
1
0
210 ||)(||),,(
n
iintotal GE
1
0
21 ||||
n
iii
1
0110 ),(),,(
n
iiiintotal EE
21
21 ||||||)(||),( iiiiii GE where
Q: give an example of snake with triple-interaction potentials?
DP Snakes
1v2v
3v
4v6v
5v
control points
Energy E is minimized via Dynamic Programming
[Amini, Weymouth, Jain, 1990]
),(...),(),(),...,,( 1132221121 nnnn vvEvvEvvEvvvE First-order interactions
DP Snakes
1v2v
3v
4v6v
5v
control points
[Amini, Weymouth, Jain, 1990]
Iterate until optimal position for each point is the center of the box,
i.e. the snake is optimal in the local search space constrained by boxes
Energy E is minimized via Dynamic Programming
),(...),(),(),...,,( 1132221121 nnnn vvEvvEvvEvvvE First-order interactions
),( 44 nvvE),( 433 vvE
)3(3E
)4(3E )4(4E
)3(4E
)2(4E
)1(4E
)4(nE
)3(nE
)2(nE
)1(nE
)2(3E
)1(3E
)4(2E
)3(2E
Dynamic Programming (DP)Viterbi Algorithm
),(...),(),( 11322211 nnn vvEvvEvvE
),( 322 vvE
)1(2E
)2(2E
),( 211 vvE
)( 2nmOComplexity:
0)1(1 E
0)2(1 E
0)3(1 E
0)4(1 E
Here we will concentrate on first-order interactions
states
1
2
…
m
site
s
1v 2v 3v 4v nv
This is iterative, as search nbd changeseach time.
Dynamic Programming for a closed snake?
),(...),(),( 11322211 nnn vvEvvEvvE Clearly, DP can be applied to optimize an open ended snake
Can we use DP for a “looped” energy in case of a closed snake?
1n
),(),(...),(),( 111322211 vvEvvEvvEvvE nnnnn
1n
2
1n
3 4
Dynamic Programming for a closed snake
),(),(...),(),( 111322211 vvEvvEvvEvvE nnnnn
Unfortunately, DP can not be directly applied in case of a “loop”.However, some (approximation) tricks are often used in practice…
1. Use DP to optimize snake energy with fixed (according to a given initial snake position).
2. Use DP to optimize snake energy again. This time fix position of an intermediate node where is an optimal position obtained in step 1.
1
2/2/ ˆnn ̂
Problems with snakes• Depends on number and spacing of control
points
• Snake may oversmooth the boundary
• Not trivial to prevent curve self intersecting
• Can not follow topological changes of objects
(having a model helps!)
Problems with snakes• May be sensitive to initialization
• may get stuck in a local energy minimum near initial contour
• Numerical stability can be an issue for gradient descent and variational methods (continuous formulation)– E.g. requires computing second order derivatives
• The general concept of snakes (deformable models) does generalize to 3D (deformable mesh), but many robust optimization methods suitable for 2D snakes do not apply in 3D– E.g.: dynamic programming only works for 2D snakes
Problems with snakes• External energy: may need to diffuse image gradients,
otherwise the snake does not really “see” object boundaries in the image unless it gets very close to it.
image gradientsare large only directly on the boundary
I
Diffusing Image GradientsIimage gradients diffused via
Gradient Vector Flow (GVF)
Chenyang Xu and Jerry Prince, 98
http://iacl.ece.jhu.edu/projects/gvf/
Alternative Way to Improve External Energy
• Use instead of where D() is– Distance Transform (for detected binary image features, e.g.
edges)
1
0
)(n
iiex vDE
1
0
)(||n
iiex vIE
binary image features(edges)
Distance Transform ),( yxD
Distance Transform can be visualized as a gray-scale image
• Generalized Distance Transform (directly for image gradients)
Distance Transform
34
23
23
5 4 4
223
112
2 1 1 2 11 0 0 1 2 1
0001
2321011 0 1 2 3 3 2
101110 1
2
1 0 1 2 3 4 3 210122
Distance Transform Image features (2D)
Distance Transform is a function that for each image pixel p assigns a non-negative number corresponding to
distance from p to the nearest feature in the image I
)(D)( pD
Some more Variations..
Internal Energy
• Controls the contour smoothness and continuity
dvvvvvv
vvvvvE ii
iiii
iiiiiint
11
11
111 )1()(
External Energy
• Limitations of the constraint of intensity homogeneity – Cannot be applied to open contour– Does not work for band-shaped object
Ultrasound image
Key-chain ring image
Band Energy
• For the contour element , we can define its tangent as:
• Two regions and can be defined for contour element
iv
iviR iR'
Band Energy
• Normalized mean intensity difference
• Band energy (given ring example is opposite)
• Combination of intensity and gradient:
Penalty
Performance
Without band energy
With band energy
Image Initialization Without band energy With band energy
3D reconstruction Experiments
Input
Results