the university of ontario how to fit a surface to a point cloud? or optimization of surface...
TRANSCRIPT
The University of
Ontario
How to fit a surface to a point cloud?or
optimization of surface functionals in computer vision
Yuri Boykov
TRICS seminarComputer Science Department
The University of
Ontario
Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision
• templates, pictorial structures, trees, deformable models, contours/snakes, meshes, surfaces, complexes, graphs, weak-membrane model, Mumford-Shah, Potts model,……
Optimization in computer vision• dynamic programming, gradient descent, PDEs, shortest paths, min. spanning
trees, linear and quadratic programming, primal-dual schema, network flow algorithms, QPBO, ...
Applications• segmentation, stereo, multi-view reconstruction, optical
flows• surface fitting
The University of
Ontario+Shading
M.C. EscherDrawing hands
3D shapeunderstanding
The University of
Ontario+Color
Da VinciMadonna Litta
Recognition
The University of
Ontario+Texture
MagritteSouvenir de Voyage
recognizingmaterial
The University of
Ontario+Texture
The New Yorker Album of Drawings, The Viking Press, NY, 1975
recognizing 3D perspective
The University of
OntarioWhat do humans get by ‘looking’?
J. VermeerThe Guitar Player
Contours Shading Color Texture …
basic image cues:
The University of
OntarioWhat do humans get by ‘looking’?
Contours Shading Color Texture …
basic image cues:
Segmentation Motion 3D shape perception 3D scene geometry Detection/Recognition …
higher-level perception:
The University of
OntarioWhat do computers get by ‘looking’?
x
y
3D plot of image intensity I(x,y)
x
y
I(x,y)
x
y
The University of
OntarioWhat do computers get by ‘looking’?
P. PicassoThe Guitar Player
Intensity discontinuities (contours)
Intensity gradients (shading)
Multi-valued intensities (color)
Filtering (e.g. texture)
…
basic image cues: higher-levelgrouping?
The University of
Ontario
model
Bayesian approach
Prior + Data Low-level cues
(local info)high-level knowledge
(global picture)
Fit some prior model into data
The University of
OntarioRigid Template Matching
• In matching we estimate “position” of a rigid template in the image
• “Position” includes global location parameters of a rigid template:
- translation, rotation, scale,…
Face templateimage
image
translation,rotation,scaling
The University of
OntarioNon-rigid (parametric) matching
1. Pick one image (red)
2. Warp the other images to match it (homographic transform)
3. Blend
panorama mosaicing
The University of
Ontarioe.g…. using flexible templates
• In flexible template matching we estimate “position” of each rigid component of a template
• For tree-structured models, efficient global optimization is possible via DP (Felzenswalb&Huttenlocher 2002)
The University of
Ontariotracking parameters => activity recognition
Bottom-up tracker
The University of
Ontario
5-18
deformable contours (“snakes”)
2D curve which matches to image data Initialized near target, iteratively refined Can restore missing data
initial intermediate final
Optimization gets harder when a loop is introduced. DP does not apply.
One solution: gradient descent
Kass, Witkin, Terzopoulos 1987
The University of
Ontario
6-19Cremers, Tischhäuser, Weickert, Schnörr, “Diffusion Snakes”, IJCV '02
local minima, fixed contour topology
The University of
Ontario
6-20
A contour may be approximated from
u(x,y) with near sub-pixel accuracy
C
-0.8 0.2
0.5
0.70.30.6-0.2
-1.7
-0.6
-0.8
-0.4 -0.5
),( ppp yxuu
• Level set function u(x,y) is normally discretized/stored over image pixels• Values of u(p) can be interpreted as distances or heights of image pixels
Implicit representation of contoursOsher&Sethian 1989
The University of
Ontario
6-21
[Visualization is courtesy of O. Juan] Simple evolution
Morphological Operation:Erosion
1p
ppp NCd
|| ppp udu
The University of
Ontario
6-22
Visualization is courtesy of O. Juan
Example of gradient descent evolution
2
2
2
2
)(yu
xu
p dtu
Gradient descent w.r.t.
Euclidean length
The University of
Ontario
6-23
Example of gradient descent evolution
Laplacian
Osher&Sethian 1989
Gradient descent w.r.t.
Euclidean length
2
2
2
2
)(yu
xu
p dtu
The University of
Ontario
6-24
[example from Goldenberg, Kimmel, Rivlin, Rudzsky, IEEE TIP ’01] Geodesic Active Contours via Level-sets
y
ug
xug
pyxdtu
)()()(
C
dsgCE )()(
The University of
Ontario
6-25
Other geometric energy functionals besides length [courtesy of Ron Kimmel]
weighted length C
dsgCE )()(
Functional E( C ) gradient descent evolution
,~ Ngg
weighted area
dafCE )( ~ f
alignment(flux)
C
dsNCE ,)(
v )div(~ v
W
Geometric measures commonly used in segmentation
NdC
The University of
Ontario
in 3D…deformable meshes, level-sets, …
Sphere.mpg
Estimation of positionfor mesh points
Many loops.optimization - gradient
descent
GOALS: global optima (?) “right” functional (?)
Typical problems:- local minima (clutter,
outliers)-over-smoothing
The University of
Ontario
Global Optimization andSurface Functionals
The University of
Ontario
More generally...
Estimate labels for graph nodes
I
p
L
along one scan line in the image
observed noisy image I image labeling L(restored intensities)
NOTE:similar to robust regression modelestimation
The University of
Ontario
(simple example)
Piece-wise smooth restoration
Markov Random Fields (MRF) approach
weak membrane model (Geman&Geman’84, Blake&Zisserman’83,87)
pL qL
Nqp
qpp
pp LLVILLE),(
2 ),()()(
discontinuity preserving prior
optimizing E(L) is NP hard!
,V
T T
(Continuous analogue: Mamford-Shah functional, 1989)
The University of
Ontario
I
p
L
observed noisy image I image labeling L(restored intensities)
(simple example)
Piece-wise constant restoration
along one scan line in the image
The University of
Ontario
(simple example)
Piece-wise constant restoration
Potts model Boykov Veksler Zabih ’01Greig et al.’89 (for 2 labels)
Nqp
qpp
pi
i
LLILLEi ),(
2 )()()(
global optimization is still NP hard, but there are fast provably good
combinatorial approximation algorithms, linear and quadratic programming, QPBO, primal-dual
schema
pL qL
qp LL
0}2L:p{ p2
“perceptual grouping”
}1L:p{ p1
}0L:p{ p0
The University of
OntarioPerceptual grouping from stereo
(Birchfield &Tomasi’99) constant label = plane
The University of
Ontario
Binary labeling(binary image restoration)
)|Pr()()( 2pppppp LIILLD
}1,0{pL
original binary image I optimal binary labeling L
Nqp
qpp
pp LLLDLE),(
)()()( Greig Porteous Seheult ’89
Globally optimal solution is possible using combinatorial graph cut algorithms• pseudo-boolean optimization Hammer’65, Picard&Ratlif’75
The University of
Ontario
Binary labeling(object extraction)
1pL
0pL
object segmentation
left ventricle of heart
}1,0{pL
The University of
Ontario
Binary labeling(object extraction)
10
C
Globally optimal solution is possible using graph cut algorithms• pseudo-boolean optimization (Hammer’65, Picard&Ratlif’75)
surface extraction
Nqp
qppqp
pp LLwLDLE),(
)()()(
Boykov&Jolly’01
left ventricle of heart
}1,0{pL
The University of
OntarioImplicit surface representation via graph-cuts
• Any contour (or surface in 3D) satisfying labeling of exterior/interior points (pixel centers) is acceptable if some explicit surface has to be output.
0 1
1
1 1 1 0
0
0
0
0 0
The University of
Ontario
)(
)(,)()(CC C
x dpxfdsdsgCE vN
)int(
)(,)()(CC
x
C
dpxfdsdsgCE vN
Geometric length any convex,
symmetric metric (e.g. Riemannian)
Flux any vector field v
Regional bias any scalar function f
“edge alignment”
Tight characterization for geometric functionals of contour C that can be globally optimized by graph cut algorithms (Kolmogorov&Boykov’05)
disclaimer: for pairwise interactions only
Global optimization of geometric surface functionals
N)q,p(
qpp
pp )L,L(V)L(D)L(E
The University of
OntarioGlobally optimal surfaces in 3D
Volumetric segmentation(BJ01,BK’03,KB’05)
The University of
Ontario
Binary labeling(object extraction)
Blake et al.’04, Rother et al.’04
}1,0{pL
iteratively re-estimate color models e.g. using mixture of Gaussians
The University of
OntarioOptimal surfaces in 3D
3D reconstructionVogiatzis, Torr, Cippola’05
Local cues: voxel’s photoconsistency
Prior:smoothness,
projective geometry constraints
The University of
OntarioGlobally optimal surfaces in 3D
Lempitsky&Boykov, 2006
from a cheap digital camera
The University of
Ontario
3D model
multi-view reconstruction set up
Furukawa&Ponce 2006
Globally optimal surfaces in 3D
(texture mapped)
The University of
OntarioSurface fitting to point cloud
a cloud of 3D points (e.g. from a laser scanner)
3D model:
surface fitting:
The University of
OntarioSurface fitting functional
CC
dsdsCE 1,)( vN
Fitting a surface into a cloud of oriented points (Lempitsky&Boykov, 2007)
ip
in
S
data fit prior
The University of
OntarioOptimal surfaces in 3DFitting a surface into a cloud of oriented points (Lempitsky&Boykov, 2007)
From 10 views
No initialization is needed
CC
dsdsCE 1,)( vN
The University of
OntarioGlobal vs. local optimization
regional potentials )(vdivf
CC
dsdpdivCE 1)()int(
v
Fitting a surface into a cloud of oriented points (Lempitsky&Boykov, 2007)
initial solution local minima global minima
The University of
OntarioSummary
Global optimization- Your solution is as good as your functional
- No need to worry about initial guess or convergence issues
- Polynomial algorithms, but many practical issues (efficient data structures, memory limitations, parallelization, dynamic applications,…)- Many useful functionals are NP hard (lots of approximation methods are developed)- New approaches allowing global optimization are introduced (including new version of level-sets)