Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano1
Projective Geometryfor Computer Vision
Raquel A. RomanoMIT Artificial Intelligence
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano2
3D Computer Vision
Classical Problem:Given a collection of 2D images,build a model of the 3D world.
Example Applications:•virtual/immersive environments•robotics & autonomous vehicles•minimally invasive surgery
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano3
Outline
1. Projective Geometry Overview2. Minimal Projective Parameters3. Projective Parameter Estimation4. Motion Boundary Detection5. Conclusion
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano4
imaging
Image Formation
3D scene 2D images
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano5
measurement
Computer Vision
3D scene model 2D images
analysis
data
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano6
scene point
optical center
image point
optical ray
optical axis
Camera Geometry:Single View
pinhole model of perspective projection
unknown internal camera parameters
y
x
y
x
c
c
y
x
f
sf
y
x
1
Z
Yy
Z
Xx
unknown depth at each point
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano7
jkjk TR
ijij TR
ikik TR
jx
kxix
X
Camera Geometry:Multiple Views
unknown rotations and translations
TR
Z
Y
X
Z
Y
X
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano8
Measured Data:Image Points and Lines
geometric constraint: optical rays intersect in 3Dprojective geometry: express constraint in terms of
measured 2D image features
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano9
Projective Camera Model
• linear model of image formation
• depth-independent expression for optical ray intersections
• multilinear relations among point and line matches
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano10
Bilinear Constraints
fundamental matrix
ijijj ijTxRx
01 i
ij
iijT
jTj ij
x
F
ARTAx x
jjj
iii
xAx
xAx1
1
iixX
0iijTj ij
xRTx x
XTRAx jx
ijTix
ijR
XiA jA
0iijTj xFx
(Longuet-Higgins ,1981, Faugeras, 1992; Hartley, 1992)
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano11
Fundamental MatrixMaps a point in one image to a line in the
other image that contains its match
jx
kxix ijF kjF
Given matching points in two views, predict the matching point in a third image.
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano12
Projective Models in Practice
• View synthesis and interpolation: point transfer function for dense point correspondences
• Self-calibration: automatic recovery of internal camera parameters from fundamental matrices
• Bundle adjustment initialization: initial rotation and translation for nonlinear Euclidean optimization
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano13
Outline
1. Projective Geometry Overview2. Minimal Projective Parameters3. Projective Parameter Estimation4. Motion Boundary Detection5. Conclusion
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano14
Practical Problem
• Few point matches between some views.
• Unstable for estimating geometric relationships.
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano15
Geometric Consistency
Pairwise geometric relations may be inconsistent.
?
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano16
Goals
• Impose algebraic geometric constraints on stationary points seen in arbitrarily many views.
• Avoid estimating too many parameters: depths, rotations, translations
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano17
ijF
ijF
ijF
ijF
ijF
ijF
ijFijF
Geometric Dependencies
• Pairwise projective geometric relations are interdependent.
• Approach: define projective dependencies and restrict solutions to be globally consistent
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano18
Projective Bilinear Parameters
ixjx
0iijTj xFx
1 iijijT
jij ARTAFx
X
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano19
imaged 3Dtranslation & rotation
Projective Bilinear Parameters
epipoles
ijhije jie
ixjx
xx ijT
i
Ti
ijjjjiij ep
qhqpeF
jiij ee
ijhepipolar collineation
0iijTj xFx
(Csurka, et.al., 1997)
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano20
Projective Parameters
ijjiij hee ,,
ijjiij hee ,,
ijjiij hee ,,
ijjiij hee ,,
ijjiij hee ,,
ijjiij hee ,,
ijjiij hee ,,
ijjiij hee ,,
• provide a complete projective model of camera configuration
But...
• set of all pairwise parameters are still redundant
• not all images have sufficient overlap
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano21
Trifocal Dependencies
• derive dependencies among three fundamental matrices
• correctly models degreesof freedom in camera configuration
• geometrically consistent parameterized model of view triplets
kie kje
ike
ije jie jke
kih kjh
ijh
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano22
Trifocal Dependencies
trifocal lines available from two fundamental matrices
• derive dependencies among three fundamental matrices
• correctly models degreesof freedom in camera configuration
• geometrically consistent parameterized model of view triplets
ktkie kje
it jtike
ije jie jke
kih kjh
ijh
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano23
Outline
1. Projective Geometry Overview2. Minimal Projective Parameters3. Projective Parameter Estimation4. Motion Boundary Detection5. Conclusion
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano24
Recovering Camera Geometry
view i view jview k
fewcorrespondences
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano25
Linear Initialization8-point Algorithm
(Hartley, 1995)
Tij fffffffff 333231232221131211f
ji
iijTj
xx
xFx,
Rewrite bilinear constraints as
where
and solve linear system
0Af ij
0f ijjjjjijijjiji yxyyyyxxxyxx 1
Minimize over all matching point pairs.
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano26
Projection to Parameter Space
Map linear estimate of fundamental matrix to projective parameter space:
},,{7 ijjiijij heep ijF },,{4 ijji
ij hγγp
• parameterization requires choice of projective basis
• basis affects shape of error surface for nonlinear optimization
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano27
Geometric Objective Function
point-to-epipolar-line distance ~ image reprojection error
iTjij
ijji ijwE xFxpxx
p7);,( 7
22
21
22
21 )()(
1
)()(
1
jT
jT
iijiijij
ijij
wxFxFxFxF
weighted residual of bilinear constraint
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano28
gamma(i,j) (h1,h2)
Error Surface Depends on Basis
(h1,h2) (h2,h3) (h1,h3)
(e1,e2) (e3,e4)
(h1,h2) (h2,h3) (h1,h3)
gamma(i,j) (h1,h2)
(e1,e2) (e3,e4)
canonical basis geometrically defined basis
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano29
Nonlinear Trifocal Estimation
1. Initialize epipolar geometry• 8-point algorithm: linear solution to
fundamental matrix for all view pairs• extract epipoles and epipolar collineations
2. 7D nonlinear minimization: bifocal parameters for view pairs (i,k) (j,k)
3. Trifocally constrained estimation for view pair (i,j)• compute trifocal lines• project parameters to trifocally constrained space• 4D nonlinear minimization for bifocal parameters
i jk
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano30
Ground Truth8-point Algorithm7-Parameter SearchTrifocal Projection4-Parameter Search
Convergenceeij erroreji
-4000 -2000 0
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano31
Ground Truth
8-Point Algorithm
7-Parameter Algorithm
4-Parameter Algorithm
Results
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano32
knossos sequenceview i view k view j
fewcorrespondences
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano33
4-Parameter Algorithm
Ground Truth
7-Parameter Algorithm
8-Point Algorithm
Results
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano34
Summary
• Imposing projective constraints on camera geometry corrects the estimation of epipolar geometry
• Resulting camera configuration for multiple cameras is globally consistent
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano35
Outline
1. Projective Geometry Overview2. Minimal Projective Parameters 3. Projective Parameter Estimation4. Motion Boundary Detection5. Conclusion
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano36
Camera and Scene Motion
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano37
Combining Intensity and Geometrytrifocal tensor
projective linear form relating a point-line-line(Spetsakis & Aloimonos, 1990; Shashua, 1994)
0),,( kjiΤ llx
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano38
Tensor Brightness Constraint
(Shashua & Hannah, 1995; Shashua & Stein, 1997)• Horn-Schunk brightness
constraint is linear in point coordinates
• Defines line in each image containing matching point
• Spatiotemporal gradient at every pixel provides test of rigid motion
u Ix + v Iy + It = 0
ax + by + c = 0
(a,b,c)T Ix
Iy
It - x0 Ix – y0 Iy
u = x - x0 v = y - y0
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano39
Motion Boundary Detection
• Partition image into windows and solve for trifocal tensor coefficients.
• Sum residual error of tensor solution.
• Only regions with rigid 3D motion have a good fit.
• High residuals indicate regions that cross a motion boundary.
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano40
Multiple Frame Flow
• Multi-frame tracks fall into separable classes
• Track points over many frames
• Robustly fit tracks to linear approximation of instantaneous planar motion
x(t) = x0+ t [Ax0 + b]
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano41
Detecting Independent Motions
Residual error of estimated motion model on all point
tracks
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano42
Complexity of Motion Model
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano43
Conclusions
When possible, use domain and task knowledge to choose model:• What type of information is needed• What aspects of the imaging conditions
are known or controlled• What types of uncertainty can be
modeled and compensated for
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano44
Future NeedsRole of learning in motion analysis:
• Supervised learning of geometric motion classes
• Data-driven model selection by flow classification
• Robust estimation of appropriate motion model
• Adaptive, time-varying estimation
Scientific Computing SeminarMay 12 , 2004
Projective Geometryfor Computer Vision
Raquel A. Romano45
END