projective geometry for computer vision

Scientific Computing SeminarMay 12 , 2004

Projective Geometryfor Computer Vision

Raquel A. Romano1


Raquel A. RomanoMIT Artificial Intelligence

[email protected]



Raquel A. Romano2

3D Computer Vision

Classical Problem:Given a collection of 2D images,build a model of the 3D world.

Example Applications:•virtual/immersive environments•robotics & autonomous vehicles•minimally invasive surgery



Raquel A. Romano3

Outline

1. Projective Geometry Overview2. Minimal Projective Parameters3. Projective Parameter Estimation4. Motion Boundary Detection5. Conclusion



Raquel A. Romano4

imaging

Image Formation

3D scene 2D images



Raquel A. Romano5

measurement

Computer Vision

3D scene model 2D images

analysis

data



Raquel A. Romano6

scene point

optical center

image point

optical ray

optical axis

Camera Geometry:Single View

pinhole model of perspective projection

unknown internal camera parameters

y

x

y

x

c

c

y

x

f

sf

y

x

1

Z

Yy

Z

Xx

unknown depth at each point



Raquel A. Romano7

jkjk TR

ijij TR

ikik TR

jx

kxix

X

Camera Geometry:Multiple Views

unknown rotations and translations

TR

Z

Y

X

Z

Y

X



Raquel A. Romano8

Measured Data:Image Points and Lines

geometric constraint: optical rays intersect in 3Dprojective geometry: express constraint in terms of

measured 2D image features



Raquel A. Romano9

Projective Camera Model

• linear model of image formation

• depth-independent expression for optical ray intersections

• multilinear relations among point and line matches



Raquel A. Romano10

Bilinear Constraints

fundamental matrix

ijijj ijTxRx

01 i

ij

iijT

jTj ij

x

F

ARTAx x

jjj

iii

xAx

xAx1

1

iixX

0iijTj ij

xRTx x

XTRAx jx

ijTix

ijR

XiA jA

0iijTj xFx

(Longuet-Higgins ,1981, Faugeras, 1992; Hartley, 1992)



Raquel A. Romano11

Fundamental MatrixMaps a point in one image to a line in the

other image that contains its match

jx

kxix ijF kjF

Given matching points in two views, predict the matching point in a third image.



Raquel A. Romano12

Projective Models in Practice

• View synthesis and interpolation: point transfer function for dense point correspondences

• Self-calibration: automatic recovery of internal camera parameters from fundamental matrices

• Bundle adjustment initialization: initial rotation and translation for nonlinear Euclidean optimization



Raquel A. Romano13

Outline




Raquel A. Romano14

Practical Problem

• Few point matches between some views.

• Unstable for estimating geometric relationships.



Raquel A. Romano15

Geometric Consistency

Pairwise geometric relations may be inconsistent.

?



Raquel A. Romano16

Goals

• Impose algebraic geometric constraints on stationary points seen in arbitrarily many views.

• Avoid estimating too many parameters: depths, rotations, translations



Raquel A. Romano17

ijF

ijF

ijF

ijF

ijF

ijF

ijFijF

Geometric Dependencies

• Pairwise projective geometric relations are interdependent.

• Approach: define projective dependencies and restrict solutions to be globally consistent



Raquel A. Romano18

Projective Bilinear Parameters

ixjx

0iijTj xFx

1 iijijT

jij ARTAFx

X



Raquel A. Romano19

imaged 3Dtranslation & rotation

Projective Bilinear Parameters

epipoles

ijhije jie

ixjx

xx ijT

i

Ti

ijjjjiij ep

qhqpeF

jiij ee

ijhepipolar collineation

0iijTj xFx

(Csurka, et.al., 1997)



Raquel A. Romano20

Projective Parameters

ijjiij hee ,,

ijjiij hee ,,

ijjiij hee ,,

ijjiij hee ,,

ijjiij hee ,,

ijjiij hee ,,

ijjiij hee ,,

ijjiij hee ,,

• provide a complete projective model of camera configuration

But...

• set of all pairwise parameters are still redundant

• not all images have sufficient overlap



Raquel A. Romano21

Trifocal Dependencies

• derive dependencies among three fundamental matrices

• correctly models degreesof freedom in camera configuration

• geometrically consistent parameterized model of view triplets

kie kje

ike

ije jie jke

kih kjh

ijh



Raquel A. Romano22

Trifocal Dependencies

trifocal lines available from two fundamental matrices

• derive dependencies among three fundamental matrices

• correctly models degreesof freedom in camera configuration

• geometrically consistent parameterized model of view triplets

ktkie kje

it jtike

ije jie jke

kih kjh

ijh



Raquel A. Romano23

Outline




Raquel A. Romano24

Recovering Camera Geometry

view i view jview k

fewcorrespondences



Raquel A. Romano25

Linear Initialization8-point Algorithm

(Hartley, 1995)

Tij fffffffff 333231232221131211f

ji

iijTj

xx

xFx,

Rewrite bilinear constraints as

where

and solve linear system

0Af ij

0f ijjjjjijijjiji yxyyyyxxxyxx 1

Minimize over all matching point pairs.



Raquel A. Romano26

Projection to Parameter Space

Map linear estimate of fundamental matrix to projective parameter space:

},,{7 ijjiijij heep ijF },,{4 ijji

ij hγγp

• parameterization requires choice of projective basis

• basis affects shape of error surface for nonlinear optimization



Raquel A. Romano27

Geometric Objective Function

point-to-epipolar-line distance ~ image reprojection error

iTjij

ijji ijwE xFxpxx

p7);,( 7

22

21

22

21 )()(

1

)()(

1

jT

jT

iijiijij

ijij

wxFxFxFxF

weighted residual of bilinear constraint



Raquel A. Romano28

gamma(i,j) (h1,h2)

Error Surface Depends on Basis

(h1,h2) (h2,h3) (h1,h3)

(e1,e2) (e3,e4)

(h1,h2) (h2,h3) (h1,h3)

gamma(i,j) (h1,h2)

(e1,e2) (e3,e4)

canonical basis geometrically defined basis



Raquel A. Romano29

Nonlinear Trifocal Estimation

1. Initialize epipolar geometry• 8-point algorithm: linear solution to

fundamental matrix for all view pairs• extract epipoles and epipolar collineations

2. 7D nonlinear minimization: bifocal parameters for view pairs (i,k) (j,k)

3. Trifocally constrained estimation for view pair (i,j)• compute trifocal lines• project parameters to trifocally constrained space• 4D nonlinear minimization for bifocal parameters

i jk



Raquel A. Romano30

Ground Truth8-point Algorithm7-Parameter SearchTrifocal Projection4-Parameter Search

Convergenceeij erroreji

-4000 -2000 0



Raquel A. Romano31

Ground Truth

8-Point Algorithm

7-Parameter Algorithm


Results



Raquel A. Romano32

knossos sequenceview i view k view j

fewcorrespondences



Raquel A. Romano33


Ground Truth


8-Point Algorithm

Results



Raquel A. Romano34

Summary

• Imposing projective constraints on camera geometry corrects the estimation of epipolar geometry

• Resulting camera configuration for multiple cameras is globally consistent



Raquel A. Romano35

Outline

1. Projective Geometry Overview2. Minimal Projective Parameters 3. Projective Parameter Estimation4. Motion Boundary Detection5. Conclusion



Raquel A. Romano36

Camera and Scene Motion



Raquel A. Romano37

Combining Intensity and Geometrytrifocal tensor

projective linear form relating a point-line-line(Spetsakis & Aloimonos, 1990; Shashua, 1994)

0),,( kjiΤ llx



Raquel A. Romano38

Tensor Brightness Constraint

(Shashua & Hannah, 1995; Shashua & Stein, 1997)• Horn-Schunk brightness

constraint is linear in point coordinates

• Defines line in each image containing matching point

• Spatiotemporal gradient at every pixel provides test of rigid motion

u Ix + v Iy + It = 0

ax + by + c = 0

(a,b,c)T Ix

Iy

It - x0 Ix – y0 Iy

u = x - x0 v = y - y0



Raquel A. Romano39

Motion Boundary Detection

• Partition image into windows and solve for trifocal tensor coefficients.

• Sum residual error of tensor solution.

• Only regions with rigid 3D motion have a good fit.

• High residuals indicate regions that cross a motion boundary.



Raquel A. Romano40

Multiple Frame Flow

• Multi-frame tracks fall into separable classes

• Track points over many frames

• Robustly fit tracks to linear approximation of instantaneous planar motion

x(t) = x0+ t [Ax0 + b]



Raquel A. Romano41

Detecting Independent Motions

Residual error of estimated motion model on all point

tracks



Raquel A. Romano42

Complexity of Motion Model



Raquel A. Romano43

Conclusions

When possible, use domain and task knowledge to choose model:• What type of information is needed• What aspects of the imaging conditions

are known or controlled• What types of uncertainty can be

modeled and compensated for



Raquel A. Romano44

Future NeedsRole of learning in motion analysis:

• Supervised learning of geometric motion classes

• Data-driven model selection by flow classification

• Robust estimation of appropriate motion model

• Adaptive, time-varying estimation



Raquel A. Romano45

END

projective geometry for computer vision

Documents