computer vision cmput 428/615 3d modeling from images 3d modeling from images martin jagersand

Computer Visioncmput 428/615

3D Modeling from images3D Modeling from images

Martin JagersandMartin Jagersand

Pinhole camera• Central projectionCentral projection

• Principal point & aspectPrincipal point & aspect

10100

000

000

~

1

)/,/(),,(

Z

Y

X

f

f

Z

fY

fX

y

x

ZfYZfXZYX TT

1100

10

01

1

1

1

~

1

y

x

cp

cp

cyp

cxp

v

u

yy

xx

yy

xx c

px

py

The projection matrix:

cam

camyy

xx

IK

cp

f

cp

f

X0x

Xx

]|[

0100

00

00

Projective camera

• Camera rotation and translationCamera rotation and translation

• The projection matrixThe projection matrix XtXXtX TT

camcam RRR

Xtx IKRT

PIn general:

•P is a 3x4 matrix with 11 DOF

•Finite: left 3x3 matrix non-singular

•Infinite: left 3x3 matrix singular

Properties: P=[M p4]

•Center:

•Principal ray (projection direction)

0,01

0

41

dd

Cp

C

C

MM

P

3)det( mv M

• Infinite cameras where the last row of Infinite cameras where the last row of PP is is (0,0,0,1)(0,0,0,1)• Points at infinity are mapped to points at infinityPoints at infinity are mapped to points at infinity

Affine cameras

z

z

y

x

Ty

x

td

t

t

t

R

d

t

t

KP

0

0

t

k

j

i

0

j

i

td0

j

k

i{camera}

Z

X

Y{world}

x0

• ErrorError)( 0

0

xxxx

projperspaff d

Good approximation:

• small compared to d0

• point close to principal ray

X

X’

xaff

xpersp

Camera calibration

• 11 DOF => at least 6 points11 DOF => at least 6 points• Linear solutionLinear solution

– Normalization requiredNormalization required

– Minimizes algebraic errorMinimizes algebraic error

• Nonlinear solutionNonlinear solution– Minimize geometric error (pixel re-projection)Minimize geometric error (pixel re-projection)

• Radial distortionRadial distortion– Small near the center, increase towards peripherySmall near the center, increase towards periphery

1

0min

p

pA

ii P Xx known?known

...1 221 rKrKr

Application: raysets

Gortler and al.; Microsoft Lumigraph

t

v

s

u (s,t)

(u,v)

H-Y Shum, L-W He; MicrosoftConcentric mosaics

Ck

Cl C0

Li

Lj

vi

vj i j

• Projection equationProjection equation

xxii=P=PiiXX

• Resection:Resection:– xxii,X P,X Pii

Multi-view geometry - resection

Given image points and 3D points calculate camera projection matrix.

• Projection equationProjection equation

xxii=P=PiiXX

• Intersection:Intersection:– xxii,P,Pi i XX

Multi-view geometry - intersection

Given image points and camera projections in at least 2 views calculate the 3D points (structure)

• Projection Projection equationequation

xxii=P=PiiXX

• Structure from Structure from motion (SFM)motion (SFM)– xxii P Pii,, XX

Multi-view geometry - SFM

Given image points in at least 2 views calculate the 3D points (structure) and camera projection matrices (motion)

•Estimate projective structure

•Rectify the reconstruction to metric (autocalibration)

Depth from stereo

•Calibrated aligned camerasCalibrated aligned cameras

Z

X(0,0) (d,0)

Z=f

xl xrrl

rl

xx

dfZ

dXx

fX

x

fZ

)(

Disparity d

Trinocular Vision System

(Point Grey Research)

Application: depth based reprojection

3D warping, 3D warping, McMillanMcMillan

Plenoptic modeling, Plenoptic modeling, McMillan & BishopMcMillan & Bishop

Application: depth based reprojection

Layer depth images,Layer depth images, Shade et al. Shade et al.

Image based objects,Image based objects, Oliveira & Bishop Oliveira & Bishop

Affine camera factorization3D structure from many images

The affine projection equations are The affine projection equations are

1

j

j

j

yi

xi

ij

ij

Z

Y

X

P

Py

x

10001

1

j

j

j

yi

xi

ij

ij

Z

Y

X

P

P

y

x

~

~

4

4

j

j

j

yi

xi

ij

ij

yiij

xiij

Z

Y

X

P

Py

x

Py

Px

Orthographic factorization

The ortographic projection equations are The ortographic projection equations are

where where njmijiij ,...,1,,...,1,Mm P

All equations can be collected for all All equations can be collected for all ii and and jj

wherewhere

n

mmnmm

n

n

M,...,M,M,,

mmm

mmm

mmm

212

1

21

22221

11211

M

P

P

P

Pm

MPm

M ~

~m

j

j

j

jyi

xi

iij

ijij

Z

Y

X

,P

P,

y

xP

Note that P and M are resp. 2mx3 and 3xn matrices and

therefore the rank of m is at most 3

(Tomasi Kanade’92)


Factorize Factorize mm through singular value decomposition through singular value decomposition

An affine reconstruction is obtained as followsAn affine reconstruction is obtained as follows

TVUm

TVMUP ~,

~


n

mmnmm

n

n

M,...,M,M

mmm

mmmmmm

min 212

1

21

22221

11211

P

P

P

Closest rank-3 approximation yields MLE!

0~~

1~~

1~~

TT1

TT1

TT1

yi

xi

yi

yi

xi

xi

PP

PP

PP

QQ

QQ

QQ

0~~

1~~

1~~

T

T

T

yi

xi

yi

yi

xi

xi

PP

PP

PP

C

C

C

A metric reconstruction is obtained as followsA metric reconstruction is obtained as follows

Where A is computed from Where A is computed from


Factorize Factorize mm through singular value decomposition through singular value decomposition

An affine reconstruction is obtained as followsAn affine reconstruction is obtained as follows

TVUm

TVMUP ~,

~

MQMQPP~

,~ 1

0

1

1

T

T

T

yi

xi

yi

yi

xi

xi

PP

PP

PP 3 linear equations per view on

symmetric matrix C (6DOF)

Q can be obtained from C through Cholesky factorisation

and inversion


Weak perspective factorization

[D. Weinshall][D. Weinshall]

•Weak perspective cameraWeak perspective camera

•Affine ambiguityAffine ambiguity

•Metric constraintsMetric constraints

j

i

s

sM

)ˆ)(ˆ(ˆˆˆ 11 XQQMXQQMW

0ˆˆ

ˆˆˆˆ 2

ji

jjii

sQQs

ssQQssQQsTT

TTTT

Extract motion parameters Eliminate scale Compute direction of camera axis k = i x j parameterize rotation with Euler angles

Full perspective factorization

The camera equations The camera equations

for a fixed image for a fixed image ii can be written in matrix form can be written in matrix form asas

where where

njmijiijij ,...,1,,...,1,Mmλ P

MPm iii

imiii

mimiii mmm

λ,...,λ,λdiag

M,...,M,M,,...,,

21

2121

Mm

Perspective factorization

All equations can be collected for all All equations can be collected for all ii as as

wherewherePMm

mnn P

P

P

P

m

m

m

m...

,...

2

1

22

11

In these formulas In these formulas mm are known, but are known, but ii,,PP and and MM are are

unknownunknown

Observe that Observe that PMPM is a product of a 3 is a product of a 3mmx4 matrix and a x4 matrix and a

4x4xnn matrix, i.e. it is a rank 4 matrix matrix, i.e. it is a rank 4 matrix

Perspective factorization algorithm

Assume that i are known, then PM is known.

Use the singular value decomposition PM=U VT

In the noise-free case

S=diag(1,2,3,4,0, … ,0)and a reconstruction can be obtained by setting:

P=the first four columns of U.M=the first four rows of V.

Iterative perspective factorization

When i are unknown the following algorithm can be used:

1. Set ij=1 (affine approximation).

2. Factorize PM and obtain an estimate of P and M. If 5 is sufficiently small then STOP.

3. Use m, P and M to estimate i from the camera equations (linearly) mi i=PiM

4. Goto 2.

In general the algorithm minimizes the proximity measure P(,P,M)=5Note that structure and motion recovered

up to an arbitrary projective transformation

N-view geometryAffine factorization

(HZ Ch 17, 18)[Tomasi &Kanade ’92][Tomasi &Kanade ’92]

•Affine camera Affine camera

•ProjectionProjection

•nn points, points, mm views: measurement matrix views: measurement matrix

]|[ tMP M 2x3 matrix; t 2D vector

t

Z

Y

X

My

x

nmm

nm

n

M

M

W XX

xx

xx

1

1

1

111

~~

~~

W: Rank 3XMVDUW

UDVWT

nm

T

ˆˆˆ33332

Assuming isotropic zero-mean Gaussian noise, factorization achieves ML affine reconstruction.

txx ~

Projective factorizationHomogeneous coord &scale factors

[Sturm & Triggs’96][ Heyden ‘97 ][Sturm & Triggs’96][ Heyden ‘97 ]

•Measurement matrixMeasurement matrix

•Known projective depthKnown projective depth

– Projective ambiguity Projective ambiguity

• Iterative algorithmIterative algorithm– Reconstruct withReconstruct with– Reestimate depth and iterate Reestimate depth and iterate

nmm

nmn

mm

nn

P

P

W XX

xx

xx

1

1

11

1111

11

3mxn matrix

Rank 4

ij

XPVDUW

UDVWT

nm

T

ˆˆˆ44442

1ij

ij

Further Factorization work

Factorization with uncertaintyFactorization with uncertainty

Factorization for dynamic scenesFactorization for dynamic scenes

(Irani & Anandan, IJCV’02)

(Costeira and Kanade ‘94)

(Bregler et al. 2000, Brand 2001)

Sequential 3D structure from motionusing 2 and 3 view geom

• Initialize structure and motion from two viewsInitialize structure and motion from two views• For each additional viewFor each additional view

– Determine poseDetermine pose

– Refine and extend structureRefine and extend structure

• Determine correspondences robustly by jointly Determine correspondences robustly by jointly estimating matches and epipolar geometry estimating matches and epipolar geometry

Images

2 view geometry Epipolar geometry and Fundamental matrix F

The epipolar geometry

C,C’,x,x’ and X are coplanar

The epipolar plane

All points on project on l and l’

The epipolar planes

Family of planes and lines l and l’ Intersection in e and e’

The epipoles

epipoles e,e’= intersection of baseline with image plane = projection of projection center in other image= vanishing point of camera motion direction

an epipolar plane = plane containing baseline (1-D family)

an epipolar line = intersection of epipolar plane with image(always come in corresponding pairs)

Example: converging cameras

Example: motion parallel with image plane

Example: forward motion

e

e’

The fundamental matrix F

algebraic representation of epipolar geometry

l'x

we will see that this mapping is (singular) correlation (i.e. projective mapping from points to lines) represented by the fundamental matrix F


geometric derivation

xHx' π

x'e'l' FxxHe' π

mapping from 2-D to 1-D family (rank 2)

Step 1: X on a plane

Step 2: epipolar line l’


algebraic derivation

λCxPλX IPP

PP'e'F

xPP'CP'l

(note: doesn’t work for C=C’ F=0)

xP

λX

He'F RKKH 1

Alternatively can write:


correspondence condition

0Fxx'T

The fundamental matrix satisfies the condition that for any pair of corresponding points x↔x’ in the two images 0l'x'T


F is the unique 3x3 rank 2 matrix that satisfies x’TFx=0 for all x↔x’

(i) Transpose: if F is fundamental matrix for (P,P’), then FT is fundamental matrix for (P’,P)

(ii) Epipolar lines: l’=Fx & l=FTx’(iii) Epipoles: on all epipolar lines, thus e’TFx=0, x

e’TF=0, similarly Fe=0(iv) F has 7 d.o.f. , i.e. 3x3-1(homogeneous)-1(rank2)(v) F is a correlation, projective mapping from a point x to

a line l’=Fx (not a proper correlation, i.e. not invertible)

Fundamental matrix

• Algebraic representation of epipolar geometryAlgebraic representation of epipolar geometry

Step 1: X on a plane

Step 2: epipolar line l’xxe

xexel

xx

FH

H

]'[

']'['''

'

0' xx FT

F

•3x3, Rank 2, det(F)=0

•Linear sol. – 8 corr. Points (unique)

•Nonlinear sol. – 7 corr. points (3sol.)

•Very sensitive to noise & outliers

[Faugeras ’92, Hartley ’92 ]

Epipolar lines:

Epipoles:

Projection matrices:

'|']'['

]|[

0'0

''

evee

0

ee

xlxl

T

T

T

FP

IP

FF

FF

(i) Correspondence geometry: Given an image point x in the first view, how does this constrain the position of the

corresponding point x’ in the second image?

(ii) Camera geometry (motion): Given a set of corresponding image points {xi ↔x’i}, i=1,…,n, what are the cameras P and P’ for the two views?

(iii) Scene geometry (structure): Given corresponding image points xi ↔x’i and cameras P, P’, what is the position of (their pre-image) X in space?

F Relates to three questions:

Relating 3D geometry and 2D imagesThe Fundamental Matrix F

Computing F; 8 pt alg

0Fxx'T

separate known from unknown

0'''''' 333231232221131211 fyfxffyyfyxfyfxyfxxfx

0,,,,,,,,1,,,',',',',',' T333231232221131211 fffffffffyxyyyxyxyxxx

(data) (unknowns)(linear)

0Af

0f1''''''

1'''''' 111111111111

nnnnnnnnnnnn yxyyyxyxyxxx

yxyyyxyxyxxx

0

1´´´´´´

1´´´´´´

1´´´´´´

33

32

31

23

22

21

13

12

11

222222222222

111111111111

f

f

f

f

f

f

f

f

f

yxyyyyxxxyxx

yxyyyyxxxyxx

yxyyyyxxxyxx

nnnnnnnnnnnn

8-point algorithm

Solve for nontrivial solution using SVD:

0Af

TUSVA TSVUSVT Vxx

Var subst: Vxy Now Min T1,0,...,0,0 ySyHence x = last vector in V

(Also 3,4,N view geometry. HZ 15,16)• Trifocal tensor (3 view geometry)Trifocal tensor (3 view geometry)

],,[: 321 TTTT 3x3x3 tensor;

27 params. (18 indep.)

0]"[)(]'[

"]3,2,1['

xxx

lll

ii

i

T

T

TTT lines

points

• Quadrifocal tensor (4 view geometry) Quadrifocal tensor (4 view geometry) [Triggs ’95]

•Multiview tensors Multiview tensors [Hartley’95][ Hayden ‘98]

There is no additional constraint between more than 4 images. All the constraints There is no additional constraint between more than 4 images. All the constraints can be expressed using F,triliear tensor or quadrifocal tensor. can be expressed using F,triliear tensor or quadrifocal tensor.

[Hartley ’97][Torr & Zisserman ’97][ Faugeras ’97]

Using Fundamental Matrix F to compute structure and motion

eeaFeP

0IPT

x2

1

Epipolar geometry Projective calibration

012 FmmT

compatible with F

Yields correct projective camera setup(Faugeras´92,Hartley´92)

Obtain structure through triangulation

Use reprojection error for minimizationAvoid measurements in projective space

1. Compute P1 and P2

2. Triangulate 3D points

2D-2D

2D-3D 2D-3D

mimi+1

M

new view

Determine coordinates of 3D Pointscompatible with P1 and P2

Structure from images:3D Point reconstruction

PXx XP'x'

Linear triangulation

XP'x' PXx

0XP'x

0XpXp

0XpXp

0XpXp

1T2T

2T3T

1T3T

yx

y

x

2T3T

1T3T

2T3T

1T3T

p'p''p'p''pppp

A

yxyx

0AX

homogeneous

1X

)1,,,( ZYX

inhomogeneous

invariance?

e)(HX)(AH -1

algebraic error yes, constraint no (except for affine)

Linear triangulation

Alternative way of linear intersection:Alternative way of linear intersection:

•Formulate a set of linear equations explicitly Formulate a set of linear equations explicitly solving for solving for ’s’s

See our VR2003 tutorial p. 26

Reconstruction uncertainty

consider angle between rays

ObjectiveGiven two uncalibrated images compute (PM,P‘M,{XMi})(i.e. within similarity of original scene and cameras)Algorithm(i) Compute projective reconstruction (P,P‘,{Xi})

(a) Compute F from xi↔x‘i(b) Compute P,P‘ from F(c) Triangulate Xi from xi↔x‘i

(ii) Rectify reconstruction from projective to metricDirect method: compute H from control points

Stratified method:(a) Affine reconstruction: compute ∞

(b) Metric reconstruction: compute IAC

ii HXXE -1

M PHP -1M HPP ii HXXM

π0|I

H

100AH

-1 1TT ωMMAA

Summary: 2view Reconstuction

Compute Pi+1 using robust approachFind additional matches using predicted projectionExtend, correct and refine reconstruction

2D-2D

2D-3D 2D-3D

mimi+1

M

new view

Determine pose towards existing structure

Non-sequential image collections

4.8im/pt64 images

3792

po

ints

Problem:Features are lost and reinitialized as new features

Solution:Match with other close views

For every view iExtract featuresCompute two view geometry i-1/i and matches Compute pose using robust algorithmRefine existing structureInitialize new structure

Relating to more views

Problem: find close views in projective frame

For every view For every view iiExtract featuresExtract featuresCompute two view geometry Compute two view geometry ii-1/-1/ii and matches and matches Compute pose using robust algorithmCompute pose using robust algorithmFor all For all closeclose views views kk

Compute two view geometry Compute two view geometry kk//ii and matches and matchesInfer new 2D-3D matches and add to listInfer new 2D-3D matches and add to list

Refine pose using all 2D-3D matchesRefine pose using all 2D-3D matchesRefine existing structureRefine existing structureInitialize new structureInitialize new structure

Determining close views

• If viewpoints are If viewpoints are closeclose then most image changes then most image changes can be modelled through a can be modelled through a planar homographyplanar homography

• Qualitative distance measureQualitative distance measure is obtained by is obtained by looking at the looking at the residual errorresidual error on the on the best possible best possible planar homographyplanar homography

Distance = m´,mmedian min HD

9.8im/pt

4.8im/pt

64 images

64 images

3792

po

ints

2170

po

ints

Non-sequential image collections (2)

Refining structure and motion

• Minimize reprojection errorMinimize reprojection error

– Maximum Likelyhood Estimation Maximum Likelyhood Estimation (if (if error zero-mean Gaussian noise)error zero-mean Gaussian noise)

– Huge problem but can be solved efficientlyHuge problem but can be solved efficiently(Bundle adjustment)(Bundle adjustment)

m

k

n

iikD

ik 1 1

2

kiM̂,P̂

M̂P̂,mmin

Refining a captured model:Bundle adjustment

• Refine structure XRefine structure Xjj and motion P and motion Pii

• Minimize geometric errorMinimize geometric error• ML solution, assuming noise is GaussianML solution, assuming noise is Gaussian• Tolerant to missing dataTolerant to missing data

ji

ijj

iPd,

2),ˆˆ(min xX

Projective ambiguity andself-calibration

Given an uncalibrated image sequence with corresponding point it is possible to reconstruct the object up to an unknown projective transformation

• Autocalibration (self-calibration): Determine a projective transformation T that upgrades the projective reconstruction to a metric one.

T

´M´M))((Mm 1 PTPTP

Remember:Stratification of geometry

15 DOF 12 DOFplane at infinity

parallelism

More general

More structure

Projective Affine Metric

7 DOFabsolute conicangles, rel.dist.

Constraints ?

• Scene constraintsScene constraints– Parallellism, vanishing points, horizon, ...Parallellism, vanishing points, horizon, ...– Distances, positions, angles, ...Distances, positions, angles, ...

Unknown scene no constraints

• Camera extrinsics constraints–Pose, orientation, ...

Unknown camera motion no constraints • Camera intrinsics constraints

–Focal length, principal point, aspect ratio & skew

Perspective camera model too general some constraints

Euclidean projection matrix

tRRKP TT

1yy

xx

uf

usf

K

Factorization of Euclidean projection matrix

Intrinsics:

Extrinsics: t,R

Note: every projection matrix can be factorized,

but only meaningful for euclidean projection matrices

(camera geometry)

(camera motion)

Constraints on intrinsic parameters

Constant Constant e.g. fixed camera:e.g. fixed camera:

KnownKnowne.g. rectangular pixels:e.g. rectangular pixels:

square pixels:square pixels: principal point known:principal point known:

21 KK

0s

1yy

xx

uf

usf

K

0, sff yx

2,

2,

hwuu yx

Self-calibration

Upgrade from Upgrade from projectiveprojective structure to structure to metricmetric structure using structure using constraintsconstraints onon intrinsicintrinsic camera parameters camera parameters– Constant intrinsicsConstant intrinsics

– Some known intrinsics, others varyingSome known intrinsics, others varying

– Constraints on intrincs and restricted motionConstraints on intrincs and restricted motion(e.g. pure translation, pure rotation, planar motion)(e.g. pure translation, pure rotation, planar motion)

(Faugeras et al. ECCV´92, Hartley´93,

Triggs´97, Pollefeys et al. PAMI´98, ...)

(Heyden&Astrom CVPR´97, Pollefeys et al. ICCV´98,...)

(Moons et al.´94, Hartley ´94, Armstrong ECCV´96, ...)

A counting argument

• To go from projective (15DOF) to metric (7DOF) at least 8 To go from projective (15DOF) to metric (7DOF) at least 8 constraints are neededconstraints are needed

• Minimal sequence length should satisfyMinimal sequence length should satisfy

• Independent of algorithmIndependent of algorithm• Assumes general motion (i.e. not critical)Assumes general motion (i.e. not critical)

8#1# fixednknownn

Conics•Conic: Conic:

– Euclidean geometry: hyperbola, ellipse, parabola & degenerateEuclidean geometry: hyperbola, ellipse, parabola & degenerate

– Projective geometry: equivalent under projective transformProjective geometry: equivalent under projective transform

– Defined by 5 pointsDefined by 5 points

•TangentTangent

•Dual conic C*Dual conic C*

0

022

xx C

feydxcybxyaxT

fed

ecb

dba

C

2/2/

2/2/

2/2/

xl C

0* ll CT

Quadrics

Quadrics: Quadrics: QQ 4x4 symmetric matrix 4x4 symmetric matrix

9 DOF (defined by 9 points in general pose)9 DOF (defined by 9 points in general pose)

•Dual: Dual: Q*Q*Planes tangent to the quadricPlanes tangent to the quadric

0XX QT

0* ππ QT

Summary:Conics & Quadrics

0mmT C 0ll *T C

1* CC

conics

0MMT Q 0*T Q1* QQ

quadrics

1T´~ CHHCC T*** ´~ HHCCC T´ TTQ~QQ ***

1T´~ QTTQQ

transformations

T** ~ PPQC

projection

The absolute conic

• Absolute conic is a imaginary circle onAbsolute conic is a imaginary circle on• The absolute dual quadric (rim quadric)The absolute dual quadric (rim quadric)

• In a metric frameIn a metric frame

π

0),,(),,(

0

)1,0,0,0(

321321

4

23

22

21

TxxxIxxx

x

xxx

π

*

0

0*

*

T

T

I

0

0

Note: is the nullspace of

*

On : π

*

Fixed under similarity transf.

Self-calibration

•Theoretically formulated by Theoretically formulated by [Faugeras ’92][Faugeras ’92]

•2 basic approaches2 basic approaches– Stratified: recover Stratified: recover – Direct: recover Direct: recover [Triggs’97][Triggs’97]

•Constraints:Constraints:– Camera internal constraintsCamera internal constraints

–Constant parameters Constant parameters [Hartley’94][ Mohr’93][Hartley’94][ Mohr’93]

–Known skew and aspect ratioKnown skew and aspect ratio [Hayden&[Hayden&ÅÅstrströöm’98][Pollefeys’98]m’98][Pollefeys’98]

– Scene constraints (angles, ratios of length)Scene constraints (angles, ratios of length)

•Choice of H:Choice of H: Knowing camera Knowing camera KK and and

π

*

TTT K

KH )1,(,

1p

p

0

π

Absolute Dual Quadric and Self-calibration

Eliminate extrinsics from equationEliminate extrinsics from equation

Equivalent to projection of dual quadric

))(Ω)((Ω *1* TTTTT PTTTPTPPKK

Abs.Dual Quadric also exists in projective world

T´Ω´´ * PP Transforming world so thatreduces ambiguity to metric

** ΩΩ´

*

*

projection

constraints

Absolute conic = calibration object which is always present but can only be observed through constraints on the intrinsics

Tii

Tiii Ωω KKPP

Absolute Dual Quadric and Self-calibration

Projection equation:Projection equation:

Translate constraints on Translate constraints on K K through projection equationthrough projection equation to to constraints on constraints on **

KRd0d]C

~|KR[IPXx

mapping between ∞ to an image is given by the planar homogaphy x=Hd, with H=KR

image of the absolute conic (IAC) = I

1-T-1T KKKKω 1TCHHC

(i) IAC depends only on intrinsics(ii) angle between two rays(iii) DIAC=*=KKT

(iv) K (cholesky factorisation)(v) image of circular points

2T

21T

1

2T

1

ωxxωxx

ωxxcos

Image of the absolute conic

HZ 7.5.1:

Constraints on *

1

ω 22

222

*

yx

yyyyxy

xyxyxx

cc

ccfccsf

cccsfcsf

Zero skewZero skew quadraticquadratic mm

Principal pointPrincipal point linearlinear 22mm

Zero skew (& p.p.)Zero skew (& p.p.) linearlinear mm

Fixed aspect ratio (& Fixed aspect ratio (& p.p.& Skew)p.p.& Skew)

quadraticquadratic m-1m-1

Known aspect ratio Known aspect ratio (& p.p.& Skew)(& p.p.& Skew)

linearlinear mm

Focal length Focal length

(& p.p. & Skew)(& p.p. & Skew)

linearlinear mm

*23

*13

*33

*12 ωωωω

0ωω *23

*13

0ω*12

*11

*22

*22

*11 ω'ωω'ω

*22

*11 ωω

*11

*33 ωω

condition constraint type #constraints

Summary: Self calibration based on the IADC

•Calibrated cameraCalibrated camera–Dual absolute quadric (DAC)Dual absolute quadric (DAC)

–Dual image of the absolute conic (DIAC)Dual image of the absolute conic (DIAC)

•Projective cameraProjective camera–DACDAC

–DIACDIAC

•AutocalibrationAutocalibration–Determine based on constraints onDetermine based on constraints on

–Decompose Decompose

TKK

diagI

*

)0,1,1,1(~

Tii

iTii

T

KKPQP

HIHQ

**

* ~

* i*THIHQ

~*

i

j

Ci

Cj

Illustration of self-calibration

Projective Affine Metric

Degenerate configurations

• Pure translation: Pure translation: affine transformation (5 DOF)affine transformation (5 DOF)

• Pure rotation: Pure rotation: arbitrary pose for (3 DOF) arbitrary pose for (3 DOF)

• Planar motion: Planar motion: scaling axis perpendicular to plane scaling axis perpendicular to plane (1DOF)(1DOF)

• Orbital motion: Orbital motion: projective distortion along rotation axis projective distortion along rotation axis (2DOF) (2DOF)

π

Not unique solution !Not unique solution !

A complete modeling systemprojective

Sequence of frames scene structureSequence of frames scene structure

1.1. Get corresponding points (tracking).Get corresponding points (tracking).

2.2. 2,3 view geometry: 2,3 view geometry: compute F,T between consecutive frames compute F,T between consecutive frames (recompute correspondences).(recompute correspondences).

3.3. Initial reconstruction: Initial reconstruction: get an initial structure from a get an initial structure from a subsequence with big baseline (trilinear tensor, factorization …) subsequence with big baseline (trilinear tensor, factorization …) and bind more frames/points using resection/intersection.and bind more frames/points using resection/intersection.

4.4. Self-calibration.Self-calibration.

5.5. Bundle adjustment.Bundle adjustment.

A complete modeling systemaffine

Sequence of frames scene structureSequence of frames scene structure

1.1. Get corresponding points (tracking).Get corresponding points (tracking).

2.2. Affine factorization. (This already computes ML Affine factorization. (This already computes ML estimate over all frames so no need for bundle estimate over all frames so no need for bundle adjustment for simple scenes.adjustment for simple scenes.

3.3. Self-calibration.Self-calibration.

4.4. If several model segments: Merge, bundle adjust.If several model segments: Merge, bundle adjust.

Examples – modeling with dynamic texture

Cobzas,Yerex,JagersandCobzas,Yerex,Jagersand

Examples: geometric modeling

Debevec and Taylor:Debevec and Taylor: Façade Façade


Pollefeys: Pollefeys: Arenberg CastleArenberg Castle


INRIA –INRIA –VISIRE projectVISIRE project


CIP Prague –CIP Prague –Projective Reconstruction Based on Cake ConfigurationProjective Reconstruction Based on Cake Configuration

Reading: FP Chapter 11.

• The Stereopsis Problem: Fusion and Reconstruction• Human Stereopsis and Random Dot Stereograms• Cooperative Algorithms• Correlation-Based Fusion• Multi-Scale Edge Matching• Dynamic Programming• Using Three or More Cameras

Dense stereo

•Go back to original images, do dense matching.Go back to original images, do dense matching.

•Try to get dense depth mapsTry to get dense depth maps

Stereo image rectification

Stereo image rectification

• reproject image planes onto a commonreproject image planes onto a common• plane parallel to the line between optical centersplane parallel to the line between optical centers• pixel motion is horizontal after this transformationpixel motion is horizontal after this transformation• two homographies (3x3 transform), one for each input two homographies (3x3 transform), one for each input

image reprojectionimage reprojectionC. Loop and Z. Zhang. C. Loop and Z. Zhang.

Computing Rectifying Homographies for Stereo Vision. IEEE Conf. . IEEE Conf. Computer Vision and Pattern Recognition, 1999Computer Vision and Pattern Recognition, 1999..

http://research.microsoft.com/~zhang/Papers/TR99-21.pdf

All epipolar lines are parallel in the rectified image plane.

Rectification

simplify stereo matching by warping the images

Apply projective transformation so that epipolar linescorrespond to horizontal scanlines

e

e

map epipole e to (1,0,0)

try to minimize image distortion

problem when epipole in (or close to) the image

He

0

0

1

Image rectification throughhomography warp

ExampleUnrectified

Rectified

Depth from disparity

f

x x’

baseline

z

C C’

X

f

Stereo matching algorithms•Match Pixels in Conjugate Epipolar LinesMatch Pixels in Conjugate Epipolar Lines

– Assume brightness constancyAssume brightness constancy

– This is a tough problemThis is a tough problem

– Numerous approachesNumerous approaches–A good survey and evaluation: A good survey and evaluation:

http://www.middlebury.edu/stereo/

http://www.middlebury.edu/stereo/

Your basic stereo algorithm

For each epipolar line

For each pixel in the left image• compare with every pixel on same epipolar line in right image

• pick pixel with minimum match cost

Improvement: match windows• This should look familar...

Stereo as energy minimization

• Find disparities d that minimize an energy function

• Simple pixel / window matching

SSD distance between windows I(x, y) and J(x, y + d(x,y))=


I(x, y) J(x, y)

y = 141

C(x, y, d); the disparity space image (DSI)x

d


y = 141

x

d

Simple pixel / window matching: choose the minimum of each column in the DSI independently:

Matching windowsSimilarity MeasureSimilarity Measure FormulaFormula

Sum of Absolute Differences Sum of Absolute Differences (SAD)(SAD)

Sum of Squared Differences (SSD)Sum of Squared Differences (SSD)

Zero-mean SADZero-mean SAD

Locally scaled SADLocally scaled SAD

Normalized Cross Correlation Normalized Cross Correlation (NCC)(NCC)

http://siddhantahuja.wordpress.com/category/stereo-vision/ SAD SSD NCC Ground truth

http://siddhantahuja.wordpress.com/category/stereo-vision/

Stereo matching

Optimal path(dynamic programming )

Similarity measure(SSD or NCC)

Constraints• epipolar

• ordering

• uniqueness

• disparity limit

• disparity gradient limit

Trade-off

• Matching cost (data)

• Discontinuities (prior)

(Cox et al. CVGIP’96; Koch’96; Falkenhagen´97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

Disparity map

image I(x,y) image I´(x´,y´)Disparity map D(x,y)

(x´,y´)=(x+D(x,y),y)

Hierarchical stereo matchingD

ow

nsam

plin

g

(Gau

ssia

n p

yra

mid

)

Dis

pari

ty p

rop

ag

ati

on

Allows faster computation

Deals with large disparity ranges

(Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

Example: reconstruct image from neighboring images

computer vision cmput 428/615 3d modeling from images 3d modeling from images martin jagersand

Documents