computer vision no. 7: stereo part 2. today’s topics u projective geometry u camera projection...

Computer Vision

No. 7: Stereo part 2

Today’s topics

Projective geometry Camera Projection Matrix F Matrix Multi View Shape Reconstruction

Elementary Projective Geometry

Topics

1. Projective Plane

2. Projective Space

3. Affine Space

4. Euclidean Space

Projective Plane　 (2D projective space)

In the Euclidean plane ),(: yxp

In the Projective plane )1,,(: yxp

Simply add a third coordinate of 1 at the end

Homogeneous coordinate

! Scaling is NOT important.

),,()1,,( yxyx Therefore the point (0, 0, 0) is disallowed in the projective plane.

)0,,( yx : Ideal points or Points at infinity

Description in Projective Plane

A line in the projective plane is described as…

A line through the point (x, y)…

0 cbyax

0),,()1,,( cbayx

),,( cbal

A line through a point

0 pllp tt

Duality in Projective Plane

A point and a line ),,( cbal ),,( wyxp

The line through point and1p 2p 21 ppl

If point , and are on the same line1p 2p 3p

0det 321 ppp

The cross point of line and1l 2l 21 llp

If line , and are passing through the same point1l 2l 3l

0det 321 lll

Conics and Dual Conics

For symmetric 3 x 3 matrix …

All points which satisfy ….p

0pp tConic

Dual Conic

Therefore, for the matrix )( 1*

All lines which satisfy ….l

0* ll t

Transformation in Projective Plane

In the Projective plane, by 3 x 3 non-singular matrix T

A point is transformed intop p pTp

A line is transformed intol l

lTlplTpTTlpl ttttt )(0 1

A conic is…

11 )(0

pTTppTTTTpppt

tttttt

Similarly a dual conic is…* tTT **

In the Euclidean space ),,(: ZYXP

In the Projective space )1,,,(: ZYXP

Homogeneous coordinate

! ),,,()1,,,( ZYXZYX

)0,,,( ZYX : Ideal points, Points at infinity

A plane in the projective space is described as…

),,,( DCBAL

A plane through a point 0 PLLP ttL ),,,( WZYXP

Projective Space (3D Projective space)

Quadrics and Dual Quadrics

For symmetric 4 x 4 matrix …All points which satisfy ….P

0PP tQuadric

Dual Quadric

Therefore, for the matrix )( 1*

All planes which satisfy ….L

0* LLt

Similarly a dual quadric is…* tTT **

A quadric is transformed by T into 1 TT t

N-Dimensional Affine Space

Among transformations T in the N-d projective space

0,0det,matrixsingular -non: 0

bAT aat

Affine Transformation

In N-D affine space… )0,( p

: point at infinity

The group of the points at infinity )1,0(

: plane at infinity

By an affine transformation

a point at infinity

N-Dimensional Euclidean Space

Among N-d affine transformations

0,,matrix orthogonal: 0

cAANNA

Euclidean Transformation

a group of points at infinity, which satisfy)}0,{( ep

By an Euclidean transformation

e pppAAppp

N=3: Absolute conic

N=2: Absolute point

3-Dimensional Euclidean Space

1,33: 0

By an Euclidean transformation

Absolute Dual Quadric

The absolute dual quadric is invariant under Euclidean transformation!

Camera Projection

Topics

1. Pinhole Camera Model

2. Camera Parameters

3. Lens Distortion

Pinhole Camera

Pinhole Camera Modelgeometry

(X, Y, Z)

Image plane

(x, y)

Xyx ,),(

Perspective projection

View point

(Optical center)

(sX, sY, sZ)

Pinhole Camera ModelThe projected point on the image plane (z = -1)

The Rotation matrix of camera pose

The camera position t

In Projective Geometry

tIRtPR

Intrinsic parameters

yAn ideal image on the Image plane

θ An actual picture

(x, y)

(u, v)

the Principal point

Intrinsic parameters

yAn ideal image on the Image plane

θ An actual picture

(x, y)

(u, v)

the Principal point

f : Focal Length

a : aspect ratio

Camera Projection Matrix

PtIARpAv

Transformation of a point in the 3D world into a 2D picture…

tIARP tc

,Camera Matrix (3 x 4)

Camera calibration

11 parameters: rotation 3 + translation 3 + intrinsic 5

Estimate of these 11 parameters (= camera matrix)

Lens distortion

Short focal length Long focal length

Removal of lens distortion

Lens DistortionZ.Zhang (‘00) PAMI

R. Sagawa (‘05) IROS Non-parametric

F Matrix

Topics

1. Epipolar Geometry

2. E & F Matrix

3. Homography

4. Multi View Geometry

Correspondence Problem

Given two images, we want to solve a problem……

For a point in image 1, decide which point in image 2 it corresponds to.

plane 1

plane 2

Eye 1Eye 2

Epipolar Geometry

3D point Epipolar plane

EpipoleEpipole

Epipolar line

Baseline

Essential Matrix

23 2pRt

021 pRtp t

Essential matrix : E

Essential & Fundamental Matrix

021 pEp t Image planes

Pictures

Fundamental matrix : F

Image 1 Image 2

Fundamental Matrix

image 1image 2

constraint

t Depends on the camera configuration!!

Fundamental Matrix PtIRApAu

PtIRApAu

222222

111111

tRAPRAPtIRApAu

PAPIIApAu

222222

tRAuARAu tt 21

tuRAuA

1 are on the same plane!Three vectors

uRAtAuuRAtuA ttt

Epipolar Constraint

FvuuFu t

tt lFvuFu 1111 1,,

Epipolar Constraint

FvuuFu t

Property of F Matrix

AARtAF tt

at most Rank 2

0det F

There are some points which satisfye

Null Space of F Matrix (epipole)

1, uF for 021 eFu t

All epipolar lines in image 2 pass through the point 2e

Null Space of F Matrix (epipole)

Similarly, 2ufor 021 uFe t

21,ee : Epipoles

All epipolar lines in image 1 pass through the point 1e

Epipoles

Epipoles:• intersections of baseline with image planes• projection of the optical center in another image• the vanishing points of camera motion direction

Examples

Computing F Matrix (8-point algorithm)021 uFu t

333231

232221

131211

112212122121

vuvvvvuuuvuu

Computing F Matrix

Epipolar pencil by linear solution (due to noise and error)

Computing F MatrixSingular value decomposition (SVD)

2rank F Without noise, σ3 must be 0

modification

Computing F Matrix

Homography

In Camera 1

Camera 1

Camera 2

PAPIAu

1Pnd t

The distance between camera 1 and the plane

In Camera 2

121222 Pd

ntIRAtPRAPtIRAu

Homography and F Matrix

Camera 1

Camera 2

122 uAd

ntIRAu

12 uHu

ttAtAAd

ntIRARAtAFH t

0)( 111

AttAAtAAtAFHFH tttttt

Planer homography mosaicing

camera

Image plane’

Image rectification

Camera 1 Camera 2

Rectification

Fusiello (MVA 2000) Pollefeys

Constraints of 3 images

Line – line – line

Tensor TrifocaliT

3 x 3 matrix

Constraints of 3 imagesL

Line – line – line

ijkkj lTll

Tensor description

Point – line – line

1u 0321 ijkkji Tllu

Constraints of 3 images

Point – point – line

pkkjprji Tluu 0321

pqkqskjprji Tuuu 0321

Point – point – point

otherwise

123 ofn permutatio oddan is

123 ofn permutatioeven an is

2-Views Geometry

matrix 43 scalar,

M : (3 x 2) x (4 + 2) matrix

0detif M 0

2-Views Geometry

matrix 43 scalar,

M : (3 x 2) x (4 + 2) matrix

0det M 021 ijji Fuu

In tensor form

3-Views Geometry

matrix 43 scalar,

M : (3 x 3) x (4 + 3) matrix

0det M rsi

pqkqskjprji Tuuu 0321

4-Views Geometry

0det M

wxyzpqrslszlkrykjqxjipwi Quuuu 04321

tensorlquadrifocapqrsQ

Camera Calibration

Topics

1. 3D Ref.

2. 2D Ref.

3. 1D Ref.

4. Self Calibration (Kruppa equations)

The meaning of camera calibration

tIARP tc

Absolute Dual Quadric

ttcEc AARA

IItIARPP

Camera Calibration = to estimate the image of the absolute quadric

Camera Calibration (3D Ref.)

333231

232221

131211

t PPAA **

tIARP tc

Homography 　（ 2D-2D mapping ）

333231

232221

131211

(x, y)

(u, v)

Homography (3 x 3)

the origin

Without loss of generality…

djiAdji

AMPH c

Z.Zhang (‘00) PAMI

11 333231

232221

131211

tttt AA

Camera Calibration (2D Ref. Results)

Camera Calibration (1D Ref.)Z.Zhang (‘04) PAMI

213 )1( PPP

On the line in 3D

2133 )1( uuPPu c

22111 ,0 PAuPAPIAu

1,1)1(

122 uuAAuuuAuAPPL tt

:const. 1P :fix

Camera Calibration (1D Ref. Results)

Self Calibration

In most cases, there are NOT any reference objects…

be able to calibrate the camera by unknown scenes

Homography at infinity

Camera 1

Camera 2

Considering P on the infinity plane …

)0,,,()0,( ZYXPP t

0222 ZYXPP t

and on the absolute quadric…

PRPtIRp tt

0 PPPRRPpp tttt

In camera 2 coordinate system

The image of the absolute quadric

Camera 1

Camera 2

The image of the absolute quadric at the infinity plane

uAAuPPt

111 ueue t

11 ll t

The Kruppa equations

Camera 1

Camera 2

22 ll t

tt lFu 21

21 uFFu tt

tt FFee *21

2Given the F Matrix…

References

Multiple View GeometryHartley and Zisserman, Cambridge

The Geometry of Multiple ImagesFaugeras and Luong, MIT

Three-Dimensional Computer VisionFaugeras, MIT

Algebraic Projective GeometrySemple and Kneebone, Oxford

Multiple View Shape Reconstruction

Topics

•Basic Principle•Surface Approach

Multi-baseline stereo•Volumetric Approach

Volume intersection method•Combinatorial Approach

Photo hull

•Extended Algorithm•Level Set Approach•Graph-cut Approach

Basic Principle #1

Surface Approach

Surface Approach Try to reconstruct a surface of an object

– Find depth d that minimizes some matching cost function F (d) defined on N views

Camera 1

Camera 2Camera N

Matching Cost Function

),(),()(

fxIyxIdF

)),(()),((

)),(),,(()(

1, ybdf

xIVaryxIVar

xIyxICovdF

(b) SSSD (sum of sum. of squared difference)

(a) SSAD (sum. of sum of absolute difference)

fxIyxIdF ),(),()( 1

(c) SNCC (sum of normalized cross correlation)

Camera 1 (reference) Camera 2 Camera N

Epipolar line

Why Multiple Views? Binocular stereo is sometimes ambiguous

e.x. F(d) = SAD (sum. of absolute difference)

fxIyxI ),(),( 21

Camera 1 Camera 2

Which is the right match?

Multi-baseline Stereo

T. Kanade, M. Okutomi, and T. Nakahara, "A Multiple-Baseline Stereo Method," Proc. ARPA Image Understanding Workshop, pp. 409-426, 1992.

Camera 1(reference)

Camera 2

Camera 3 Camera N

Unique minima

Video-rate Stereo Vision Hardware implementation is easy

S. Kimura, T. Kanade, H. Kano, A. Yoshida, E. Kawamura, K. Oda, “CMU video-rate stereo machine", Proc. of Mobile Mapping Symposium, 1995

Image capture

Apply LoG filter

SSAD computationDisparity computation

Block matchingLUT instead of rectificationSAD computation

Real-time Robot Vision

9 cameras stereo

Real-time Robot Vision

Model-based 3D Object Tracking

Virtualized Reality ™ 51 video cameras surrounding the scene

– Recorded to VCR

– Off-line 3D modeling

T. Kanade, P. Rander, P. J. Narayanan, “Virtualized Reality: Constructing Virtual Worlds from Real Scenes”, IEEE MultiMedia, vol.4, no.1, pp.34-47, 1997

Large scene from overlapped dense stereo images

Input images

3D video

Basic Principles #2

Volumetric Approach

Volumetric Approach Try to reconstruct a volume that contains an object

– Detect silhouette in each image– Backproject each silhouette– Intersect backprojected volumes

Binary ImagesBinary Images

Volume Intersection

Reconstruction contains the true scene– But is generally not the same

Visual Hull Minimum convex hull of an object

⇒ A volume obtained from infinite number of surrounding cameras

Visual hullTrue shape

Reconstructed

volume

Concave area is never recovered

Bound of the reconstruction

Voxel Algorithm for Volume Intersection

Find voxel that is projected inside silhouette in all the images– O(NM3), for N images, M3 voxels

– Don’t have to search 2M3 possible scenes!

Octree representation

Acceleration via Octree

Find voxel that is projected inside silhouette in all the images– Multi-resolution approach– Voxels that project to only background pixels in any image are carved– Voxels that project to only foreground pixels in all images remain– Ambiguous voxels are subdivided

Resolution 1Resolution 1

Results Reconstruction from 4 synthetic images

Silhouette images

16x16x16 voxel space 32x32x32 voxel space 64x64x64 voxel space

How do we get silhouette images?

Dependent on the purpose of the application

– Interactive Foreground Separation » Building static 3d model from small number of images

– Statistical Background Subtraction » Motion capture system in controlled environment from large

number of image sequence (multiple video streams)

Interactive Foreground Separation

Use of an interactive segmentation program

Y. Li, J. Sun, C.K. Tang, H.Y. Shum, “Lazy snapping,” ACM Trans. on Graphics Vol.23,No. 3, pp.303—308, 2004.

Foreground region Background region

Separation Wrong segmentation

Background region

Separation

See “Example of Graph-cut” in the lecture 6

Building Static 3d Model Volume intersection from 8 images

– No concavities

•Brightness component

•Chromaticity component

Statistical Background Subtraction

2)min(arg iiii EE

|||| iii IECD

Background image Image of interest

CDiCD if then pixel Ii is foreground

else if then pixel Ii is shadow

else pixel Ii is shadow

Choice of Threshold Learn statistics of brightness and chromaticity component for

each pixel

1 2 CDBrightness distribution Chromaticity distribution

P % P %P is around 99%

Motion Capture System in Controlled Environment

Input image Foreground & shadow Silhouette image

Basic Principles #3

Combinatorial Approach

Combinatorial Approach1. Start with an initial volume

Arbitrary rectangular shape Result of volume intersection

2. The volume is iteratively carved to be getting closer to the true shape Considering photo-consictency 2 approaches: Voxel coloring and Space-carving

Carve an inconsistentsurface voxel

Photo-consistency

Color variance of the imaged voxel among the cameras

If color variance > thresholdthe voxel is carved

If color variance < thresholdthe voxel is preserved

Voxel of interest

True surface

Voxel Coloring Approach

Input ImagesInput Images

(Calibrated)(Calibrated)

Goal: Goal: Assign RGBA values to voxels in VAssign RGBA values to voxels in Vphoto-consistentphoto-consistent with images with images

Discrete formulation

1. Choose voxel1. Choose voxel2. Project and correlate2. Project and correlate3. Color if photo-consistent3. Color if photo-consistent

Visibility Problem

In what order do we have to choose a voxel?

In which images is each voxel visible?

The Global Visibility Problem

Inverse Visibility known images

Unknown SceneUnknown Scene

Which points are visible in which images?

Known SceneKnown Scene

Forward Visibility Unknown images

LayersLayers

Solution: Depth Ordering

SceneScene

TraversalTraversal

Condition:Condition:

voxel space and group of cameras must be completely separatedvoxel space and group of cameras must be completely separated

Visit occluders first

Space-carving Algorithm

General case

Image 1 Image N

…...

– Initialize to a volume V containing the true scene

– Repeat until convergence

– Choose a voxel on the current surface

– Carve if not photo-consistent– Project to visible input images

Output of Space-carving: Photo hull

The Photo Hull is the UNION of all photo-consistent scenes in V

– Tightest possible bound on the true scene

True SceneTrue Scene

Photo HullPhoto Hull

Algorithm finds a surface consistent with the input images, but not necessarily the actual surface

Typical Error in Photo hull

actual surface

reconstructed surface

Cusping artifact Floating voxel artifact

Space-carving Implementation Multi-pass plane sweep

– Efficient, easy to implement

Sweep in –z direction Sweep in -x direction Sweep in -y direction

Repeat

Results

Details are recovered

Volume intersection Space carving Space carvingwith textures

Excessively carved

Input image

Using 16 images300k polygons

Extended Algorithm #1

Level Set Approach

Level Set Approach Define problem in 1 higher dimension

– An additional dimension, time t, is added φ(p(t),t). [Osher98]» P(t) is a point in the space

» Φ(p(t),t) is a signed distance function

– Surface, zero level set φ=0, moves along its normal direction

– Front propagation is guided by Partial Differential Equation (PDE).

– Cross-correlation is used as a constraint in PDE. [Faugeras98]

0)),(( 11 ttp

0)0),0(( p

True shape

How to move the level set surface

Define a velocity field, F, that specifies surface points move in time– In the case of stereo, F is a function of the matching cost on p

(t) in multi-baseline stereo

– F decreases to 0 around the true surface

Build an initial value for the level set function, φ(p(0),0), based on the initial surface

Adjust φ over time– Surface at time t is defined by φ(p(t),t)=0

Level Set Formulation Constraint: level set value of a point p(t) on the surface must alwa

By the chain rule

Since F supplies the speed in the outward normal direction

Hence evaluation equation for φ is

0)),(( ttp

0)(')),(( tpttpt

/ where,)(' nFntp

Essentially a gradient-descent local optimization problem

Front Propagation by PDE Front Propagation can be efficiently calculated in level

set framework– Fast marching method

– Narrow band method

Results

Can handle topology change

Extended Algorithm #2

Graph-cut Approach

GoalCalibrated images of Lambertian scene

3D model of scene

Volumetric Graph-cuts Energy function to minimize

][][][ SESESE volsurf dAxSE Ssurf )(][

Outer surface (Visual hull)

Inner surface (at constant offset from outer surface)

dVSE SVvol )(][ Volume between outer surface and S

(to avoid over-smoothing)

Reconstructed surface (S)

Integral of photo-consistency over S(S should be photo-consistent))(x Photo-consistency

Designing a Graph

1. Assign a node for each voxel

2. Connect neighboring nodes– Weight is set as ρ(x)

3. Connect outer nodes to source– Weight is infinite

4. Connect inner nodes to sink– Weight is infinite

Outer surface

Inner surface

Source

Graph-cut algorithm minimizes the energy

Designing a Graph

1. Connect all the nodes to source– Weight is λ

Outer surface

Inner surface

Source

A cut that separates source and sink

is equal to

][][][ SESESE volsurf

Results

Volume intersection

Volumetric Graph-cut

Summary Multi-baseline stereo for accurate depth map

– Good for real-time applications, dense depth map is obtained directly

Voxel-based reconstruction– Volume intersection for conservative estimate– 2 background subtraction methods for different applications– Concavities can be recovered by considering Photo-consistency

Extended algorithm– 2 approaches to add smoothness constraint– Level set approach solves local optimization problem

» can handle topology change

– Graph-cut approach solves global optimization problem» cannot handle topology change

Volume intersection– Martin & Aggarwal, “Volumetric description of objects from multiple views”, Tran

s. Pattern Analysis and Machine Intelligence, 5(2), 1991, pp. 150-158.– Szeliski, “Rapid Octree Construction from Image Sequences”, Computer Vision, Gr

aphics, and Image Processing: Image Understanding, 58(1), 1993, pp. 23-32.

Voxel coloring and Space-carving– Seitz & Dyer, “Photorealistic Scene Reconstruction by Voxel Coloring”, Proc. Comp

uter Vision and Pattern Recognition (CVPR), 1997, pp. 1067-1073.– Kutulakos & Seitz, “A Theory of Shape by Space Carving”, Proc. ICCV, 1998, pp.

307-314.

Extended algorithm– Faugeras & Keriven, “Variational principles, surface evolution, PDE's, level set met

hods and the stereo problem", IEEE Trans. on Image Processing, 7(3), 1998, pp. 336-344.

– G. Vogiatzis and P. H. S. Torr and R. Cipolla, “Multi-View Stereo via Volumetric Graph-Cuts”, Proc. CVPR, 2005, pp.391-398.

References

Next Week

6/7 Motion understanding– Prof. Ikeuchi

computer vision no. 7: stereo part 2. today’s topics u projective geometry u camera projection...

point andif point

projective planea point

point x

infinitya point

projective planein

point quadrics

point duality

projective planesimply

Documents

projective tests

euclidean plane and its relatives - github...

scalar curvature and projective embeddings, i ·...

a new parallel architecture for sparse matrix computation...

last time pinhole camera model, projection a taste of...

project prioritization matrix stakeholders list business...

projective habitat

geometrie projective

the augmented homogeneous coordinates matrix based ... ·...

radiance theory - todor georgiev - qualcomm anti symmetric...

projective geometry

bba i-bm-u-2- matrix -

projective structures and projective bundles over compact...

projective techniques

solving complex projective...

announcements projective geometry applications of projective

projective geometryavalonlibrary.net/ebooks/nick thomas -...

projective technique

projective windows: arranging windows in space using...

portafolio projective