Computer Visioncmput 499/615

Intro toIntro to

Practical 3D VisionPractical 3D Vision

Upcoming course projectUpcoming course project

Martin JagersandMartin Jagersand

Where we are in course

•Done:Done:– Basics of computer vision, imagesBasics of computer vision, images– 2D Point correspondences, visual motion, tracking2D Point correspondences, visual motion, tracking– Euclidean and projective geometry in 2D and 3DEuclidean and projective geometry in 2D and 3D

•Upcoming:Upcoming:– 3D modeling from 2D image points3D modeling from 2D image points– How surfaces look? Texture, Light models, ReflectanceHow surfaces look? Texture, Light models, Reflectance– Human visionHuman vision– Applications of computer vision (sci/eng/ind, entertainment, Applications of computer vision (sci/eng/ind, entertainment,

educational, service)educational, service)

Texture

•Texture = Fine scale surface appearance.Texture = Fine scale surface appearance.

• Often means endowing the surface of a Often means endowing the surface of a geometric (polygonal) model with such fine scale geometric (polygonal) model with such fine scale properties.properties.

Std ref: Heckbert MSc thesis 89, Moller, Haines book 2nd ed. 2002.

What we see given a particular surface texture and light?

• BRDF BRDF

(Bi-Directional Reflectance Function)(Bi-Directional Reflectance Function) • Describes how incoming Describes how incoming

light on a surface is light on a surface is reflected back in different reflected back in different directions. directions.

Light basis

DemoE:\demos\light\light.exe

É T l = B lyl

Processing elements and Pathways of Human Vision

• Eye transforms light into Eye transforms light into nerve impulsesnerve impulses

• Optic chiasm splits left Optic chiasm splits left and right visual fieldsand right visual fields

• LGN: Exact function LGN: Exact function unknown. May have to unknown. May have to do with stereo.do with stereo.

• V1 (Striate cortex) V1 (Striate cortex) performs spatial filtering performs spatial filtering / coordinate transforms/ coordinate transforms

Industrial applications: Factory Inspection

Cognex’s “CapInspect” system:

Low-level image analysis: Identify edges, regionsMid-level: Distinguish “cap” from “no cap”

Estimation: What are orientation of cap, height of liquid?

Entertainment applications: Motion Capture / Animation

Vicon software:12 cameras, 41 markers for body capture;

6 zoom cameras, 30 markers for face

Applications educationalVirtual Heritage

Input images of multiple views

Dynamic texture

Geometric model from SFS

Rendered model

object

animationsanimations and and Internet web study materialInternet web study material..

Project and labs

•Optic flow and tracking: doneOptic flow and tracking: done•Geometry and cameras: On nowGeometry and cameras: On now•Upcoming labs:Upcoming labs:

– L3: 3D modelingL3: 3D modeling– L4: Reflectance and appearanceL4: Reflectance and appearance

•Project Applications of Computer Vision: Project Applications of Computer Vision: – Group or individualGroup or individual– Worth 2 labs.Worth 2 labs.– P1: Review and proposal. Revised-> P1a, P1bP1: Review and proposal. Revised-> P1a, P1b– P2: Implementation, demo and final report: End of classes.P2: Implementation, demo and final report: End of classes.

Goals of project

•Learn more in depth and hands-on about one of Learn more in depth and hands-on about one of the course topicsthe course topics– Read web descriptions of system and research papersRead web descriptions of system and research papers– Practically experiment with an implemented system to learn Practically experiment with an implemented system to learn

how to use it and what type of images/video it works well at.how to use it and what type of images/video it works well at.– Find our what are weaknesses and how those could be Find our what are weaknesses and how those could be

addressed.addressed.

•Study and implement a small addition or Study and implement a small addition or modification to a part of this system.modification to a part of this system.

Value of hands-on experiments on implemented systems

•Computer vision methods are fragile. (similar to Computer vision methods are fragile. (similar to much other research) much other research)

•Generally don’t work robustly and reliably on all Generally don’t work robustly and reliably on all inputs, including normal real-world video/imagesinputs, including normal real-world video/images

•Some implemented systems represent many man Some implemented systems represent many man years of effort. Working with these can learn years of effort. Working with these can learn much of how they work w/o taking years!much of how they work w/o taking years!

Systems presented in course(and where Instructor and TA can help)

• Capgui:Capgui: Shape from Silhouette Geometry and Shape from Silhouette Geometry and MultitexturingMultitexturing

• ARC3D:ARC3D: Corresponding points based SFM (Structure Corresponding points based SFM (Structure and Motion). Download geometry and camerasand Motion). Download geometry and cameras

• Photosynth:Photosynth: similar SFM. View geometry and cameras similar SFM. View geometry and cameras on web. AFAIK: format of data/geometry not published.on web. AFAIK: format of data/geometry not published.

• Capture_UI:Capture_UI: Simpler factorization used in lab 3 Simpler factorization used in lab 3• PTAM, mono SLAMPTAM, mono SLAM: Real time incremental geometry: Real time incremental geometry• XVision:XVision: Tracking system. (Other individual trackers are Tracking system. (Other individual trackers are

available in e.g. OpenCV and other web sites) available in e.g. OpenCV and other web sites)

Some possible project topicswho they may fit

•Geometry related:Geometry related: Suitable for someone with Suitable for someone with interest in computer vision and graphics and/or interest in computer vision and graphics and/or computational geometrycomputational geometry

•Appearance/texture related:Appearance/texture related: Interest/knowledge Interest/knowledge in data representation, subspace/manifold in data representation, subspace/manifold estimation helps.estimation helps.

•Dynamic vision/Tracking:Dynamic vision/Tracking: Interest in dynamics, Interest in dynamics, robotics, filtering can helprobotics, filtering can help

Multi-texturing

Multi texturing:Multi texturing: capgui blending 20 basis textures. capgui blending 20 basis textures. Generated by PCA over all the input textures. Generated by PCA over all the input textures.

Possible improvement:Possible improvement:• Partition view angles into subsets, then PCA on each Partition view angles into subsets, then PCA on each

subset. subset. • What is the best subset of view angles? Quadrants? What is the best subset of view angles? Quadrants?

Octants? Scene/object dependent?Octants? Scene/object dependent?• Figure out how to handle overlap and the boundaries. Figure out how to handle overlap and the boundaries. Study references from graphics literature, e.g. Study references from graphics literature, e.g. "A Theory of "A Theory of

Locally Low Dimensional Light Transport" by Dhruv Mahajan, Ira Locally Low Dimensional Light Transport" by Dhruv Mahajan, Ira Kemelmacher Shlizerman, Ravi Ramamoorthi, and Peter Belhumeur Kemelmacher Shlizerman, Ravi Ramamoorthi, and Peter Belhumeur

Multi-texturing(cont)

• In some cases (e.g. geometry very inaccurate) other In some cases (e.g. geometry very inaccurate) other (e.g. non-linear) subspace methods may perform (e.g. non-linear) subspace methods may perform better (e.g. isomap, kernel pca, etc). Try to find the better (e.g. isomap, kernel pca, etc). Try to find the properties of the texture manifold and matc h that to a properties of the texture manifold and matc h that to a suitable method. suitable method.

• Write a texturing plugin using some method (e.g. Write a texturing plugin using some method (e.g. original or one of the above) for Blender so captured original or one of the above) for Blender so captured objects can be integrated and rendered in blender objects can be integrated and rendered in blender scenes. scenes.

• Texture decomposition into low/high frequency. Texture decomposition into low/high frequency.

Single texturing

• Legacy SW that can only handle a single texture, Legacy SW that can only handle a single texture, • How to distill the in some measure "best" color w.r.t. all How to distill the in some measure "best" color w.r.t. all

the input data for a particular object point?the input data for a particular object point?• Heuristic methods for this Heuristic methods for this

– Blending images for texturing 3D Models, Adam Baumberg BMVC 2002)Blending images for texturing 3D Models, Adam Baumberg BMVC 2002)

• Globally optimal:Globally optimal:– Spatio-temporal Image-based Texture Atlases for Dynamic 3-D Models Spatio-temporal Image-based Texture Atlases for Dynamic 3-D Models Janko and Jean-Philippe Pons 3DIM2009, Janko and Jean-Philippe Pons 3DIM2009, – Seamless image-based texture atlases using multi-band blending, Seamless image-based texture atlases using multi-band blending, Allène, C. and Pons, J-P. and Keriven, R. ICPR 2008, Allène, C. and Pons, J-P. and Keriven, R. ICPR 2008,

– Seamless Mosaicing of Image-Based Texture Maps Seamless Mosaicing of Image-Based Texture Maps Victor Lempitsky and Denis Ivanov CVPR.2007) Victor Lempitsky and Denis Ivanov CVPR.2007)

Study these methods (we have some example code that Study these methods (we have some example code that can get you started) and see how they compare. can get you started) and see how they compare.

See if you can integrate one into the CapGui See if you can integrate one into the CapGui

Tracking

1.1. Study a different tracking modality, e.g. Study a different tracking modality, e.g. • Hyperplanes for SSD tracking (Juri and Dhome, PAMI 2004)Hyperplanes for SSD tracking (Juri and Dhome, PAMI 2004)• Meanshift (Dorin Comaniciu, and Peter Meer) Meanshift (Dorin Comaniciu, and Peter Meer)

Integrate this tracking into XVison and compare performance Integrate this tracking into XVison and compare performance with existing XVision trackers. with existing XVision trackers.

• Implement compositional tracking similar to Implement compositional tracking similar to the IFA the IFA

• Incremental Focus of Attention system by Toyama and HagerIncremental Focus of Attention system by Toyama and Hager

Dynamics in P^2

•How do Newton’s laws look in the image plane?How do Newton’s laws look in the image plane?

•What invariants are preserved? Are there others What invariants are preserved? Are there others we can compute?we can compute?

•How do we build a motion model for what we How do we build a motion model for what we observe in tracking?observe in tracking?

•How do we control motion of a robot from what How do we control motion of a robot from what we see in video (visual servoing)?we see in video (visual servoing)?

Quadri-linear Factorization

• Under the usual equality x=y we can compute a Under the usual equality x=y we can compute a low rank approximation (aka PCA) using the low rank approximation (aka PCA) using the regular SVD:regular SVD:M=U VT

• Under projective equality Under projective equality x=y we need to x=y we need to numerically account for scale factors numerically account for scale factors

• How do we compute a projective “SVD” How do we compute a projective “SVD” M=U VT ??

How to go about it

• For example in a geometry project:For example in a geometry project:– First study existing methods. The HZ book has the mathematical First study existing methods. The HZ book has the mathematical

fundamentalsThe Szeliski book describes the state of the art and has paper fundamentalsThe Szeliski book describes the state of the art and has paper refences. refences.

– Try one or more of the existing systems to get a feel for how current methods Try one or more of the existing systems to get a feel for how current methods work. Systems you can try include our capgui (shown in the labs), Arc3D, work. Systems you can try include our capgui (shown in the labs), Arc3D, Photosynth. Photosynth.

– Try to find what type of models can be created, what are limitations (e.g. how Try to find what type of models can be created, what are limitations (e.g. how do the images have to be taken).do the images have to be taken).

• The reading and trying is for the review part to go with the proposal The reading and trying is for the review part to go with the proposal P1 to be handed in first. P1 to be handed in first.

• Then find something to research and implement yourself. This could Then find something to research and implement yourself. This could be a subtopic such as geometric refinement or an application where be a subtopic such as geometric refinement or an application where you capture and use vision based models for some effects in e.g. an you capture and use vision based models for some effects in e.g. an animation or computer game. animation or computer game.

How to go about it(cont)

In a tracking project you can similarly In a tracking project you can similarly

• first try our XVision system and or other trackers first try our XVision system and or other trackers you can download, you can download,

• then propose some extension or additional then propose some extension or additional tracking modality to implement.tracking modality to implement.

Application projectPossible topics

• Mathematical/geometric: A capture system for man-made Mathematical/geometric: A capture system for man-made structure: structure: – Capture structure using some a-priori knowledge: orthogonality, parallelism, planes. Capture structure using some a-priori knowledge: orthogonality, parallelism, planes.

This makes capture easier and gives Euclidean structure w/o cumbersome “self This makes capture easier and gives Euclidean structure w/o cumbersome “self calibration”calibration”

• Systems: Live video or 3D conferencingSystems: Live video or 3D conferencing– Track, compress video (dynamic) texturesTrack, compress video (dynamic) textures– Encode, transmit and renderEncode, transmit and render

• Games: Capture 3D characters, scenes, Games: Capture 3D characters, scenes, script a plot and put into game enginescript a plot and put into game engine• Haptics/robotics: Capture something visually and allow haptic Haptics/robotics: Capture something visually and allow haptic

exploration of this. (ie as aid to blind)exploration of this. (ie as aid to blind)• Survelliance: Track people etc (ie for security system)Survelliance: Track people etc (ie for security system)

Application project

•Discuss ideas for interesting projectsDiscuss ideas for interesting projects

• Identify some books, web pages or papers to Identify some books, web pages or papers to serve as core information sources.serve as core information sources.

• Identify a handful of early classic papers to seed Identify a handful of early classic papers to seed your literature search. I will do my best to help your literature search. I will do my best to help here also. here also.

•Tips on carrying out your projectsTips on carrying out your projects– Balance between reading and doingBalance between reading and doing

Schedule

• Now (reading week) good time to think about what you Now (reading week) good time to think about what you would be interested in doingwould be interested in doing– Find available systems (ours and on Internet) and try to use them practically.Find available systems (ours and on Internet) and try to use them practically.

• MarchMarchproject proposal.project proposal.– Include reference list and a start at literature review, ie. Read some papers and Include reference list and a start at literature review, ie. Read some papers and

write a few pages summarywrite a few pages summary– Include experience from tryingInclude experience from trying

• Throughout course in class: Keep up to date on your Throughout course in class: Keep up to date on your project progress. project progress.

• End of semester: Project reports.End of semester: Project reports.

Resources

• SW: I/we will install and try to help support:SW: I/we will install and try to help support:– Real time video input (Currently SGI, linux, could be Windows)Real time video input (Currently SGI, linux, could be Windows)– Basic tracking, XVisionBasic tracking, XVision– Geometry, Hand-eye, Robotics codeGeometry, Hand-eye, Robotics code

• HW: Access to machines in Robotics/Vision labHW: Access to machines in Robotics/Vision lab– Cameras: Linux IEEE 1394 in lab. Could also use digital still camera, Cameras: Linux IEEE 1394 in lab. Could also use digital still camera,

camcorder.camcorder.– Visualization: Ok in lab (HW acc graphics), Better in research lab (new ATI Visualization: Ok in lab (HW acc graphics), Better in research lab (new ATI

and nvidia, SGI HW, projectors and CAVE)and nvidia, SGI HW, projectors and CAVE)– Vision and haptics: Have phantoms.Vision and haptics: Have phantoms.– Vision for motion control: Robot arms: Could be set up.Vision for motion control: Robot arms: Could be set up.– Anything else? Some resources for buying availableAnything else? Some resources for buying available

Martin’s tips

•Plan incremental progress and checkpoints.Plan incremental progress and checkpoints.– Makes it easier to identify promising directions as well as Makes it easier to identify promising directions as well as

difficulties and redefine plans as needed.difficulties and redefine plans as needed.

•Find balance between reading and doingFind balance between reading and doing– It is difficult to fully grasp methods by only readingIt is difficult to fully grasp methods by only reading– Some experiments are incomplete, results wrongSome experiments are incomplete, results wrong

•Practical trying out can add a lot of insight. Practical trying out can add a lot of insight. – Learn how to quickly prototype in e.g. matlabLearn how to quickly prototype in e.g. matlab

Literature search

• Goal:Goal: Find the handful to dozen most Find the handful to dozen most relevant and recent papers in a subarea.relevant and recent papers in a subarea.

• Method:Method: 1.1. Seed with a few relevant papers.Seed with a few relevant papers.2.2. Do citation search backwards and forwards.Do citation search backwards and forwards.3.3. Find common “buzz words”. Do title and abstract text Find common “buzz words”. Do title and abstract text

search.search.4.4. Do internet search. e.g. “Cora” from justresearchDo internet search. e.g. “Cora” from justresearch5.5. Check most recent proceedings manually. (They won’t Check most recent proceedings manually. (They won’t

be indexed yet)be indexed yet)

Report:

1.1. Review Review – Summarize the main contributions and comparing the results Summarize the main contributions and comparing the results

in the papers.in the papers.

2.2. Your contribution and experiments. Your contribution and experiments. – MethodsMethods– ResultsResults

3.3. DiscussionDiscussion– Where does it fit into the bigger pictureWhere does it fit into the bigger picture– Future workFuture work

Introduction to 3D captuer

1.1. CapguiCapgui

2.2. Arc 3D (separate slides)Arc 3D (separate slides)

Macro geometry:1. Shape From Silhouette

• For capturing objectsFor capturing objects1.1. Extract silhouette in each Extract silhouette in each

image.image.2.2. Extend 2D silhouettes to Extend 2D silhouettes to

3D cones.3D cones.3.3. Intersect cones.Intersect cones.• The limiting smallest The limiting smallest

volume obtainable in this volume obtainable in this way is known as the way is known as the visual visual hullhull of the object of the object..

Macro geometry:2. Structure from motion

• Best for capturing scenesBest for capturing scenes

1.1. Pick several salient scene pointsPick several salient scene points

2.2. Extract their 2D image projection in several imagesExtract their 2D image projection in several images

3.3. Use multiple view geometry to compute 3D shapeUse multiple view geometry to compute 3D shape

posesstructure

Structure from motion

algorithm

Course: 3D Computer Vision

Example SW: Photosynth, PTAM

Macro geometry:3. Dense stereo

SFM gives couarse structure, 100s-1000s ptsSFM gives couarse structure, 100s-1000s pts

We may want 10,000-100,000We may want 10,000-100,000• Go back to original images, do dense matching.Go back to original images, do dense matching.• Try to get dense depth mapsTry to get dense depth maps

All epipolar lines are parallel in the rectified image plane.

Dense stereo: Rectify and redo image matching

Geometry alone does not solve modeling!Need: Multi-Scale Model

ModelingModeling(Computer vision)(Computer vision)

RenderingRendering(Computer Graphics)(Computer Graphics)

Geometry

Depth

Dynamic texture

Multi-Scale model: Macro geometry, Meso depth, Micro texture

Demo

Geometry alone does not solve modeling!Need: Multi-Scale Model

ModelingModeling(Computer vision)(Computer vision)

RenderingRendering(Computer Graphics)(Computer Graphics)

Geometry

Depth

Dynamic texture

Multi-Scale model: Macro geometry, Meso depth, Micro texture

Demo2: houseSFS.yxv

Demo1: house.yxv

Low budget 3D from video

• InexpensiveInexpensive

•Quick and convenient Quick and convenient for the userfor the user

• Integrates with existing Integrates with existing SW e.g. Blender, MayaSW e.g. Blender, Maya

(Main talk)

Low budget 3D from video

• InexpensiveInexpensive

•Quick and convenient for the user

•Integrates with existing SW e.g. Blender, Maya

$100: Webcams, Digital Cams $100,000 Laser scanners etc.

Low budget 3D from video

• InexpensiveInexpensive

•Quick and convenient Quick and convenient for the userfor the user

• Integrates with existing Integrates with existing SW e.g. Blender, MayaSW e.g. Blender, Maya

Modeling geom primitives into scenes: >>Hours

Capturing 3D from 2D video: minutes

Low budget 3D from video

• InexpensiveInexpensive

•Quick and convenient Quick and convenient for the userfor the user

• Integrates with existing Integrates with existing SW e.g. Blender, MayaSW e.g. Blender, Maya

Application Case StudyModeling Inuit Artifacts

• New acquisition at the UofA: A group of 8 sculptures depicting Inuit seal hunt

• Acquired from sculptor by Hudson Bay Company

Application Case StudyModeling Inuit Artifacts

Results:Results:

1.1. A collection of 3D models of each componentA collection of 3D models of each component

2.2. Assembly of the individual models into Assembly of the individual models into animationsanimations and and Internet web study materialInternet web study material..

The equation of projection

Mathematically:Mathematically:• Cartesian coordinates:Cartesian coordinates:

• Projectively: x = PXProjectively: x = PX(x, y, z) ( fx

z, f

y

z)

How do we develop a consistent mathematical framework for projection calculations?

Intuitively:Intuitively:

Perspective and projection

• Euclidean geometry is not the only representation- for building models from images- for building images from models

representations ?

• But when 3d realism is the goal, how can we effectively build and use

projectiveaffinemetric

X

Y

Z

Homogeneous coordinates

X

Y

Z

s

s 0

X∞

X

Y

0

X∞ =

• Perspective imaging models 2d projective space

• Each 3D ray is a point in P2 : homogeneous coords.

• Ideal points

• P2 is R2 plus a “line at infinity” l∞

The 2D projective plane

Projective point

l∞

x

yX

Y1Z= Inhomogeneou

s equivalent

How to translate a 2D point:

•Old way:Old way:

•New way:New way:

Lines

HZ • Ideal line ~ the plane parallel to the image

A

B

C

l =X=lTX = XTl = AX + BY + CZ = 0

l∞ =

0

0

1

• Projective line ~ a plane through the origin

For any 2d projective property, a dual property holds when the role of points and lines are interchanged.

Duality:

X

Y

0

X∞ =

l1 l2X =X1 X2=l

The line joining two points The point joining two lines

X

Y

Z

X

l“line at infinity

”

Projective transformations

• Homographies, collineations, projectivitiesHomographies, collineations, projectivities

• 3x3 nonsingular H3x3 nonsingular H

3

2

1

333231

232221

131211

3

2

1

'

'

'

x

x

x

hhh

hhh

hhh

x

x

x

l0= Hà Tl

xTl = 0 x0Tl0= 0

maps P2 to P2

8 degrees of freedomdetermined by 4 corresponding points

x0= Hx• Transforming Lines?subspaces preserved

xTHTl0= 0substitution

dual transformation

GroupGroup TransformationTransformation InvariantsInvariants DistortionDistortion

ProjectiveProjective

8 DOF8 DOF

• Cross ratioCross ratio

• IntersectionIntersection

• TangencyTangency

AffineAffine

6 DOF6 DOF

• ParallelismParallelism

• Relative dist in 1dRelative dist in 1d

• Line at infinityLine at infinity

MetricMetric

4 DOF4 DOF

• Relative distancesRelative distances

• AnglesAngles

• Dual conicDual conic

EuclideanEuclidean

3 DOF3 DOF

• LengthsLengths

• AreasAreas

Geometric strata: 2d overview

1TS

sRH

O

t

1TA

AH

O

t

1TE

RH

O

t

v

AH TP v

t

C*

2 dofl

2 dof

l

C*

Projective 3D space

0),,,,( 44321 xxxxxX

),,()1,,,(

)/,/,/(),,,( 4342414321

ZYXZYX

xxxxxxxxxx

)0,,,( 321 xxx

•Projective transformation:Projective transformation: –4x4 nonsingular matrix 4x4 nonsingular matrix HH–Point transformationPoint transformation–Plane transformationPlane transformation

•PointsPoints

•PlanesPlanes0

),,,( 4321

Xπ

πT

Points and planes are dual in 3d projective

space.

ππ

XXTH

H

'

'

•Lines: 5DOF, various parameterizationsLines: 5DOF, various parameterizations

GroupGroup TransformationTransformation InvariantsInvariants DistortionDistortion

ProjectiveProjective

15 DOF15 DOF

• Cross ratioCross ratio

• IntersectionIntersection

• TangencyTangency

AffineAffine

12 DOF12 DOF

• ParallelismParallelism

• Relative dist in 1dRelative dist in 1d

• Plane at infinityPlane at infinity

MetricMetric

7 DOF7 DOF

• Relative distancesRelative distances

• AnglesAngles

• Absolute conicAbsolute conic

EuclideanEuclidean

6 DOF6 DOF

• LengthsLengths

• AreasAreas

• VolumesVolumes

Geometric strata: 3d overview

1TS

sRH

O

t

1TA

AH

O

t

1TE

RH

O

t

v

AH TP v

t

π

3DOFπ

5DOF

Faugeras ‘95

• Projection equationProjection equation

xxii=P=PiiXX

• Resection:Resection:– xxii,X P,X Pii

Computational tasks - resection

Given image points and 3D points calculate camera projection matrix.

• Projection equationProjection equation

xxii=P=PiiXX

• Intersection:Intersection:– xxii,P,Pi i XX

Computational tasks - intersection

Given image points and camera projections in at least 2 views calculate the 3D points (structure)

• Projection Projection equationequation

xxii=P=PiiXX

• Structure from Structure from motion (SFM)motion (SFM)– xxii P Pii,, XX

Computational tasks - SFM

Given image points in at least 2 views calculate the 3D points (structure) and camera projection matrices (motion)

•Estimate projective structure

•Rectify the reconstruction to metric (autocalibration)

2 Cameras:Depth from stereo

•Calibrated aligned camerasCalibrated aligned cameras

Z

X(0,0) (d,0)

Z=f

xl xrrl

rl

xx

dfZ

dXx

fX

x

fZ

)(

Disparity d

Trinocular Vision System

(Point Grey Research)

N cameras: factorizationExample: affine model

[Tomasi &Kanade ’92][Tomasi &Kanade ’92]

•Affine camera Affine camera

•ProjectionProjection

•nn points, points, mm views: measurement matrix views: measurement matrix

]|[ tMP M 2x3 matrix; t 2D vector

t

Z

Y

X

My

x

nmm

nm

n

M

M

W XX

xx

xx

1

1

1

111

~~

~~

W: Rank 3XMVDUW

UDVWT

nm

T

ˆˆˆ33332

Assuming isotropic zero-mean Gaussian noise, factorization achieves ML affine reconstruction.

txx ~

Upgrading geometric interpretation (Self) Calibration

15 DOF 12 DOFplane at infinity

parallelism

More general

More structure

Projective Affine Metric

7 DOFabsolute conicangles, rel.dist.

A complete modeling systemprojective

Sequence of frames scene structureSequence of frames scene structure

1.1. Get corresponding points (tracking).Get corresponding points (tracking).

2.2. 2,3 view geometry: 2,3 view geometry: compute F,T between consecutive frames compute F,T between consecutive frames (recompute correspondences).(recompute correspondences).

3.3. Initial reconstruction: Initial reconstruction: get an initial structure from a get an initial structure from a subsequence with big baseline (trilinear tensor, factorization …) subsequence with big baseline (trilinear tensor, factorization …) and bind more frames/points using resection/intersection.and bind more frames/points using resection/intersection.

4.4. Self-calibration.Self-calibration.

5.5. Bundle adjustment.Bundle adjustment.

A complete modeling systemaffine

Sequence of frames scene structureSequence of frames scene structure

1.1. Get corresponding points (tracking).Get corresponding points (tracking).

2.2. Affine factorization. (This already computes ML Affine factorization. (This already computes ML estimate over all frames so no need for bundle estimate over all frames so no need for bundle adjustment for simple scenes.adjustment for simple scenes.

3.3. Self-calibration.Self-calibration.

4.4. If several model segments: Merge, bundle adjust.If several model segments: Merge, bundle adjust.