MASKS © 2004 Invitation to 3D vision

Lecture 7Step-by-Step Model Buidling

MASKS © 2004 Invitation to 3D vision

Review

Feature correspondence

Camera Calibration

Featureselection

Featureselection

Euclidean Reconstruction

Sparse Structure and camera motion

Landing

Augmented Reality

Vision Based Control

MASKS © 2004 Invitation to 3D vision

Review

Feature correspondence

Camera Calibration

Dense Correspondence

Texture mapping

Epipolar Rectification

Featureselection

Featureselection

3-D ModelSparse Structure and motion

Euclidean Reconstruction

MASKS © 2004 Invitation to 3D vision

Review

Feature correspondence

Projective Reconstruction

Euclidean Reconstruction

Dense Correspondence

Texture mapping

Epipolar Rectification

Featureselection

Featureselection

3-D Model

Camera Self-CalibrationPartial Scene KnowledgePartial Motion Knowledge

Partial Calibration Knowledge

MASKS © 2004 Invitation to 3D vision

Examples

MASKS © 2004 Invitation to 3D vision

Feature Selection

• Compute Image Gradient• Compute Feature Quality measure for each pixel

• Search for local maxima

Feature Quality Function Local maxima of feature quality function

MASKS © 2004 Invitation to 3D vision

Feature Tracking

• Translational motion model

• Closed form solution

1. Build an image pyramid 2. Start from coarsest level 3. Estimate the displacement at the coarsest level 4. Iterate until finest level

MASKS © 2004 Invitation to 3D vision

Coarse to fine feature tracking

1. compute 2. warp the window in the second image by3. update the displacement 4. go to finer level 5. At the finest level repeat for several iterations

0

2

1

MASKS © 2004 Invitation to 3D vision

• Integrate around over image patch

• Solve

Optical Flow

MASKS © 2004 Invitation to 3D vision

Affine feature tracking

Intensity offsetContrast change

MASKS © 2004 Invitation to 3D vision

Tracked Features

MASKS © 2004 Invitation to 3D vision

Wide baseline matching

Point features detected by Harris Corner detector

MASKS © 2004 Invitation to 3D vision

Wide baseline Feature Matching

1. Select the features in two views2. For each feature in the first view3. Find the feature in the second view that maximizes4. Normalized cross-correlation measure

Select the candidate with the similarity above selected threshold

MASKS © 2004 Invitation to 3D vision

More correspondences and Robust matching

• Select set of putative correspondences

• Repeat 1. Select at random a set of 8 successful matches 2. Compute fundamental matrix 3. Determine the subset of inliers, compute

distance to epipolar line

4. Count the number of points in the consensus set

MASKS © 2004 Invitation to 3D vision

RANSAC in action

Inliers Outliers

MASKS © 2004 Invitation to 3D vision

Epipolar Geometry

• Epipolar geometry in two views• Refined epipolar geometry using nonlinear estimation of F

MASKS © 2004 Invitation to 3D vision

Two view initialization

• Recover epipolar geometry

• Compute (Euclidean) projection matrices and 3-D struct.

• Compute (Projective) projection matrices and 3-D struct.

calibrated

uncalibrated unknown

MASKS © 2004 Invitation to 3D vision

Projective Reconstruction

3-D reconstruction from two views

MASKS © 2004 Invitation to 3D vision

Multi-view reconstruction

• Two view - initialized motion and structure estimates (scales) • Multi-view factorization - recover the remaining camera positions and refine the 3-D structure by iteratively computing

1. Compute i-th motion given the known structure

2. Refine structure estimate given all the motions

Repeat until convergence

iteration

MASKS © 2004 Invitation to 3D vision

Projective Ambiguity

• calibrated case – projection matrices - and Euclidean structure

• Uncalibrated case

• for i-th frame

• for 1st frame

• Transform projection matrices and 3-D structure by H

MASKS © 2004 Invitation to 3D vision

• From Projective to Euclidean Reconstruction

• Remove the ambiguity characterized by

• How to estimate H ? - Absolute quadric constraint

• be recovered by a nonlinear minimization

unknowns

Upgrade to Euclidean Reconstruction

MASKS © 2004 Invitation to 3D vision

Upgrade to Euclidean Reconstruction

• Special case unknown focal length• other internal parameters are known (proj. center, aspect

ratio)• Q can be parametrized in the following way

• Zero entries in lead to linear constraints on Q

• Initial estimate of Q can be obtained using linear techniques

MASKS © 2004 Invitation to 3D vision

Example of multi-view reconstruction

Euclidean reconstructionProjective Reconstruction

MASKS © 2004 Invitation to 3D vision

Nonlinear Refinement

• Euclidean Bundle adjustment

• Initial estimates of are available • Final refinement, nonlinear minimization with respect to all unknowns

MASKS © 2004 Invitation to 3D vision

Example - Euclidean multi-view reconstruction

MASKS © 2004 Invitation to 3D vision

Example

Tracked FeaturesOriginal sequence

MASKS © 2004 Invitation to 3D vision

Recovered model

MASKS © 2004 Invitation to 3D vision

Euclidean Reconstruction

MASKS © 2004 Invitation to 3D vision

Epipolar rectification

• Make the epipolar lines parallel• Dense correspondences along image scanlines• Computation of warping homographies

1. Map the epipole to infinity Translate the image center to the origin Rotate around z-axis for the epipole lie on the x-axis Transform the epipole from x-axis to infinity

2. Find a matching transformation is compatible with the epipolar geometry is chosen to minimize overall disparity

MASKS © 2004 Invitation to 3D vision

Epipolar rectification

Rectified Image Pair

MASKS © 2004 Invitation to 3D vision

Epipolar rectification

Rectified Image Pair

MASKS © 2004 Invitation to 3D vision

Dense Matching

• Establish dense correspondences along scan-lines

• Standard stereo configuration • Constraints to guide the search 1. ordering constraint 2. disparity constraint – limit on

disparity 3. uniqueness constraint – each point

has a unique match in the second view

MASKS © 2004 Invitation to 3D vision

Dense Matching

MASKS © 2004 Invitation to 3D vision

Dense Reconstruction

MASKS © 2004 Invitation to 3D vision

Texture mapping, hole filling

MASKS © 2004 Invitation to 3D vision

Texture mapping