lecture 12 stereo reconstruction ii lecture 12 stereo reconstruction ii mata kuliah: t0283 -...

Lecture 12Lecture 12 Stereo Reconstruction IIStereo Reconstruction II

Mata kuliah : T0283 - Computer VisionTahun : 2010

January 20, 2010 T0283 - Computer Vision 3

Learning ObjectivesLearning Objectives

After carefullyAfter carefully listening this lecture, students will listening this lecture, students will be able to do the following :be able to do the following :

demonstrate 3D stereo computation by solving demonstrate 3D stereo computation by solving point-point- correspondence problems and correspondence problems and fundamentalfundamental matrix.matrix.

Calculate object-depth information using Calculate object-depth information using disparity and disparity and triangulation techniquestriangulation techniques

January 20, 2010 T0283 - Computer Vision 4

An algorithm for stereo An algorithm for stereo reconstructionreconstruction

1. For each point in the first image determine the corresponding point in the second image

(this is a search problem)

2. For each pair of matched points determine the 3D point by triangulation

(this is an estimation problem)

January 20, 2010 T0283 - Computer Vision 5

Epipolar lineEpipolar line

Epipolar constraint• Reduces correspondence problem to 1D

search along an epipolar line

January 20, 2010 T0283 - Computer Vision 6

Algebraic representation of epipolar Algebraic representation of epipolar geometrygeometryWe know that the epipolar geometry defines a We know that the epipolar geometry defines a mappingmapping

x l/

point in first image

epipolar line in second

image• the map only depends on the cameras P, P/ (not on structure)

• it will be shown that the map is linear and can be written as

January 20, 2010 T0283 - Computer Vision 7

Stereo correspondence algorithms

January 20, 2010 T0283 - Computer Vision 8

Problem statementProblem statement

GivenGiven:: two images and their associated cameras two images and their associated cameras computecompute

corresponding image points.corresponding image points.

Algorithms may be classified into two types:Algorithms may be classified into two types:

1.1. Dense:Dense: compute a correspondence at every pixel compute a correspondence at every pixel

2.2. Sparse:Sparse: compute correspondences only for features compute correspondences only for features

The methods may be top down or bottom upThe methods may be top down or bottom up

January 20, 2010 T0283 - Computer Vision 9

Top down matching Top down matching

1. Group model (house, windows, etc) independently in each image

2. Match points (vertices) between images

January 20, 2010 T0283 - Computer Vision 10

Bottom up matchingBottom up matching• epipolar geometry reduces the correspondence search from 2D to a 1D search on corresponding epipolar lines

• 1D correspondence problem

b/

a/

bca

CBA

c/

January 20, 2010 T0283 - Computer Vision 11

Example image pair – parallel Example image pair – parallel camerascameras

January 20, 2010 T0283 - Computer Vision 12

First imageFirst image

January 20, 2010 T0283 - Computer Vision 13

Second imageSecond image

January 20, 2010 T0283 - Computer Vision 14

Dense correspondence algorithmDense correspondence algorithm

Search problem (geometric constraint): for each point in the left image, the corresponding point in the right image lies on the epipolar line (1D ambiguity)

Disambiguating assumption (photometric constraint): the intensity neighbourhood of corresponding points are similar across images

Measure similarity of neighbourhood intensity by cross-correlation

Parallel camera example – epipolar lines are corresponding rasters

epipolar line

January 20, 2010 T0283 - Computer Vision 15

Intensity profiles

• Clear correspondence between intensities, but also noise and ambiguity

January 20, 2010 T0283 - Computer Vision 16

Normalized Cross CorrelationNormalized Cross Correlation

region A region B

vector a vector b

write regions as vectors

a

b

January 20, 2010 T0283 - Computer Vision 17

Cross-correlation of neighbourhood Cross-correlation of neighbourhood regionsregions

epipolar line

January 20, 2010 T0283 - Computer Vision 18

left image bandright image band

cross correlation

1

0

0.5

x

January 20, 2010 T0283 - Computer Vision 19

left image band

right image band

cross correlation

1

0

x

0.5

target region

January 20, 2010 T0283 - Computer Vision 20

Why is cross-correlation such a poor measure in the second case?

1. The neighborhood region does not have a “distinctive” spatial intensity distribution

2. Foreshortening effects

front-parallel surfaceimaged length the

same

slanting surfaceimaged lengths differ

January 20, 2010 T0283 - Computer Vision 21

Limitations of similarity constraintLimitations of similarity constraint

Textureless surfaces Occlusions, repetition

Non-Lambertian surfaces, specularities

January 20, 2010 T0283 - Computer Vision 22

Results with window searchResults with window search

Window-based matching Ground truth

Data

January 20, 2010 T0283 - Computer Vision 23

Sketch of a dense correspondence Sketch of a dense correspondence algorithmalgorithm

For each pixel in the left imageFor each pixel in the left imageccompute the neighbourhood cross correlation ompute the neighbourhood cross correlation along the corresponding epipolar line in the right along the corresponding epipolar line in the right imageimagetthe corresponding pixel is the one with the he corresponding pixel is the one with the highest cross correlationhighest cross correlation

ParametersParameterssize (scale) of neighbourhoodsize (scale) of neighbourhoodsearch disparity search disparity

Other constraintsOther constraintsuniquenessuniquenessorderingorderingssmoothmoothness ofness of disparity field disparity field

ApplicabilityApplicabilitytextured scene, largely fronto-paralleltextured scene, largely fronto-parallel

January 20, 2010 T0283 - Computer Vision 24

Example dense correspondence algorithm

left image right image

January 20, 2010 T0283 - Computer Vision 25

3D Reconstruction

intensity = depthright image depth map

January 20, 2010 T0283 - Computer Vision 26

range map

Pentagon exampleleft image right image

January 20, 2010 T0283 - Computer Vision 27

RectificationRectification

e e /

For converging cameras epipolar lines are not paralle

January 20, 2010 T0283 - Computer Vision 28

Project images onto plane parallel to baseline

epipolar plane

January 20, 2010 T0283 - Computer Vision 29

Rectification continuedRectification continued

Convert converging cameras to parallel camera geometry by an image mapping

Image mapping is a 2D homography (projective transformation)

January 20, 2010 T0283 - Computer Vision 30

Example original stereo pair

rectified stereo pair

January 20, 2010 T0283 - Computer Vision 31

Note• image movement (disparity) is inversely proportional to depth Z • depth is inversely proportional to disparity

Example: depth and disparity for a parallel camera stereo rig

Then, y/ = y, and the disparity

Derivation

x

x /d

January 20, 2010 T0283 - Computer Vision 32

TriangulationTriangulation

January 20, 2010 T0283 - Computer Vision 33

Problem statementProblem statementGiven:Given: corresponding measured (i.e. noisy) points corresponding measured (i.e. noisy) points x x and and xx// , and cameras (exact) P and P , and cameras (exact) P and P//, compute the 3D point , compute the 3D point XX

Problem: in the presence of noise, back projected rays do not intersect

rays are skew in space

Measured points do not lie on corresponding epipolar lines

C C /

x x /

January 20, 2010 T0283 - Computer Vision 34

1. Vector solution1. Vector solution

C C /

Compute the mid-point of the shortest line between the two rays

January 20, 2010 T0283 - Computer Vision 35

2. Linear triangulation (algebraic solution)2. Linear triangulation (algebraic solution)

January 20, 2010 T0283 - Computer Vision 36

January 20, 2010 T0283 - Computer Vision 37

3. Minimizing a geometric/statistical error3. Minimizing a geometric/statistical error