Introduction to Introduction to binocular stereo visionbinocular stereo vision

Introduction to binocular stereo vision 2

What is binocular stereo vision?What is binocular stereo vision?

• A way of getting depth (3-D) information about A way of getting depth (3-D) information about a scene from two 2-D views (images) of the a scene from two 2-D views (images) of the scenescene

Introduction to binocular stereo vision 3

What is binocular stereo vision?What is binocular stereo vision?

• A way of getting depth (3-D) information about A way of getting depth (3-D) information about a scene from two 2-D views (images) of the a scene from two 2-D views (images) of the scenescene

• Used by humans and animalsUsed by humans and animals

Introduction to binocular stereo vision 4

What is binocular stereo vision?What is binocular stereo vision?

• A way of getting depth (3-D) information about A way of getting depth (3-D) information about a scene from two 2-D views (images) of the a scene from two 2-D views (images) of the scenescene

• Used by humans and animalsUsed by humans and animals• Computational stereo visionComputational stereo vision

– Programming machines to do stereo visionProgramming machines to do stereo vision– Studied extensively in the past 25 yearsStudied extensively in the past 25 years– Difficult; still being researchedDifficult; still being researched

Introduction to binocular stereo vision 5

Purpose of this lecture:Purpose of this lecture:

• An introduction to:An introduction to:– Basic principle of stereo visionBasic principle of stereo vision– Computational stereo analysisComputational stereo analysis

• How does it work?How does it work?• What is required?What is required?• Where are the difficulties?Where are the difficulties?

Introduction to binocular stereo vision 6

Purpose of this lecture:Purpose of this lecture:

• An introduction to:An introduction to:– Basic principle of stereo visionBasic principle of stereo vision– Computational stereo analysisComputational stereo analysis

• How does it work?How does it work?• What is required?What is required?• Where are the difficulties?Where are the difficulties?

Introduction to binocular stereo vision 7

Fundamentals of stereo visionFundamentals of stereo vision

• A camera model:A camera model:– Models how 3-D scene points are transformed into 2-Models how 3-D scene points are transformed into 2-

D image pointsD image points– The pinhole camera: a simple linear model for The pinhole camera: a simple linear model for

perspective projectionperspective projection

Introduction to binocular stereo vision 8

Fundamentals of stereo visionFundamentals of stereo vision

• The goal of stereo analysis:The goal of stereo analysis:– The inverse process: From 2-D image coordinates to The inverse process: From 2-D image coordinates to

3-D scene coordinates3-D scene coordinates– Requires images from at least two viewsRequires images from at least two views

Introduction to binocular stereo vision 9

Fundamentals of stereo visionFundamentals of stereo vision

• 3-D reconstruction3-D reconstruction

Introduction to binocular stereo vision 10

Fundamentals of stereo visionFundamentals of stereo vision

• 3-D reconstruction3-D reconstruction

Introduction to binocular stereo vision 11

Fundamentals of stereo visionFundamentals of stereo vision

• 3-D reconstruction3-D reconstruction

Introduction to binocular stereo vision 12

Fundamentals of stereo visionFundamentals of stereo vision

• 3-D reconstruction3-D reconstruction

Introduction to binocular stereo vision 13

Fundamentals of stereo visionFundamentals of stereo vision

• 3-D reconstruction3-D reconstruction

Introduction to binocular stereo vision 14

Fundamentals of stereo visionFundamentals of stereo vision

• 3-D reconstruction3-D reconstruction

Introduction to binocular stereo vision 15

Fundamentals of stereo visionFundamentals of stereo vision

• 3-D reconstruction3-D reconstruction

Introduction to binocular stereo vision 16

Fundamentals of stereo visionFundamentals of stereo vision

• 3-D reconstruction3-D reconstruction

Introduction to binocular stereo vision 17

PrerequisitesPrerequisites

• Camera model parameters must be known:Camera model parameters must be known:

– External parameters: External parameters: • Positions, orientationsPositions, orientations

– Internal parameters:Internal parameters:• Focal length, image center, distortion, etc..Focal length, image center, distortion, etc..

Introduction to binocular stereo vision 18

PrerequisitesPrerequisites

• Camera calibrationCamera calibration

Introduction to binocular stereo vision 19

Two subproblemsTwo subproblems

• Matching Matching – Finding corresponding elements in the two imagesFinding corresponding elements in the two images

• ReconstructionReconstruction– Establishing 3-D coordinates from the 2-D image Establishing 3-D coordinates from the 2-D image

correspondences found during matchingcorrespondences found during matching

Introduction to binocular stereo vision 20

Two subproblemsTwo subproblems

• Matching Matching (hardest)(hardest)– Finding corresponding elements in the two imagesFinding corresponding elements in the two images

• ReconstructionReconstruction– Establishing 3-D coordinates from the 2-D image Establishing 3-D coordinates from the 2-D image

correspondences found during matchingcorrespondences found during matching

Introduction to binocular stereo vision 21

• Which image entities should be matched?Which image entities should be matched?– Two main approachesTwo main approaches

• Pixel/area-based (lower-level)Pixel/area-based (lower-level)• Feature-based (higher-level)Feature-based (higher-level)

The matching problemThe matching problem

Introduction to binocular stereo vision 22

Matching challengesMatching challenges

• Scene elements do not always look the same Scene elements do not always look the same in the two imagesin the two images– Camera-related problemsCamera-related problems

• Image noise, differing gain, contrast, etc..Image noise, differing gain, contrast, etc..

– Viewpoint-related problems:Viewpoint-related problems:• Perspective distortionsPerspective distortions• OcclusionsOcclusions• Specular reflectionsSpecular reflections

Introduction to binocular stereo vision 23

Choice of camera setupChoice of camera setup

• BaselineBaseline– distance between cameras (focal points)distance between cameras (focal points)

• Trade-offTrade-off– Small baseline: Matching easierSmall baseline: Matching easier– Large baseline: Depth precision betterLarge baseline: Depth precision better

Introduction to binocular stereo vision 24

Matching cluesMatching clues

• Correspondance search is a 1-D problemCorrespondance search is a 1-D problem– Matching point must lie on a lineMatching point must lie on a line

Introduction to binocular stereo vision 25

Matching cluesMatching clues

• Epipolar geometryEpipolar geometry

Introduction to binocular stereo vision 26

Matching cluesMatching clues

• Epipolar geometryEpipolar geometry

Introduction to binocular stereo vision 27

RectificationRectification

• Simplifies the correspondance searchSimplifies the correspondance search– Makes all epipolar lines parallel and coincidentMakes all epipolar lines parallel and coincident– Corresponds to parallel camera configurationCorresponds to parallel camera configuration

Introduction to binocular stereo vision 28

Goal: disparity mapGoal: disparity map

• Disparity: Disparity: – The horizontal displacement between corresponding The horizontal displacement between corresponding

pointspoints– Closely related to scene depthClosely related to scene depth

Introduction to binocular stereo vision 29

More matching heuristicsMore matching heuristics

• Always valid:Always valid:– (Epipolar line)(Epipolar line)– UniquenessUniqueness– Minimum/maximum disparityMinimum/maximum disparity

• Sometimes valid:Sometimes valid:– OrderingOrdering– Local continuity (smoothness)Local continuity (smoothness)

Introduction to binocular stereo vision 30

Area-based matchingArea-based matching

• Finding pixel-to-pixel correspondencesFinding pixel-to-pixel correspondences– For each pixel in the left image, search for the most For each pixel in the left image, search for the most

similar pixel in the right imagesimilar pixel in the right image

Introduction to binocular stereo vision 31

Area-based matchingArea-based matching

• Finding pixel-to-pixel correspondencesFinding pixel-to-pixel correspondences– For each pixel in the left image, search for the most For each pixel in the left image, search for the most

similar pixel in the right imagesimilar pixel in the right image– Using neighbourhood windowsUsing neighbourhood windows

Introduction to binocular stereo vision 32

Area-based matchingArea-based matching

• Similarity measures for two windowsSimilarity measures for two windows– SAD (sum of absolute differences)SAD (sum of absolute differences)– SSD (sum of squared differences)SSD (sum of squared differences)– CC (cross-correlation)CC (cross-correlation)– ……

Introduction to binocular stereo vision 33

Feature-based matchingFeature-based matching

• Matching features:Matching features:– Edge pointsEdge points– lineslines– cornerscorners– ……

• Sparse reconstruction setsSparse reconstruction sets• Best if scene type is known Best if scene type is known a prioria priori

Introduction to binocular stereo vision 34

Area-based matchingArea-based matching

• Choice of window sizeChoice of window size– Factors to considers:Factors to considers:

• AmbiguityAmbiguity• Noise sensitivityNoise sensitivity• Sensitivity towards viewpoint-related distortionsSensitivity towards viewpoint-related distortions• Expected object sizesExpected object sizes• Frequency of depth jumpsFrequency of depth jumps

Introduction to binocular stereo vision 35

Area-based matchingArea-based matching

• Variable window positionVariable window position– Better matching at depth jumps (disparity edges)Better matching at depth jumps (disparity edges)

Introduction to binocular stereo vision 36

Three or more viewpointsThree or more viewpoints

• More matching informationMore matching information– Additional epipolar constraintsAdditional epipolar constraints– More confident matchesMore confident matches

Introduction to binocular stereo vision 37

SummarySummary

• Stereo vision: Stereo vision: – A method for 3-D analysis of a scene using images A method for 3-D analysis of a scene using images

from two or more viewpointsfrom two or more viewpoints

• Two subproblems:Two subproblems:– MatchingMatching– ReconstructionReconstruction

• Most difficult part: MatchingMost difficult part: Matching– Two main approaches:Two main approaches:

• Area based: Dense reconstructionArea based: Dense reconstruction• Feature based: Sparse reconstructionFeature based: Sparse reconstruction

Introduction to binocular stereo vision 38

Modelling stereo quantification Modelling stereo quantification errorerror

Introduction to binocular stereo vision 39

Stereo error quantificationStereo error quantification

The variance:

Numerical solution:

Introduction to binocular stereo vision 40

Error analytical vs. Numerical Error analytical vs. Numerical solutionsolution