multiview reconstruction. why more than 2 views? baselinebaseline – too short – low accuracy –...

Multiview ReconstructionMultiview Reconstruction

Why More Than 2 Views?Why More Than 2 Views?

• BaselineBaseline– Too short – low accuracyToo short – low accuracy

– Too long – matching becomes hardToo long – matching becomes hard

Why More Than 2 Views?Why More Than 2 Views?

• Ambiguity with 2 viewsAmbiguity with 2 views

Camera 1Camera 1 Camera 2Camera 2Camera 3Camera 3

Trinocular StereoTrinocular Stereo

• Straightforward approach to eliminate Straightforward approach to eliminate bad correspondencesbad correspondences– Pick 2 views, find correspondencesPick 2 views, find correspondences

– For each matching pair, reconstruct 3D For each matching pair, reconstruct 3D pointpoint

– Project point into 3Project point into 3rdrd image image

– If can’t find correspondence near predicted If can’t find correspondence near predicted location, rejectlocation, reject

Trinocular StereoTrinocular Stereo

• Trifocal geometry: relations between Trifocal geometry: relations between points in three camera viewspoints in three camera views

• Trifocal tensor: analogue of essential Trifocal tensor: analogue of essential matrixmatrix– 3x3x3 trilinear tensor (3D cube of numbers)3x3x3 trilinear tensor (3D cube of numbers)

– Given lines in 2 views, predict lines in the 3Given lines in 2 views, predict lines in the 3rdrd

Multibaseline StereoMultibaseline Stereo

• Slightly different algorithm for Slightly different algorithm for nn cameras: cameras:

• Pick one reference viewPick one reference view

• For each candidate depthFor each candidate depth– Compute sum of squared differences to all Compute sum of squared differences to all

other views, assuming correct disparity for other views, assuming correct disparity for viewview

• Resolves ambiguities: only correct depths Resolves ambiguities: only correct depths will “constructively interfere”will “constructively interfere”


[[OkutamiOkutami & Kanade] & Kanade]

Multibaseline Stereo Multibaseline Stereo ReconstructionReconstruction

Problems with Multibaseline Problems with Multibaseline StereoStereo

• Have to pick a reference viewHave to pick a reference view

• OcclusionOcclusion– With many cameras / large baseline, With many cameras / large baseline,

occlusion becomes likelyocclusion becomes likely

– Contributes incorrect values to error Contributes incorrect values to error functionfunction

Volumetric Multiview ApproachesVolumetric Multiview Approaches

• Goal: find a model consistent with Goal: find a model consistent with imagesimages

• ““Model-centric” (vs. image-centric)Model-centric” (vs. image-centric)

• Typically use discretized volume (voxel Typically use discretized volume (voxel grid)grid)

• For each voxel, compute occupied / freeFor each voxel, compute occupied / free(for some algorithms, also color, etc.)(for some algorithms, also color, etc.)

Photo ConsistencyPhoto Consistency

• Result: not necessarily correct sceneResult: not necessarily correct scene

• Many scenes produce the same imagesMany scenes produce the same images

All scenesAll scenes

Photo-consistent scenesPhoto-consistent scenes

True sceneTrue scene ReconstructedReconstructedscenescene

Silhouette CarvingSilhouette Carving

• Find silhouettes in all imagesFind silhouettes in all images

• Exact version:Exact version:– Back-project all silhouettes, find intersectionBack-project all silhouettes, find intersection

Binary ImagesBinary Images


• Find silhouettes in all imagesFind silhouettes in all images

• Exact version:Exact version:– Back-project all silhouettes, find intersectionBack-project all silhouettes, find intersection


• Limit of silhouette carving is Limit of silhouette carving is visual hullvisual hull ororline hullline hull

• Complement of lines that don’t intersect Complement of lines that don’t intersect objectobject

• In general not the same as objectIn general not the same as object– Can’t recover “pits” in objectCan’t recover “pits” in object

• Not the same as convex hullNot the same as convex hull


• Discrete version:Discrete version:– Loop over all voxels in some volumeLoop over all voxels in some volume

– If projection into images lies inside all If projection into images lies inside all silhouettes, mark as occupiedsilhouettes, mark as occupied

– Else mark as freeElse mark as free

Voxel ColoringVoxel Coloring

• Seitz and Dyer, 1997Seitz and Dyer, 1997

• In addition to free / occupied, store In addition to free / occupied, store colorcolorat each voxelat each voxel

• Explicitly accounts for occlusionExplicitly accounts for occlusion

Voxel ColoringVoxel Coloring

• Basic idea: sweep through a voxel gridBasic idea: sweep through a voxel grid– Project each voxel into each image Project each voxel into each image in whichin which

it is visibleit is visible

– If colors in images agree, mark voxel with If colors in images agree, mark voxel with colorcolor

– Else, mark voxel as emptyElse, mark voxel as empty

• Agreement of colors based on Agreement of colors based on comparing standard deviation of colors comparing standard deviation of colors to thresholdto threshold

Voxel Coloring and OcclusionVoxel Coloring and Occlusion

• Problem: which voxels are visible?Problem: which voxels are visible?

• Solution, part 1: constrain camera viewsSolution, part 1: constrain camera views– When a voxel is considered, necessary When a voxel is considered, necessary

occlusion information must be availableocclusion information must be available

– Sweep occluders before occludeesSweep occluders before occludees

– Constrain camera positions to allow this Constrain camera positions to allow this sweepsweep

Voxel Coloring Sweep OrderVoxel Coloring Sweep Order

LayersLayers

SceneScene

TraversalTraversal

SeitzSeitz

Voxel Coloring Camera PositionsVoxel Coloring Camera Positions

Inward-lookingCameras above scene

Outward-lookingCameras inside scene

SeitzSeitz

Panoramic Depth OrderingPanoramic Depth Ordering

• Cameras oriented in many different directionsCameras oriented in many different directions

• Planar depth ordering does not applyPlanar depth ordering does not apply

SeitzSeitz


Layers radiate outwards from camerasLayers radiate outwards from camerasSeitzSeitz


SeitzSeitzLayers radiate outwards from camerasLayers radiate outwards from cameras

Voxel Coloring and OcclusionVoxel Coloring and Occlusion

• Solution, part 2: per-image mask of Solution, part 2: per-image mask of which pixels have been usedwhich pixels have been used– Each pixel only used onceEach pixel only used once

– Mask filled in as sweep progressesMask filled in as sweep progresses

Image AcquisitionImage Acquisition

•Calibrated TurntableCalibrated Turntable

•360° rotation (21 images)360° rotation (21 images)

Selected Dinosaur ImagesSelected Dinosaur Images

Selected Flower ImagesSelected Flower Images SeitzSeitz

Voxel Coloring ResultsVoxel Coloring Results

Dinosaur ReconstructionDinosaur Reconstruction72 K voxels colored72 K voxels colored7.6 M voxels tested7.6 M voxels tested7 min. to compute 7 min. to compute on a 250MHz SGIon a 250MHz SGI

Flower ReconstructionFlower Reconstruction70 K voxels colored70 K voxels colored7.6 M voxels tested7.6 M voxels tested7 min. to compute 7 min. to compute on a 250MHz SGIon a 250MHz SGI

SeitzSeitz

Voxel Coloring ResultsVoxel Coloring Results

• With texture: good resultsWith texture: good results

• Without texture: regions tend to “bulge Without texture: regions tend to “bulge out”out”– Voxels colored at earliest time at which Voxels colored at earliest time at which

projection into images is consistentprojection into images is consistent

– Model good for re-rendering: image will look Model good for re-rendering: image will look correct for viewpoints near the original onescorrect for viewpoints near the original ones

Limitations of Voxel ColoringLimitations of Voxel Coloring

• A view-independent depth orderA view-independent depth ordermay not existmay not exist

• Need more powerful general-case Need more powerful general-case algorithmsalgorithms– Unconstrained camera positionsUnconstrained camera positions

– Unconstrained scene geometry/topologyUnconstrained scene geometry/topology

pp qq

Space CarvingSpace Carving

Image 1Image 1 Image NImage N

……......

Initialize to a volume V containing the true Initialize to a volume V containing the true scenescene

Repeat until convergenceRepeat until convergence

Choose a voxel on the current surfaceChoose a voxel on the current surface

Carve if not photo-consistentCarve if not photo-consistentProject to visible input imagesProject to visible input images

Kutulakos & SeitzKutulakos & Seitz

Multi-Pass Plane SweepMulti-Pass Plane Sweep

• Faster alternative:Faster alternative:– Sweep plane in each of 6 principal Sweep plane in each of 6 principal

directionsdirections

– Consider cameras on only one side of planeConsider cameras on only one side of plane

– Repeat until convergenceRepeat until convergence


True Scene Reconstruction

Space Carving Results: African Space Carving Results: African VioletViolet

Input Image (1 of 45) Input Image (1 of 45) ReconstructionReconstruction

ReconstructionReconstructionReconstructionReconstruction

Space Carving Results: HandSpace Carving Results: Hand

Input ImageInput Image(1 of 100) (1 of 100)

Views of ReconstructionViews of Reconstruction

multiview reconstruction. why more than 2 views? baselinebaseline – too short – low accuracy –...

Documents

multibaseline stereo

intersection slide

hard slide

convex hull slide

multiview reconstruction

okutami kanade slide

error function slide

views camera