Multi-view Stereo via Multi-view Stereo via Volumetric Graph-cutsVolumetric Graph-cuts

George Vogiatzis

Roberto Cipolla

Cambridge Univ. Engineering Dept.

Philip H. S. Torr

Department of ComputingOxford Brookes University

Multi-view Dense StereoMulti-view Dense Stereo

Calibrated images of Lambertian scene

3D model of scene

Multi-view Dense StereoMulti-view Dense Stereo

Volumetric

• Two main approaches• Volumetric• Disparity (depth) map

Dense Stereo Dense Stereo reconstruction problem:reconstruction problem:

Disparity-map

• Two main approaches• Volumetric• Disparity (depth) map

Shape representation Shape representation

• Disparity-maps• MRF formulation – good optimisation

techniques exist (Graph-cuts, Loopy BP)• MRF smoothness is viewpoint dependent• Disparity is unique per pixel – only

functions represented

Shape representation Shape representation

• Volumetric – e.g. Level-sets, Space carving etc.• Able to cope with non-functions• Levelsets: Local optimization• Space carving: no simple way to

impose surface smoothness

Our approachOur approach

• Cast volumetric methods in MRF framework

• Use approximate surface containing the real scene surface• E.g. visual hull

• Benefits:• General surfaces can be represented• No depth map merging required• Optimisation is tractable (MRF solvers)• Smoothness is viewpoint independent

Volumetric Graph cuts for Volumetric Graph cuts for segmentationsegmentation

• Volume is discretized • A binary MRF is defined on the voxels • Voxels are labelled as OBJECT and

BACKGROUND• Labelling cost set by OBJECT / BACKGROUND

intensity statistics• Compatibility cost set by intensity gradient

Boykov and Jolly ICCV 2001

Volumetric Graph cuts for Volumetric Graph cuts for stereostereo

Challenges:

• What do the two labels represent• How to define cost of setting them

• How to deal with occlusion• Interactions between distant voxels

Volumetric Graph cutsVolumetric Graph cuts

(x)

1. Outer surface

2. Inner surface (at constant offset)

3. Discretize middle volume

4. Assign photoconsistency cost to voxels

Volumetric Graph cutsVolumetric Graph cuts

Source

Sink

Volumetric Graph cutsVolumetric Graph cuts

Source

Sink

Cost of a cut (x)

dS

S

S

cut 3D Surface S

[Boykov and Kolmogorov ICCV 2001]

Volumetric Graph cutsVolumetric Graph cuts

Source

Sink

Minimum cut Minimal 3D Surface under photo-consistency metric

[Boykov and Kolmogorov ICCV 2001]

Photo-consistencyPhoto-consistency

• Occlusion

1. Get nearest point on outer surface

2. Use outer surface for occlusions

2. Discard occluded views

Photo-consistencyPhoto-consistency

• Occlusion

Self occlusion

Photo-consistencyPhoto-consistency

• Occlusion

Self occlusion

Photo-consistencyPhoto-consistency

• Occlusion

N

threshold on angle between normal and viewing direction

threshold= ~60

Photo-consistencyPhoto-consistency

• ScoreNormalised cross correlation

Use all remaining cameras pair wise

Average all NCC scores

Photo-consistencyPhoto-consistency

• ScoreAverage NCC = C

Voxel score

= 1 - exp( -tan2[(C-1)/4] / 2 )

0 1

= 0.05 in all experiments

ExampleExample

Example - Visual Hull Example - Visual Hull

Example - SliceExample - Slice

Example - Slice with graphcutExample - Slice with graphcut

Example – 3DExample – 3D

Protrusion problemProtrusion problem

• ‘Balooning’ force• favouring bigger volumes that fill the visual hull

L.D. Cohen and I. Cohen. Finite-element methods for active contour models and balloons for 2-d and 3-d images. PAMI, 15(11):1131–1147, November 1993.

Protrusion problemProtrusion problem

• ‘Balooning’ force• favouring bigger volumes that fill the visual hull

L.D. Cohen and I. Cohen. Finite-element methods for active contour models and balloons for 2-d and 3-d images. PAMI, 15(11):1131–1147, November 1993.

(x) dS -

dV

S V

Protrusion problemProtrusion problem

Protrusion problemProtrusion problem

wij

SOURCE

wb

wb

GraphGraph

h

ji

wb = h3

wij = 4/3h2 * (i+j)/2[Boykov and Kolmogorov ICCV 2001]

ResultsResults

• Model House

ResultsResults

• Model House – Visual Hull

ResultsResults

• Model House

ResultsResults• Stone carving

ResultsResults

• Haniwa

SummarySummary

• Novel formulation for multiview stereo

• Volumetric scene representation

• Computationally tractable global optimisation using Graph-cuts.

• Visual hull for occlusions and geometric constraint

• Occlusions approximately modelled

Questions ?