What Does Motion Reveal About Transparency ?

Moshe Ben-Ezra and Shree K. NayarColumbia University

ICCV ConferenceOctober 2003, Nice, France

This work was supported by an NSF ITR Award IIS-00-85864

Transparency is Very Challenging

• Existence of a transparent object.• Finding its shape and pose

Real and Virtual Features

Lambertian

V1 V2

F`

V2

V1 F

Specular

F

F`

V1 V2

Transparent

Environmental Matting*

* Zongker, el al. SIGGRAPH 99,

Alternatingpattern Object Camera

• Does not recover shape and pose.• Requires controlled environment.

Shape from Polarization in Highlight*

* Saito et al. CVPR’99.

Object

CameraLight

RotatingPolarizer

• Limited to a single interface at the object’s surface.• Requires controlled environment.

N

Shape from Refraction and Motion*

* H. Murase. PAMI, 1992

Camera

Water

• Single interface only.

Fixed Pattern

Motion is Key to Transparency

Transparent Shape From Motion

Given: Views

1,,222

,

n

ne

eene zyxzyxs

And a Parametric Model (such as super-ellipse)

Recover:Shape: Values of parameters (e, n)Pose: Rotation R, Translation T

General analytic solution does not exist.

Transparency From Motion

Reversed rays are parallel to each other regardless ofthe complexity of their paths

Distantfeature

Approach: Initialization

Image Plane

Image Plane

Approach: Initial Guess

Approach: Refine

Error Function

(0,0,1)

r1,1 .. r1,n

r2,1 .. r2,n

j iji rzTRff

Feature Ray ,1 ,cosvar,,

• - Object’s shape parameter vector• R,T - Object’s pose

Simulation SetupParallel rays

from features

Transparentobject

Cameraside rays

Example (Simulation)

Single Parameter. Newton-Raphson optimization

Initial GuessSymmetric Superellipse (n=e)

Evaluation (Simulation)

GT

Both init

Pos res

Sphere

Ground Truth

Initial Guess

ComputedResult

ShapeError

GT

Both Init

Both Res

Lens

GT

Both Init

Both Res

Cube

GT

Both Init

Both Res

Water Pipe

mm15013.0:t mm

8003.0:f

25.00001.0:emm

160064.0:d

Real Experiment: Sphere

Features

Initial Guess

Setup: Initial Guess

Initial Guess: Diameter: 8 inch

Setup: Result

Ground Truth: Diameter: 3 inch. Computed: 3.18 inch

Result

Real Test: Water Filled Pipe

Features

Initial Guess

Setup: Initial Guess

Initial guess: Diameter: 200.0mm Thickness: 20.0mm

Setup: Result

Ground Truth: Diameter: 117.0mm Thickness: 3.0mmComputed: Diameter: 116.1mm Thickness: 2.3mm

Result

Real Test: Superquadric

Features

Initial Guess

Result

Ground truth: e = ? Computed: e = 0.18

Summary

Shape and poseparameters

Multiple interfaces

No Segmentationrequired

Parameterizations of Interest

• Polynomials: modeling surfaces, lenses

• CAD models: shape of industrial objects

• Dynamic models: time dependent parameters

Assumptions

• Camera parameters are known.

• Features are far* and are trackable.

• A proper model and a hypothesis (an initial guess) are given.

* One possible assumption.

Real Tests Setup

Implementation

• Features were manually selected and tracked (9 views).

• Captured rays, a model, refraction index and a hypothesis were given as inputs.

• Shape and pose were recovered using simple gradient decent (with derivatives).

The Physics of TransparencyFirst Interface:

μ1→ μ2

Second Interface: μ2 →μ1

3

11

3

N1

N2

2 2

3211 sinsin :Refraction

2

111 sin

:reflection internal Total

2

21

221

,21

R :Reflection

Parametric Shape Examples

Super-Ellipse2 parameters

Spherical Harmonics8 parameters

No analytic solution