3D Priors for Scene Learning3D Priors for Scene Learningfrom a Single Viewfrom a Single View

Diego Rother, Kedar Patwardhan, Iman Aganj Diego Rother, Kedar Patwardhan, Iman Aganj and Guillermo Sapiroand Guillermo Sapiro

University of MinnesotaUniversity of Minnesota

1

Search in 3D Workshop (CVPR 2008)Search in 3D Workshop (CVPR 2008)

AutoCalibration AlgorithmsCamera

Calibration

Moving Camera?

Tracking Local Features

Boujou, 3D-Equalizer, Matchmover, Voodoo,

…

yes no

Known Structure?

[1] D. Liebowitz and A. Zisserman, “Metric Rectification for Perspective Images of Planes.” CVPR, 1998.

[2] A. Criminisi, I. Reid and A. Zisserman, “Single View Metrology.” IJCV, 1999.

[1][1] [2][2]

Exploit Known Structure

yesExploit Known

Objects

no

Common in Surveillance

Main Idea 1

• Correct Camera Matrix → Pedestrian observations are consistent (no height change).

3

Main Idea 1

• Incorrect Camera Matrix → Pedestrian grows or shrinks.

• Pedestrians can be used as a measuring stick to calibrate the camera.

4

Main Idea 2

Image

Plane

Camera Center

3D WorldP1

Camera Matrix (PF):

PF= P1

Light Source

P1

P2

Shadow Camera Matrix (PS):

PS= P1 o P2

5

Main Idea 2

• Correct light source position → Pedestrian shadow observations are consistent .

6

• Analogously, a reflection camera can be defined.

In summary

Simultaneously Estimate: 1- Ground Positions (in 3D) 2- Horizon height (in 2D) 3- Light source position (in 3D) 4- Pedestrian height (in world units) 5- Axes scaling (to define the unit of length)

X

Z

Y

7That are Mutually Consistent and Explain the observations.

Object 3DBounding Box

Single Frame Consistency?

Camera

ConsistencyTest

Height Ground PositionObservation

Consistency(Likelihood)

Camera matrix(or Shadow Camera)

Model(3D prior)

8

3D Priors

Voxel

V4 V6

V1 V2 V3

V7 V8 V9

V5Pixel

Camera Q2Q3

Q1

Q4

Voxel Vi:- Occupied (vi = 1) with probability pi.- Blocks light if it is occupied.- Independent of other voxels.

Problems:- Discretization matters.- Equal contributions voxels ray.

Solution: Beer-Lambert law correction (predicts light attenuation in solutions),

R1,1R2,1

R6,1R3,1

- measured in [blocking probability / meter].- Same to traverse 1 big voxel or 2 of half the size.

9

2D Prior 3D Prior

3D Priors

10

Whole walking cycle Part of the walking cycle

Graphical Model

Observed PixelColors in Frame tC1 CM

Voxels(3D Prior)PV1 PV2 PVN

Pixel Class(2D Prior)

ForegroundQF1 QFM QS1 QSMShadow

Geometry(Projection)

CameraMatrixLight PositionGround Position

BackgroundShadowColor

Models

11

F1

Likelih

ood

Trajectory unregularized

Scene ParametersF1

Likelih

oodG1

12

F1

Likelih

oodG2

F1

Likelih

oodG3

Trajectory Regularized

F1

Likelihood(F2)

G1

F1

G2

F1

G3Prior

Acceleration

Optimum trajectory and F2 computed in O(NF . NG3) using Dynamic Programming.

13

F2

Search Solution Space

• Search the solution in the whole 4D parameter space:1. Horizon Height2. Y-Axis Scale.3. Light Theta4. Light Phi

14

Likelihood

CameraMatrixLight Position

F2

Optimum trajectory

Camera Matrix

Light Direction

Results

• To speed up computation, search first in the lowest resolution.

15

Half Resolution

Original Resolution

• Then, refine in the next higher, and so on.• Fast, so the whole space can be searched.

Results

16

Half Resolution

Original Resolution

• Solution superimposed.• Shape of the peak defines the types of errors.

Estimated Horizon

ResultsMetrology comparison

MeasureGround Truth

(m)

Estimated (m)

P1 4.18 4.27

P2 4.26 4.25

P3 4.38 4.36

P4 4.13 4.29

Localization errorNo Shadows

(cm)Shadows

(cm)32.3 21.5

• Mean error lower than 2% (relative to the people average height).

17

• Shadows are not disturbances, their use improve localization.

Estimated Horizon

ConclusionsPresented:

• Novel object model (not limited to people) and probabilistic framework • For camera calibration and simple lighting estimation.• Using the Foreground and the Shadows.• That works in situations where other methods fail.

18

Learning 3D Priors

V4 V6

V1 V2 V3

V7 V8 V9

V5

C1 C2

Method of Moments, yields one Equation per ray:

This is the Fan Beam Radon transform.Just solve linear system.

Silhouette in frame t Average

19

3D Priors

20

Search Solution Space

x

y

z

21