Surface Light Fieldsfor 3D Photography

Daniel N. WoodUniversity of Washington

SIGGRAPH 2001 Course

Collaborators(and co-authors on SIGGRAPH 2000

paper)Daniel Azuma Wyvern Aldinger

Brian Curless Tom DuchampDavid Salesin Werner Stuetzle

University of Washington

Outline

1. Surface light field representation

2. Compression primer

3. Surface light fields for 3D photography• With details of compression• And a preliminary look at a new

compression algorithm

Surface light fields

Lumisphere-valued “texture” maps

Lumisphere

Surface light fields in flatland

s

Surface light fields in flatland

s

Surface light fields in flatland

s

Compression primer

• Singular value decomposition(or principal components analysis)

– Handling color

– Using regions

– Reflection parameterization

• Vector quantization

• Others… (Wavelets, DCT)

Singular value decomposition

SLF

U VT

SVD in two matrices

EigenTextures

EigenLumispheres

SLF

First eigenvectors

EigenTextures

EigenLumispheres

SLF First eigentexture and corresponding first eigenlumisphere

Truncated SVD

Outer product of first eigen-texture and first eigen-lumisphere is closest rank 1 (separable) matrix.

=

Eigen-lumisphere

Eig

en

-textu

re

~~

Handling color

Separatematrices

Colors insurface texture

(columns)

Colors indirections

(rows)

Handling color

Separatematrices

Nishino et al.Wood et al.Chen et al.

Colors insurface texture

(columns)

Colors indirections

(rows)

SVD in color

EigenTextures

EigenLumispheres

SLF

Zooming in…

EigenTextures

EigenLumispheres

SLF

Zoom into important vectors

... ...SLF

Reconstruction using SVD

Original Rank 1 Rank 7 (1:20)

Surface light field structure

Rows are points on surface

What are the columns? And, can they be made more coherent?

Surface light field structure

Rows are points on surface

Increasing column coherence:

1. Break into regions, and / or

2. Use reflection parameterization

Separate SVD for regions

EigenTextures

EigenLumispheres

EigenTextures

EigenLumispheres

Reconstruction using regions

Original One region(Rank 5)

Two regions(Rank 5)

Reflection reparameterization

Reflection reparameterization

Reflection reparameterization

Reflection in flatland*

*sort of

Un-reflected Reflected

Reflection doesn’t happenin the plane

Un-reflected Reflected

Reflection in 3 space

Un-reflected

Reflected

Reflection in “flatland”

Un-reflected Reflected

Original Rank 5 Original Rank 5

Other compression strategies

• Discrete cosine transform[ Miller et al. 1999

– Eurographics Workshop on Rendering]

• Vector quantization[ Wood et al. 2000 - SIGGRAPH ]

• Wavelet decomposition[ Magnor and Girod 2000 - SPIE VCIP]

Vector quantization (unreflected)

Uncompressed Codebook Vector-quantized

Vector quantization (reflected)

Uncompressed Codebook Vector-quantized

Surface light fields for 3D photography

(SIGGRAPH 2000)Goals

Rendering and editing

InputsPhotographs and geometry

RequirementsEstimation and compression

Overview

Dataacquisition

Estimationand

compressionRendering

Editing

Overview

Dataacquisition

Estimationand

compressionRendering

Editing

Scan and reconstruct geometry

Reconstructed geometryRange scans(only a few shown . . .)

Take photographs

Camera positions Photographs

Register photographs to geometry

GeometryPhotographs

Register photographs to geometry

User selected correspondences (rays)

Parameterizing the geometry

Base mesh Scanned geometry

Map

Sample base mesh faces

Base mesh Detailed geometry

Assembling data lumispheres

Data lumisphere

Overview

Dataacquisition

Estimationand

compressionRendering

Editing

Pointwise fairing

Faired lumisphereData lumisphere

Pointwise fairing results

Input photograph Pointwise faired(177 MB)

Pointwise fairing

Many input data lumispheres Many faired lumispheres

Compression

Small set of prototypes

Compression / Estimation

Small set of prototypesMany input data lumispheres

Median removal

+

Reflected

Median(“diffuse”)

Median-removed(“specular”)

+

Median removal

Median values Specular Result

Function quantization

Codebook of lumispheres

Input data lumisphere

Lloyd iteration

Input data lumispheres

Lloyd iteration

Codeword

Lloyd iteration

Perturb codewords to create larger codebook

Lloyd iteration

Form clusters around each codeword

Lloyd iteration

Optimize codewords based on clusters

Lloyd iteration

Create new clusters

Function quantization results

Input photograph Function quantized(1010 codewords, 2.6 MB)

Principal function analysis

Subspace of lumispheres

Input data lumisphere

Prototype lumisphere

Principal function analysis

Approximating subspace

Prototype lumisphere

Principal function analysis

Principal function analysis

Principal function analysis

++

+

Median

PFA decomposition

Principal function analysis

+ + …

Median

PFA decomposition

=Final

approximation

Principal function analysis results

Input photograph PFA compressed(Order 5 - 2.5 MB)

Compression comparison

Pointwise fairing(177 MB)

Function quantization(2.6 MB)

Principal functionanalysis (2.5 MB)

Rewind…what we didn’t want to do

Instead, a middle ground

Regularly sampled directions,but not all there.

New method:Principal components with missing

data

SLF with missing data

Hallucinatedata

Use either fairing or pair-wise present covariance matrix to fill holes

Hole-filled SLF

New method:Principal components with missing

data

SLF with missing data Low rankapproximation

Hallucinatedata

Find best low rank

approximation

Hole-filled SLF

New method:Principal components with missing

data

SLF with missing data Low rankapproximation

Hallucinatedata

Improvehallucination

Preliminary comparison

Principal function analysisOrder 3, RMS error 26.9

SVD with missing dataOrder 3, RMS error 26.1

Overview

Dataacquisition

Estimationand

compressionRendering

Editing

Interactive rendererscreen capture

Overview

Dataacquisition

Estimationand

compressionRendering

Editing

Lumisphere filtering

Original surface light field Glossier coat

Lumisphere filtering

Rotating the environment

Original surface light field Rotated environment

Deformation

Original Deformed

Deformation

Summary

1. Estimation and compression• Function quantization• Principal function analysis

2. Rendering• From compressed representation• With view-dependent level-of-detail

3. Editing• Lumisphere filtering• Geometric deformations and transformations

Future work

• Better geometry-to-image registration

• Derive geometry from images

• More complex surfaces (mirrored, refractive, fuzzy…) under more complex illumination

Acknowledgements

• Marc Levoy and Pat Hanrahan– (Thanks for the use of the Stanford

Spherical Gantry)

• Michael Cohen and Richard Szeliski

• National Science Foundation

Nearly the end

For more information

http://graphics.cs.washington.edu/projects/slf

Talks, papers, … and raw data.