shape and visual appearance acquisition for photo...

Shape and Visual Appearance Acquisition for Photo-realistic Visualization

Fabio Ganovelli & Massimiliano Corsini

Speaker: Massimiliano Corsini

Visual Computing Lab, ISTI - CNR - Italy

Appearance Acquisition • 2.1 Introduction Light-matter interaction. Radiometry in a nutshell. Bidirectional Reflectance Distribution Function (BRDF) and Bidirectional Surface Scattering Reflectance Distribution function (BSSRDF).

• 2.2 BRDF measurement Gonioreflectometer. Image-based estimation. Analytical BRDF models. BRDF factorization (texture decomposition, spherical harmonics, Haar wavelets).

• 2.3 Reflectance as N-dimensional function estimation Taxonomy. Plenoptic function. Light field. Reflectance Field. Bidirectional Texture Function (BTF). Reflection Transformation Imaging: polynomial (PTM) and hemispherical harmonics approximation (HSH).

• 2.4 Texture registration Multi-modal matching through feature-based (keypoints, lines) and statistical methods (mutual information). Statically-fixed shading problems. Color mapping strategies. Intrinsic images (brief notes).

Introduction

3D Scanning pipeline

• 3D Scanning:

– [ Acquisition planning ]

– Acquisition of multiple range maps

– Range map filtering

– Registration of range maps

– Merge of range maps

– Mesh Editing

– Geometry simplification

– Capturing of visual appearance (color/reflectance properties)

Light and matter

• Light is the flow of radiant energy. A light source emits such energy. This energy travels from the source to the objects and interacts with them bouncing until an equilibrium is reached.

• We see all stable because this process is very fast due to the speed of the light.

Light-matter interaction

Radiometry in a nutshell

• Concepts:

– Solid angle

– Radiant flux

– Ingoing and outgoing Radiance

– Irradiance

Solid angle • It represents the angular dimension of an infinitesimal conoid

along a given direction.

• It can be seen as the conjunction representation of a direction and an infinitesimal area on a sphere.

• The unit measure is steradians (sr).

Solid angle

• It can be seen as the extension of the concept of angle at the 3D space.

• In fact, ϴ is measured (in radians) as the ratio s / r where s is the length of the arc or ray r under ϴ.

• Analougously, the solid angle Ω is measured (in steradians, sr) as the ratio A / r2 where A is the area of the spherical surface of ray r under Ω.

• Examples: – Square angle: 2 π r / 4 r = π / 2 radians

– Solid angle of an hemisphere: 4 π r2 / 2 r2 = 2 π steradians

Radiant flux

• Radiant flux (watt) is the amount of light passing through (leaving or reaching) an area or a volume in the unit of time:

Irradiance

• Irradiance (E) (measured in watt/m2) is the radiant flux incident on a surface element per unit area:

• Exitance (indicated also with E) is the flux leaving a surface per unit area:

Radiance • Radiance (L) (watt/m2sr) represents the flow

of radiant energy from (or to) a surface per unit projected area and per unit solid angle:

Radiance and Irradiance

• The integral of the incoming radiance along all the possible directions corresponds to the irradiance:

Rendering a 3D scene

• Rendering is the processing to generate a synthetic image starting from some data.

• Rendering of realistic 3D scenes is obtained by defining the light sources, the geometry of the scene and the reflection properties of each surface of the scene.

• A common way to specify the reflection property of a surface is the Bidirectional Reflectance Distribution Function (BRDF).

Bidirectional Reflectance Density Function (BRDF)

• The BRDF function is defined as the ratio between the radiance and the (directional) irradiance:

Bidirectional Reflectance Density Function (BRDF)

• Taking into account the relation between radiance and irradiance it is possible to write the BRDF as a function of ingoing and outgoing radiance only

BRDF parameterization

BRDF example

• BRDF is a 4D function

• Here, a visualization of a 2D slice:

BRDF Properties

• Energy conservation:

• Helmholtz reciprocity (simmetry of the light transport):

Isotropic vs Anisotropic BRDF

• A number of real world surfaces exhibit invariance with respect to therotation of the surface around the normal vector at the incident point.

Anisotropic

Isotropic

Materials

Diffuse reflection Mirror reflection

Anisotropic reflection

Subsurface scattering

Layered-material

The rendering equation

• A rendering algorithm, to generate the synthetic image must compute radiance contributions from each surface point to the virtual camera.

Warning!! All these lighting contributions are local (!)

Generalized Rendering equation

Reflected light Emitted light

Global and local effects

caustics

shadows Color

bleeding

Images from

Henrik Wann Jensen

Generalized Rendering equation

Visibility Term

Bidirectional Scattering Surface Reflectance Distribution Function (BSSRDF)

• BSSRDF is an 8D function, BRDF is a 4D function

• It can be plugged in the rendering equation

BRDF

Measurement

Acquisition through Gonioreflectometer

• A gonioreflectometer consists of a moving light source and a photometric sensor.

Orientable material sample Moving light source

Figure from Hendrik P.A. Lensch, “Efficient, Image-Based Appearance

Acquisition of Real-World Objects. PhD Thesis, 2003.

Acquisition through Gonioreflectometer

Limitation

• Gonioreflectometers are relatively slow since the sensor and the light source have to be repositioned for every pair of incident and outgoing direction (one sample at time).

• BRDF measured “a single material” at a time spatially-varying materials require a lot of effort to be captured.

Image-based Acquisition

• Using images, instead of using other sensors, permits to capture more BRDF samples at time.

Figure from Hendrik P.A. Lensch, “Efficient, Image-Based Appearance

Acquisition of Real-World Objects. PhD Thesis, 2003.


• High Dynamic Range (HDR) images may be preferably used

• Images should be aligned with the geometry to obtain the surface normal

• Intensive data processing


Stanford Spherical Gantry

Image-based Acquisition University of Virginia Spherical Gantry


light source

camera

black felt

Minerva head

calibration target

On-site acquisition setup (MPI Saarbucken, Goesele et al.)

Representation

• The most common representation is to store the data acquired in tabulated form (discretized directions) and interpolate.

• Enormous amount of storage to get high accuracy (!)

• Undersampling problems.

Limitations

• Flexibility

– The moving setup is always very cumbersome.

– How to cope with large objects ?

• Time required (several hundreds images to process)

• Controlled lighting conditions ?

– Field conditions have a lot of difficulties (museums, archeological sites)

Parametric Acquisition/Representation

• Parametric acquisition consists in assume a model for the BRDF and estimate its parameters (nonlinear system) sparse vs dense acquisition.

• If a particular model is not assumed, the BRDF can be factorized with different basis functions and its coefficients estimated.

• Parametric representation/factorization can be used also to obtain a compact form starting from a dense acquisition.

BRDF Models

• DIFFUSE REFLECTION: light is equally diffuse in every direction due to the uniform random microfacet of the surface

• Lambertian materials.

Diffuse reflection

• Lambertian Law: a diffusive surface reflects an amount of light which depends on the incident light direction. Maximum when perpendicular, then it reduces as cosine of the angle between the normal and the direction of incidence.

BRDF Models

• BLINN-PHONG REFLECTION MODEL: it is composed by an ambient component plus a diffuse component plus a specular reflection component.

• It is used to model glossy materials.

Blinn-Phong Model – Half Vector

BRDF Models

• LAFORTUNE REFLECTION MODEL: It consists of a diffuse terms and a set of specular lobes.

• Depending on the coefficients, it can exhibit different reflection behaviours like retro-reflection and anisotropic reflection.

BRDF Factorization

• BRDF can be factorized in several ways:

– Texture decomposition

– Spherical Harmonics

– Haar Wavelets

Texture Decomposition

• This approach ([Kautz99], [McCool2000]) use the general concept of separable decomposition approximation to obtain a lower dimension of the 4D BRDF.

[Kautz99] Jan Kautz and Michael D. McCool. 1999. Interactive rendering with arbitrary

BRDFs using separable approximations. In Proc. of the 10th Eurographics conference on

Rendering (EGWR'99).

[McCool01] M. McCool, J. Ang and A. Ahamd, “A Homomorphic Factorization of BRDFs

for High-Performance Rendering” In Proc. of SIGGRAPH, pages 171–178, August 2001.


• To obtain this, a Singular Value Decomposition (SVD) can be employed.

• SVD consists in decomposing a matrix M as the product of three matrix such that M = USVT where U = [uk] and V = [vk] are orthonormal matrices and S = diag(σk):


• Rewriting the BRDF as:

• And defining the matrix M as:


• Decomposing M so defined, and truncating the series at N the approximation is obtained:

Result

Ward Anisotropic Model

SVD approximation (N = 5)

Spherical Harmonics Decomposition

• We introduce here the spherical harmonics as basis functions to approximate functions defined on a spherical domain.

• The following material is based on ”Spherical Harmonic Lighting: The Gritty Details” Robin Green, R&D Programmer, Sony Computer Entertainment America.

Linear basis function

Orthonormal basis:

Linear Basis Function

Projection:

Spherical Harmonics

• The idea is to use spherical harmonics as a linear basis functions for the BRDF.

• Spherical harmonics are based on the Associated Legendre Polynomials.

Spherical Harmonics - Definition

Legendre Polynomials

Spherical Harmonics

l = 0

l = 1

l = 2

BRDF decomposition with SH

• The BRDF is projected on the basis to obtain the coefficients (as previously explained).

• The final form of the BRDF is:

BRDF decomposition with SH

• Rendering algorithms become very efficient from a computational viewpoint convolution between lighting and the BRDF [Ramamoorthi2001][Kautz2002].

• In [Ramamoorthi2001] also important theoretical results on BRDF acquisition.

[Ramamoorthi2001] Ravi Ramamoorthi and Pat Hanrahan. 2001. A signal-processing

framework for inverse rendering. In Proceedings of the 28th annual conference on

Computer graphics and interactive techniques (SIGGRAPH '01). ACM, New York, NY, USA,

117-128.

[Kautz2002] Jan Kautz, Peter-Pike Sloan, and John Snyder. 2002. Fast, arbitrary BRDF

shading for low-frequency lighting using spherical harmonics. In Proceedings of the 13th

Eurographics workshop on Rendering (EGRW '02).

Wavelet Factorization

• Wavelet can de used to approximate the BRDF function efficiently.

• We examine here the work of Matusik et al. [Matusik2003].

[Matusik2003] W. Matusik, H. Pfister, M. Brand, and L. McMillan. Efficient Isotropic BRDF Measurement. Eurographics Symposium on Rendering 2003.

Haar Wavelets

• Consider an 1-dimensional image.

• Considering the vector space Vj defining all the piecewise-costant (on 2j intervals) functions defined over the [0,1) interval.

• This space is such that:

Haar Wavelets

• A basis for the vector space Vj is the “box basis” define as:

Haar Wavelets

• Wavelets are used to pass from the space Vj to the space Vj+1 they define the new vector space Wj which contains all the functions of Vj+1 which are orthogonal (under the standard inner product) to Vj , i.e. Wj is the orthogonal complement of Vj.

Haar Wavelets

• The wavelet corresponding to the box basis functions are called Haar wavelets and are defined as:

Haar Wavelets BOX BASIS V2

HAAR WAVELET W1

From Wavelets for Computer Graphics: A Primer. Eric J. Stollnitz, Tony D. DeRose, and

David H. Salesin. IEEE Computer Graphics and Applications, 15(3):76-84, May 1995.

1D function approximation



2D image approximation

INCREASING COEFFICIENTS



Acquisition setup


The Rusinkiewicz BRDF coordinate system.


• These three angles are discretized such that 90 x 90 x 360 samples are acquired.

• 90 x 90 x 180 samples are sufficient due to the simmetry of light transport (Helmholtz reciprocity) for a total of 1,458,000 samples (!)

• The data are remapped in a 256x256x256 3D array of data.

• Such data are approximated with Haar wavelets.


• The coefficients to obtain an error less than 3% is retained, the other are discarded.

• After the acquisition of 100 different BRDFs 69,000 coefficients are in common (4.7% of the initial storage 1 : 25 compression).

Results

Only the Common Wavelet Basis is used (!)

Thanks for the attention.

Question ?

shape and visual appearance acquisition for photo...

Documents