rendering & reconstructing under complex brdf ’ s

Rendering & Reconstructing Under Complex BRDF’s

AGENDA Introduction

Motivation Basic concepts

Photometric Stereo

Image Based Rendering (IBR)

Summary

IntroductionIn this lecture we will discuss the way

lightinteracts with matter and how to improverealism in CV and in other related areas

such ascomputer graphics, using this knowledge.

Motivation (1) – Constructing Geometry of an Object

Left: no light .

Right: A spot light is pointing down on the object from above and behind, reflecting off the surface of the sphere.

This simple highlight gives the viewer a completely different reading of the scene.

Motivation (2) – Giving Clues To an Object's Material

The objects reflects highlights differently.

Left: soft - as though the object were made of chalk.

Right: glossy - creates the perception of very shiny plastic.

Motivation (3) – Image Based Rendering - Changing View Point

Motivation (4) – Image Based Rendering - Changing Lights Direction

Given 1 object image (face) taken under single light source (unknown direction).

Motivation (4) – Image Based Rendering - changing lights

render same face under new lighting direction:

Basic Concepts Radiometry Solid angle Radiance Irradiance Radiosity & Exitance BRDF “Helmholtz” Reciprocity Rule Isotropic vs. Anisotropic Special BRDF

Diffuse surface & Lambertian Low Specular surface & Fong Model

Local vs. Global Illumination

Radiometry

Deals with the following Questions: How do we measure light? How “bright” will surfaces be? How does light interacts with surfaces?

Surface material

Radiometry – Some Answers..

How much light the surface receives

LN

How much of the received light is reflected

Brightness of a surface

Example

Same light source hitting two different surfaces:

Light hits the surface directly Light hits the surface at an angle

As a result the right surface receives less light per square inch !

Light Behavior

Absorbed transmitted

reflected

Combination

Fluorescence

Absorbing light at one

wavelength, and radiate

light at different

wavelength.

Simplifying Assumptions The light leaving a point on a surface is

due only to light arriving at this point.

No Fluorescence

Surfaces do not generate light internally - treating sources separately.

Radiometry – Formalization

(i,i)

(o, o)

LN

R

V

Radiometry – Formalization

Around any point there is a hemisphere ofdirections:

Spherical Coordinates

x = r sincos y= r sin sinz= r cos


Diffuse surface & Lambertian Low Specular surfac & Fong Model


Intro. To Solid Angle

Light is form of energy

light is measured in terms of flow through an area

light coming from a single direction

light coming from a small region

Solid Angle - Definition

The solid angle is the area of the projection of the object onto the unit sphere.

Units : steradians, abbreviated sr.

Solid Angle of a Small Patch

The solid angle subtended by a small patch area dA is:

2

cos

r

dAd

ddd sin

dA

Basic Concepts Radiometry Solid angle Radiance & Irradiance Radiosity & Exitance BRDF “Helmholtz” Reciprocity Rule Isotropic vs. Anisotropic Special BRDF



Radiance

Denoted: L(x,,) Units: Wm-2sr-1 .

Amount of energy traveling at some point in a specified direction, per unit time, per unit area perpendicular to the direction of travel, per unit solid angle

Radiance is Constant Along a Straight Line

Assuming light does not interact with the medium through which it travels – i.e. that we are in .

How much light is arriving at a surface.

A surface experiencing radiance L(x) coming in from d experiences irradiance:

Note: While the radiance is per area perpendicular to the direction of travel, the Irradiance is not.

Units: W*m -2

Irradiance

dxL cos,, ,,xEi

d cos

d

dLB cos),,(xx

Radiosity

Total power leaving a surface, per unit area on the surface.

To get it, integrate radiance over the hemisphere of outgoing directions: X

Exitance Light sources emit light, they are

sources of radiance

Exitance is the equivalent of radiosity for emitters:

dLE e cos),,(xx

Intuition: BRDF is a function that specifies the ratio between the incident light in one direction and the emitted light in a second direction.

The function defines properties of the surface (shininess,..)

BRDF – more formally

the ratio of the radiance in the outgoing direction to the incident irradiance at a point on the surface

Range: [0,infinity] (surprising?)Units: inverse steradians = sr -1

dL

L

Ei

L

iiii

ooo

ii

oooiioobrdf cos,

,

,

,,,,,

Outgoing radiance

irradiance

Helmholtz Reciprocity Rule

brdf is symmetric:

ooiiiioo brdfbrdf ,,,,,,,

=(i,i) (i,i)

(r,r) (r,r)

Isotropic vs. Anisotropic

Isotropic reflection - reflection that does not vary as the surface is rotated about the normal (the angle).

Isotropic – useful assumption.

Special BRDFs

Diffuse Light Illumination that a surface reflects

equally in all directions.

BRDF is constant:

The brightness is independent of the observer position.

Also called “Lambertian” Reflection.

xx brdfiioobrdf ,,,,

Ideal Diffuse Surfaces – ALBEDO definition

Albedo - The fraction of the incident radiance in a given direction that is reflected by a point on diffuse surface (in all possible directions).

Denoted d.

Also called diffuse reflectance.

xx

xx

brdf

oobrdf

oobrdfd

d

d

cos

cos

The radiant energy I from a diffuse surface:

Lambert’s Law

Unit normal

intensity of light source

Light unit vector

radiant

albedo

NLNLII Lˆ*ˆˆ**

L

N

Specular Surface

Light reflected from the surface unequally to all directions.

These are the bright spots on objects (polished metal, apple ...).

Phong Model – Specular Light

• How much reflection light you can see depends on where

you are

Different BRDF

PerfectlyPerfectly SpecularSpecular “ “Mirror”Mirror”

n n ∞ ∞

Different BRDFIncidentIncidentLightLightRayRay

SurfaceSurfaceNormalNormalReflectedReflected

LightLight

SlightlySlightly scatteredscattered Specular:Specular:

Different BRDF

perfectly Diffuse

IncidentIncidentLightLightRayRay

SurfaceSurfaceNormalNormal

Different BRDF

Combination ofDiffuse and Specular

IncidentIncidentLightLightRayRay

SurfaceSurfaceNormalNormal


Local illumination Everything is lit only by light sources

Global illumination Everything is lit by everything else

Local illumination

global illumination

O.K. So Now What?!

Photometric Stereo

The Problem

Given a set of images of the same object, from the same view point, under different given light sources…

Can We Recover The 3D Shape of The Object?

General Schema

Recover surface normal

Recover shape out of normals

Overview

Classic approach Recover surface normal when the

light is known Recover surface normal when the

light is unknown

New idea – “shape by example”

Since we keep the camera and the scene intact, each image pixel of the three images correspond to the same 3D point :

Classic Approach - Basic Idea

Classic Approach

Assumptions: “Lambertian” surfaces Point light sources that are distant

Lambert’s law:

normal

intensity of light source

light vector

Image intensity

albedo

NLNLII Lˆ*ˆˆ**

N

L

Vector Form

For each pixel p, the normals are the same and we

get 3 conditions respectively:

i = 1,2,3For each pixel p we get a vector :

pNLpN

L

L

L

I

I

I

Ip p

p

p

p

p

p

p

ˆ**ˆ**

3

2

1

3

2

1

Lp = 3*3 matrix

NpiLIipp **

Simple Case – Light is Known

In that case we get for each pixel:

pNLIp p ˆ**

IpLpN p1ˆ*

IpL

IpLpN

p

p

1

1

ˆ

N is a unit vector

More Complex – Light is Not Known - Factorization

For each pixel:

For f frames and p pixels, we get:

pNLIp p

*

pLp LIL ˆ*

NpN ˆ*

LNI f*p intensity matrix F*3 light

matrix3*p normals matrix

Factorization

If there is no noise, then rank (I) = 3.

By Singular Value Decomposition (SVD):

But, there are many solutions, since:

NLVUI T ~*

~

NLNAALNLI~~

*~~~

**~~

*~ 1

Shape and Materials by Example:A Photometric Stereo Approach

Aaron Hertzmann, University of Toronto

& Steve Seitz, University of Washington

The Idea

Consider the simple case of two objects photographed together

Orientation-Consistency Cue

Suppose, we would like to determine the shape of the bottle.

Under the right conditions it holds that:

“Two points with the same surface orientation reflect the same light toward the viewer”

Orientation-Consistency Cue

For example, if a point is in highlight on the bottle, then it must have the same surface normal as the region in highlight on the sphere.

Ambiguous But, what happens when:

There are multiple highlights on the sphere? Multiple points on the sphere with the same intensity.

Solution: taking pictures under more lighting conditions.

More Lighting Conditions…

General Assumptions

• At least one reference object of the same or similar material must be imaged under the same illumination.

• The shape of the reference object (sphere) is known.

• Lighting is distant.

• The camera is orthographic.

• Local illumination only – shadows, intereflection and so on are ignored.

Formalization

Given multiple images of reference and target objects - same viewpoint, different illuminations :

Ir1 , . . . , Irn - the reference images.

It1, . . . , Itn - target images.

Corresponding reference Iri and target Iti images are

captured under the same illumination.

Let Ir1,p be the intensity of pixel p in

reference image #1.

Define vector V to be the intensities at a same pixel over the n images. Vr

p = [Ir1,p , . . . , Irn,p]T

Vtp = [It1,p , . . . , Itn,p]T

Formalization – cont.

Basic Algorithm

Given Pixel p on the target object look for pixel q on thereference object s.t. ||Vp – Vq|| is minimized.

Pixels p and q have the same normal if

|| Vp – Vq|| is minimized.

Determining Normal of a Point

- =

Determining Normal of a Point

- =

- =

Reference objectTarget object

Distant light

Limitations of Basic Algorithm (1)

Must have uniform BRDF for each point on target object

Limitations of Basic Algorithm (2)

Reference object Target object

Distant light

Reference and target object are made of the same material

Target Object Has Different BRDF’s To overcome it, target object must be

either pure diffuse or pure specular. For diffuse object use lambert’s low:

light

ppt

pt lnI **

albedo normal

light

ppt

pt lnI **

Light source (direction & intencity)

light

pr

pr lnI **

Target object has different BRDF’s – The Trick

p and q have the same normal if

is minimized

qr

q

pt

pt

V

V

V

V r

Target Object Made of Multiple Materials

Assume every material can be represented as linear combination of k (base) materials .

Use k (independent) reference objects.

Each pixel in target material can be represented as a linear combination of the k reference materials.

Find material coefficient and pixel q for best corresponding with pixel p.

Advantages

The BRDF may be arbitrary.

BRDF may vary over the surface.

The illumination may be unknown.

Any number of light source.

Result – Uniform BRDFBottle Reconstruction

8 in total

Result – Unifrom BRDFVelvet Reconstruction

reference target

14 in total

Result – Multiple materialsCat Reconstruction

Gray, diffuse sphere Ceramic cat

Shiny, black sphere

13 in total

Image Based Rendering

Image Based Rendering (IBR) Input: Dense set of images from different

viewpoints or different illumination.

Goal: Create pictures of synthetic scenes under new illumination conditions or from new viewpoints.

The picture should be undistinguishable from photographs of real environments.

Rendering Algorithms

Differ in the assumptions made regarding lighting and reflectance in the scene and in the solution space.

local vs. global illumination algorithms.

view dependent vs. view independent solutions.

Agenda Local illumination, view dependent

algorithm for rendering a human face

Global illumination, view independent algorithm for acquiring the reflectance properties of complete scenes

Summary

Local IlluminationView Dependent

Acquiring the Reflectance Field of a Human Face

Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, Mark

SagarSIGGRAPH 2000

Goals Acquire images of the face from 2

viewpoints under a dense sampling of incident illumination directions.

Construct a reflectance function for each pixel.

Render the face under new illumination conditions.

Challenges• Complex and individual shape of the face.

• Subtle and spatially varying reflectance properties of the skin.

• Complex deformation of the face during movement.

• Viewers are extremely sensitive to the appearance of other people’s faces.

Light Stage

Constructing Reflectance Functions

For each pixel location (x, y) in each camera, that location on the face is illuminated for 64 x 32 directions of and .

For each pixel we keep all radiance values under 2000 different illumination direction (reflectance function).

Rxy(, ) corresponding to the ray through the pixel (x,y) with illumination direction (, ) .

Novel Form of Illumination Rxy(, ) represents how much light is

reflected towards the camera by pixel (x,y) as a result of the illumination from direction (, ).

Solid angle covered by each of the illumination

directions

Illumination Map

One can capture illumination at a point in the real world with a single spherical “photograph” or environment map.

Two different projections of the same spherical image

Novel Form of Illumination

Grace Cathedral in San Francisco ,St. Peter's Basilica, The Uffizi Gallery in Florence ,the UC Berkeley Eucalyptus Grove and a synthetic test environment.

Results

Watching a movie…

Render a human face - summary

2000 images taken from a fixed viewpoint under different illumination conditions

A reflectance function for each pixel was created using these images

A linear combination for each pixel together with the illumination map enable rendering the face from natural illumination conditions

Global Illumination View Independent

Global illumination , view independent

Some basis The Global illumination equation Basic Radiosity methods

New idea – Inverse Global Illumination

Summary

Recall

Radiance – Amount of light.

BRDF – Ratio between out going radiance and coming irradiance.

Radiosity - Total power leaving a surface.

Exitance – Total power leaving a point on a light source.

Global Illumination Equation

Total power leaving a point in a specified direction:

Radiance Exitance BRDF Irradiance

dLLL ioobrdfooeoo cos),,(),,,,(),,(),,( xxxx

Total radiance leaving the point x on the surface

in direction (o,o)

Radiance emitted from the surace at point x in direction

(o,o) , equal zero for non light sources

The fraction of the incoming irradiance at point x, in direction (,) which is

reflected by the surface in direction (o,o)

The incoming irradiance at point x, in

direction (,)

Total light reflected by the surface

Basic Radiosity Methods

Originally introduced in 1950s as a method for computing radiant heat exchange between surfaces

Radiosity Algorithms

Solve the global illumination equation under a restrictive set of assumptions All surfaces are

perfectly diffuse Surfaces can be

broken into patches with constant radiosity

Assumptions allow us to simplify the global illumination equation

The Radiosity Algorithm For Image Synthesis

Form Factor Calculation

Solution to the system of equations

VisualizationRadiosity solution

Radiosity image

Input of scene geometry

Input of reflectance properties(albedo for each patch)

Viewing direction

Radiosity algorithm

The Form Factor

The form factor Fij is the fraction of the total radiance leaving a patch i which is received by patch j

A function of the scene geometry only

Sum to unity1,

1

N

j ijFi

The Discrete Radiosity Equation

N

jjijiii BFEB

1

dLLL ioobrdfooeoo cos),,(),,,,(),,(),,( xxxx

From Total radiance leaving point x in a

specific direction to the radiosity leaving a patch

i

From the radiance emitted by point x

in a specific direction to the exitance leaving

patch j

From BRDF to albedo

From integration of irradiance

over the hemisphere to

the sum over all the patches

The Discrete Radiosity Equation

B = E + F x B

E = MB where M = (IN - F)

N

jjijiii BFEB

1

NNNNNNNN

N

NN B

B

B

FFF

FF

FFF

E

E

E

B

B

B

2

1

21

222212

11121111

2

1

2

1

The Discrete Radiosity Equation (cont)

E = M x B

Dimension of M is given by the number of patches in the scene: N xN It’s a big system Iterative solution

NNNNNNNN

N

N B

B

B

FFF

FF

FFF

E

E

E

2

1

21

222212

11121111

2

1

1

1

1

Radiosity Algorithm – Pro & Cons Needs only be calculated once for

different viewing conditions

when geometry changes there is a need to recalculate the form factors

If lighting changes then only the equation needs resolving

Radiosity Algorithm - Results

Walking through the scene

Inverse Global IlluminationRecovering Reflectance Models

of Real Scenes from Photographs

Computer Science DivisionUniversity of California at

Berkeley

Yizhou Yu, Paul Debevec, Jitendra Malik & Tim Hawkins

Global Illumination

Reflectance Properties

Radiance Images

Geometry Illumination

Inverse Global Illumination

Reflectance Properties

Radiance Images

Geometry Illumination

Inverse Global illumination Outline

Motivation Goal Partial solutions

Inverse Radiosity Specular Parameters

Mutual Illumination Results Conclusion

Motivation Most Image Based Rendering

methods allow novel viewpoints, but not changes in lighting.

This paper shows recovery of reflectance parameters of a scene.

Can then relight scene.

Motivation - cont

Many authors have previously recovered reflectance parameters. e.g., Specular and diffuse parameters Spatially varying BRDFs

However, this is done in laboratory with controlled illumination

Good for individual objects, but not for an entire scene

Goal Estimation of the reflectance

properties of all surfaces in the scene at once.

Surfaces are illuminated in situ rather than as isolated samples.

Perform all of this from a relatively sparse set of photographs.

Simplifying Assumptions

No transmission Known geometry Known light source positions Known cameras positions Radiance maps Specular reflectance parameters

constant over large surface regions

Simplifying Assumptions - cont Each surface point captured in at

least one image

Each light source captured in at least one image

Image of highlight in each specular surface region in at least one image

First Step Toward The Full Solution

Inverse radiosity

Pure diffuse scenes

The environment is broken into patches with constant diffuse albedo

Inverse Radiosity

Input: Scene Geometry Lighting conditions Radiance distribution

Output: Diffuse albedo at each patch in the

environment

Input -Geometry and Camera Positions

Input - Light Sources

j

ijjFB

ijjFB

Inverse Radiosity

j

ijjiii FBEB iB

? ii EB ?iii EB

jB

)/()( j

ijjiii FBEB

Second Step Toward The Full Solution

Local illumination

Single surface

Single known light source.

Uniform BRDF’s – allows both diffuse and specular reflection

Local Illumination

Radiance Li obtained by a

measurement of each

Irradiance Ei obtained by known light source

Goal BRDF estimation using (Li , Ei)

Ei Li

Ward Reflectance Model Variant of Phong model. Using Ward’s model, the radiance of a patch is

given by:

d - albedo

s K(, ) - specular term

K - nonlinear function of , the incident and viewing directions .

- surface roughness (blur) vector.

iisd

i IKL

),(

Ward Reflectance Model

isotropic specular highlight

( is scalar)

anisotropic specular highlight( is 3-component vector)

Local Illumination

Li is radiance at Pi

iisd

i IKL

),(

Ei is irradiance at Pi

i is light & camera position 3 or 5 unknown parameters: d, s

and to be estimated

iE

i

Li

Local Illumination

One equation for each pixel of surface in image

Can be solved using nonlinear optimisation

2

,,)),(( min arg iisi

i

di IKIL

sd

2

,,)),(( min arg iisi

i

di IKIL

sd

Ready For The Real thing ….

jk ACL

jiji APAP FL

Mutual Illumination Very similar to

inverse radiosity

j

vC kC

With specular surface, no longer true

Before, radiance towards Pi from Aj was same as radiance towards Ck

j j

iAPCAPsAPAPdPCPC jivjijijiiv

ivKLFLEL ),(

j j

iAPCAPsAPAPdPCPC jivjijijiiv

ivKLFLEL ),(

Mutual Illumination

We can express the difference between the two as S

This is purely due to specularity e.g. Aj might look diffuse from Pi’s

viewpoint, but have a specular highlight from camera’s viewpoint

jikjkji APCACAP SLL

Mutual Illumination To recover all BRDF parameters for

all the surfaces we need: Radiance images covering the whole

scene Each surface patch needs to be assigned

a camera from which its radiance image is selected

At least one specular highlight on each surface needs to be visible in the set of images

Each sample point gives an equation

Mutual Illumination

Idea for iterative algorithm: assume zero S initially do

calculate L radiances from S estimates using global illumination

update all d, s, using L radiances re-estimate S using d, s, and L

loop until convergence

Mutual Illumination Highlight regions need special

treatment: detect in advance.

No guarantees for convergence.

No error bound on the recovered BRDF parameter values.

In practice work well.

Results

Inverse Global Illumination - Summary

Inverse radiosity Recovering specular reflectance

properties from direct illumination The reflected light was divided into diffuse

and specular components Specular component was modeled using

Ward’s model A new technique for determining

reflectance properties of entire scenes taking into account mutual illumination.

Rendering & Reconstructing Under Complex BRDF’s - Summary

Few basic concepts

Photometric Stereo

Image Based Rendering (IBR)

The End

From Normals to Shape Given pixel (x,y) and its normal n = ,

we wish to find the z coordinate.

The corresponding surface point is(x,y,Z(x,y))

The x component of n: (1,0,Zx)

The y component of n: (0,1,Zy)

z

y

x

n

n

n

Factorization – cont.

Since the normal is orthogonal to its x and y components we get:

(1,0,Zx) x (0,1,Zy) = (-Zx, -Zy, 1)

And after normalization:

11

1ˆ

22 y

x

yxz

y

x

Z

Z

ZZn

n

n

n

Now, we can integrate Z over x and y to find out Z(x,y).

z

xx n

nZ

z

yy n

nZ

Factorization – cont.

11

1ˆ

22 y

x

yxz

y

x

Z

Z

ZZn

n

n

n

Radiosity & Exitance for diffuse surfaces

dLB cos),,()( xx

Diffuse surfaces, by definition, have outgoing radiance that does not depend on direction

)(cos)()( xxx oo LdLB

dLE e cos),,()( xx

)(cos)()( xxx ee LdLE

Radiance to Radiosity

Recall:

Simplifying the global illumination equation gives:

)(),,,,(

)()(

)()( 0

xx

xx

xx

diioobrdf

eLE

LB

dLLL ioobrdfooeoo cos,,,,,,,,,, xxxx

dLLL id

e cos,,xx

xx

dLEB id cos,,xxxx

Switching the Domain

We still have annoying radiance terms inside the integral

Radiance is constant along lines

The radiance arriving is coming from a diffuse surface, y :

,,,, yx LL

yBLL ,,,, yx

Switching the Domain (cont)

We can convert the integral over the hemisphere of solid angles into one over all the surfaces in a scene:

otherwise

isiblemutually v arey and x if

0

1,

cos2

yxV

r

dyd

Sd dyr

VyB

EBy

yxxxx2

coscos,

)(

dLEB id cos,,xxxx

If x and y are mutually visible

Discrete Formulation

Assume world is broken into N disjoint patches, Pj, j=1..N, each with area Aj

Define:

i

i

Pi

i

Pi

i

dxEA

E

dxBA

B

x

x

x

x

)(1

)(1

Discrete Formulation (cont)

Change the integral over surfaces to a sum over patches:

N

jPd

j

dyVr

BEB1

2,

coscos)(

yyxyxxx

ij

i P

N

jPd

iPi

dxdyVr

BEA

dxBA x

yx

yxyxxx1

2,

coscos)(

11

N

j P Pijiii

i j

dydxVrA

BEB1

2,

coscos1

x y

yx

Sum over all patches in the

scene

Sum all points x in patch i

The Form Factor

dydxyxVrA

Fi jPx Py

iij ),(

coscos12

Fij is the proportion of the total power leaving patch Pi that is received by patch Pj

N

j P Pijiii

i j

dydxVrA

BEB1

2,

coscos1

x y

yx

N

jjijiii BFEB

1

Form Factor Properties

Depends only on geometry

Reciprocity: AiFij=AjFji

Additivity: Fi(jk)=Fij +Fik

•Reverse additivity is not true (F (jk)i Fji +Fki , it’s the area weighted average of the individual form factors)

1,1

N

j ijFi

Sum to unity (all the power leaving patch i must get somewhere):

Solving the Linear System

The matrix is very large – iterative methods are preferred

Start by expressing each radiosity in terms of the others:

ijiijijij

N

jij FMNiEBM

,1 ,1

NiM

EB

M

MB

ii

ij

N

ijj ii

iji

1 ,1

Relaxation Methods

Jacobi relaxation: Start with a guess for Bi, then (at iteration m):

NiM

EB

M

MB

ii

imj

N

ijj ii

ijmi

1 ,)1(

1

)(

NiM

EB

M

MB

M

MB

ii

imj

N

ij ii

ijmj

i

j ii

ijmi

1 ,)1(

1

)(1

1

)(

Gauss-Siedel relaxation: Use values already computed in this iteration:

rendering & reconstructing under complex brdf ’ s

Documents

area light

energy light

received light

spot light

solid angle light

single direction light

right surface

surface materialhow