selected topics in computer vision - institut de robòtica ... · • this is the lambert’s...

Selected Topics in Computer VisionSelected Topics in Computer Vision

Computational Photography

Lecture #2

Francesc Moreno‐Noguer

[email protected] @g

Today

Shape From XRadiometry and ReflectanceRadiometry and ReflectancePhotometric Stereo

Shape from Xp

Shape from X

Methods for obtaining the 3D shape fromMethods for obtaining the 3D shape from images. “X” can be many different cues:– Stereo (2 or more views)( )– Motion (moving camera or object)– ShadingShading– Changing Lighting (Photometric Stereo)– Texture variation– Texture variation– Focus/Defocus

Structured light– Structured light

Shape from XShape From Stereo and Motion

-Triangulate the same point on two or more imagesg p g-Images acquired simultaneously SF Stereo-Images from a moving camera or object SF Motion

Shape from XShape From Shading

Shading: “Variation of brightness due to changes in surfaceShading: Variation of brightness due to changes in surface orientation”

Shading reveals 3D surface geometryS f SShape-from-Shading: Just one image to recover shape Requires many constraints

Shape from XPhotometric Stereo

Single viewpoint, and multiple images under different lighting. g p , p g g g

Shape from XShape From Focus/Defocus

Camera Focus/Defocus is another cue to estimate depth.p

Shape from XShape From Texture

Texture: “Repetition of an element or the appearance of a specific template over a surface”of a specific template over a surface

Shape-from-Texture: guess the shape of a surface f h d f i f i T l (TEX EL )from the deformation of its Texels (TEXture ELements)

Shape from XStructure Light and Laser Range Finders

Active TriangulationActive Triangulation−Projector-plane / Image-plane triangulation

Shape from XTraditional Approaches

– Stereo– Motion– Shading– Photometric Stereo Lecture #2– Texture variation

Camera Focus/Defocus Lecture #3– Camera Focus/Defocus Lecture #3– Structured Light

Novel Techniques– Helmholtz Stereopsis Lecture #3

Projector Defocus Lecture #4– Projector Defocus Lecture #4– Coded and Multi Aperture Lecture #5

Radiometry and Reflectancey

Digital Image Formation

LightingCamera g g

Physical Models

Computer

Scene

We need to understand the relation between the lighting, surface reflectance and medium

and the image of the scene.


• Light Source – Image Plane:

SurfaceRadiance

Lens ImageIrradiance

SurfaceLight S

SurfaceIrradiance Radiance IrradianceSource

surfaceL EIrradiance

surfaceE

• Image Plane – Measured Pixels:

CameraElectronics

ImageIrradiance E

Measured Pixel Values, I



SurfaceRadiance


SurfaceLight S

SurfaceIrradiance Radiance IrradianceSource Irradiance

surfaceE surfaceL E


CameraElectronics

ImageIrradiance E


Digital Image Formationz source

viewingincidentdirection

= normal direction

y

θviewingdirection

normal

direction

),( ii φθ ),( rr φθ

x

φsurfaceelement

),( iisurfaceE φθ

),( rrsurfaceL φθ

Irradiance at Surface in direction ),( ii φθRadiance of Surface in direction ),( rr φθ

BRDF :),(),(),;,(

iisurface

rrsurface

rrii ELf

φθφθ

=φθφθ),( ii φ

Bidirectional Reflectance Distribution Function



SurfaceRadiance


SurfaceLight S


surfaceE surfaceL E


CameraElectronics

ImageIrradiance E



surface patch

image plane

θdA

αα

sdA

idAimage patch

fz fz

2⎞⎛ d

It can be shown that:

4cos4

α⎟⎟⎠

⎞⎜⎜⎝

⎛π=

fdLE surface

• Image irradiance is PROPORTIONAL to Surface Radiance!Image irradiance is PROPORTIONAL to Surface Radiance!• Small field of view Effects of 4th power of cosine are small• d/f is the inverse of the f-number.



SurfaceRadiance


SurfaceLight S


surfaceE surfaceL E


CameraElectronics

ImageIrradiance


E


• The camera response function relates image irradiance at the image plane to the measured pixel intensity values.p y

IEg →:

Pixel values in the final image

(Grossberg and Nayar, CVPR 2003)

Usually this is a Non-linear Mapping!


• Important preprocessing step for many vision and graphics algorithms such ash t t i t i i t d th i i d i i b d d i t

Radiometric Linearization of the Cameraphotometric stereo, invariants, de-weathering, inverse rendering, image based rendering, etc.

EIg →− :1

• Use a color (Magbeth) chart with precisely known reflectances.

Valu

es

255

?1−g

Pix

el V

0

g

Irradiance = const * Reflectance3.1%9.0%19.8%36.2%59.1%90%

• Method assumes constant lighting on all patches and works best when source is

00 1?

far away (example sunlight).

• Unique inverse exists because g is monotonic and smooth for all cameras.

Mechanisms of Reflectionsource

surfacereflection

incidentdi ti reflectiondirection

bodyreflection

surface

• Body Reflection: • Surface Reflection:

S l R fl tiDiffuse ReflectionMatte AppearanceNon-Homogeneous MediumCla paper etc

Specular ReflectionGlossy AppearanceHighlightsDominant for MetalsClay, paper, etc Dominant for Metals

Image Intensity = Body Reflection + Surface Reflection

Example Surfaces

Diffuse Reflection Specular ReflectionDiffuse + SpecularReflection

Reflectance Models

Diffuse Reflection: Lambertian ModelHow to model the mechanisms of reflectance?

Diffuse Reflection: Lambertian Model• Surface appears equally bright from ALL directions! (independent of )vr

Source intensity K

viewing

normal

incidentnr

viewingdirection

f

incidentdirection

iθvrsr

surfaceelement

lb d

πρφθφθ d

rriif =),;,(• Lambertian BRDF is simply a constant :albedo

Reflectance Models

Diffuse Reflection: Lambertian Model

• Surface Radiance : snKKL di

d rr⋅

πρ

=θπρ

= cossource intensity

• Commonly used in Vision and Graphics!

• This is the Lambert’s cosine law of reflection from matte surfaces.

Reflectance ModelsSpecular Reflection (Mirror BRDF)

• Valid for very smooth surfaces.Valid for very smooth surfaces.

• All incident light energy reflected in a SINGLE direction

source intensity K specular/mirrory

incident

rrp

direction ),( rr φθ

sr

viewingdirectionsurface

normal

c de tdirection

),( ii φθ)( φθ

nr

vrs

directionsurfaceelement

),( vv φθv

• Mirror BRDF is simply a double-delta function :Mirror BRDF is simply a double delta function :

)()(),;,( vivisvviif φπφδθθδρφθφθ −+−=specular albedo

Reflectance ModelsSpecular Reflection (Mirror BRDF)

• Surface Radiance : )()( vivisKL φ−π+φδθ−θδρ=

( ) ( )⎧ π−φ=φθ=θρ andifK ii( ) ( )

⎪⎩

⎪⎨

⎧ πφφθθρ=

otherwise

andifKL

vivis

0 z specular/mirrordirection⎩ otherwise0

θ),( ii φθ( ) ( )π+φθ=φθ iirr ,,

direction

x

yφ

• Viewer receives light only when rv rr=

x

Reflectance ModelsDiffuse + Specular Reflection

• Many real surfaces combine diffuse and glossy components ofMany real surfaces combine diffuse and glossy components of reflection

• In order to approximate these surfaces Models combining Diffuse and Specular reflection (Phong, Dichromatic, …)

Observed Image Color = α x Body Color + β x Specular Reflection Color

Diffuse and Specular reflection (Phong, Dichromatic, …)

diffuse specular diffuse+specular

Reflectance ModelsPhong Model

( ) ( )α∑Ambient Diffuse Specular

( ) ( ) sα

slights

ddaa ikikikI vrns ⋅+⋅+= ∑• ia id is: Intensities of the ambient

Models an angular falloff of the highlights

s rv

nia,id,is: Intensities of the ambient specular and diffuse components of the light sources.

v•ka,kd,ks: material components

•α : shininess how evenly the light is reflected

sn)ns(r −⋅= 2is reflected

Reflectance ModelsHomework: Implement Phong Model for a Sphere

(for a sphere is easy to compute the normals)

)Ryxy,(x,R1y)N(x, 222 −−=

Reflectance Models

Results

Phong Model Parametersia:0.19 id:0.44 is:0.59



ka:0.03 kd:0.47 ks:0.50alpha:8.94

ka:0.13 kd:0.47 ks:0.53alpha:17.09

ka:0.14 kd:0.49 ks:0.43alpha:8.63

Photometric Stereo

Surface Orientation

N Surface : ( )y)x,fy,x,y)s(x, (=surface normalz

Tangent vectors : xf1,0,

xy)s(x,

⎟⎠⎞

⎜⎝⎛

∂∂

=∂

∂f(x,y)

yf0,1,

yy)s(x,

⎟⎟⎠

⎞⎜⎜⎝

⎛∂∂

=∂

∂y

x

ys

xs

∂∂

×∂∂

=NNormal :

⎞⎛ ∂∂ ss

'yImage Plane

⎟⎟⎠

⎞⎜⎜⎝

⎛∂∂

−∂∂

−= 1ys

xs ,,

( ) 1qp ,,=

'x

Gradient Spacez

Unit Normal vector

1=zq

1N

S ( )1

1,,22 ++

==qpqp

NNn

yp s niθ

( )1,,== SS qpSs

Source vector

x 122 ++==

SS qpSs

1=z plane is called the Gradient Space (pq plane)• Its components p and q, are the surface slopes in the x- and y- directions• Every point on it corresponds to a particular surface orientationEvery point on it corresponds to a particular surface orientation

Reflectance Map• Relates image irradiance I(x,y) to surface orientation (p,q)

for GIVEN source direction and surface reflectance• Tool used in developing method for recovering shape from

images. Let’s consider an example:• Lambertian case: ( )yxI• Lambertian case: ( )yxI ,

: source brightness

: surface albedo (reflectance)

kρ

s vniθ

: surface albedo (reflectance)

: constant (optical system)

ρc

Image irradiance:Image irradiance:

sn ⋅== kckcI i πρθ

πρ cos

Let 1=kcπρ

then sn ⋅== iI θcos

Reflectance Map

( ) ( )qqpp 1++

Lambertian Case( ) ( )qpR

qpqpqqppI

SS

ssi ,

111cos

2222=

++++

++=⋅== snθ

R fl t MReflectance Map(Lambertian)

• Contours of constant brightness are conic sections in• Contours of constant brightness are conic sections in the pq-plane:

( ) ( )( )( ) cqpR =, ( ) ( )( )111 222222 ++++=++⇒ SSss qpqpcqqpp

Reflectance MapLambertian Case

( ) ( )( )111 222222 ++++++

Iso-brightness contour ( ) cqpR =,

( ) ( )( )111 22222 ++++=++ SSss qpqpcqqpp

p

cone of constant iθq

Reflectance MapLambertian Case

qiso-brightnesscontour

0.1

9.08.0

( ) 7.0, =qpR( )

contour

( )SS qp ,

po90=iθ

( )01=++ SS qqpp

3.0

0.0Note: is maximum when( )qpR , ( ) ( )SS qpqp ,, =

Reflectance MapDiffuse + Specular

• Reflectance map with two peaks: orientations that

q Diffuse peak

p pmaximize each of the two different types of reflection.

( )SS qp ,

p

Specular peak

p

Specular peak

( ) 5.0, =qpR

Shape from a Single Image?• Given a single image of an object with known surface

reflectance taken under a known light source, can we grecover the shape of the object?

• Given R(p,q) ( (pS,qS) and surface reflectance) can we determine (p q) uniquely for each image point?determine (p,q) uniquely for each image point?

q

All the points on the line, are solutions for I(x,y)

I(x,y)

NO

p

( y)

NO

Solution

• Take more imagesTake more images– Photometric stereo

• Add more constraints– Shape-from-shading (Too restrictive p g (

to be useful )

Photometric Stereo

q

( )11 ,SS

qp

p( )22 ,

SSqp

( )33( )33 ,SS

qp

Photometric Stereo

⎠⎞

⎜⎝⎛ =1kc

Lambertian case:

sn ⋅== ρθρikcI cos

Image irradiance:⎠⎝ π

11 sn ⋅= ρIn

2s3s

ρπ i

11 ρ1s v

22 sn ⋅= ρI33 sn ⋅= ρI

• We can write this in matrix form:

s ⎤⎡⎤⎡ TIns

s

⎥⎥⎥⎤

⎢⎢⎢⎡

=⎥⎥⎤

⎢⎢⎡

TII

22

1 1

ρs ⎥⎦⎢⎣⎥

⎥⎦⎢

⎢⎣

TI 32

Solving the Equations

s⎥⎤

⎢⎡

⎥⎤

⎢⎡ TI 1 1

nss ρ

⎥⎥⎥

⎦⎢⎢⎢

⎣

=⎥⎥⎥

⎦⎢⎢⎢

⎣T

T

II

3

2

2

2

⎦⎣⎦⎣ 32

I S n~133313 13×33×13×

ISn 1~ −= inverse

n~=ρnn ~~

Since n is a unit vector,the norm of will be ρ

ρn

nnn ~ ==

More than Three Light Sources• Get better results by using more lights

s ⎤⎡⎤⎡ TIn

s

sρ

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

TI

IMM11

s ⎥⎦⎢⎣⎥⎦⎢⎣ NNI

• Least squares solution:~nSI ~=

nSSIS ~TT =1331 ××× = n~SI NN

( ) ISSSn TT 1~ −=

M P• Solve for as beforen,ρ Moore-Penrose pseudo inverse

Color Images• The case of RGB images

– get three sets of equations, one per color channel:g q p

SnI ρ=SnI RR ρ=SnI GG ρ=SnI BB ρ=

– Simple solution: first solve for using one channel– Then substitute known into above equations to get

( )ρρρn

n

– Or combine three channels and solve for

( )BGR ρρρ ,,

nSnIIII ρ=++= 222

BGR

Trick for Handling Shadows• Weigh each equation by the pixel brightness:

( ) ( )• Gives weighted least-squares matrix equation:

( ) ( )iiii III sn ⋅= ρ

g q q

s⎥⎤

⎢⎡

⎥⎤

⎢⎡ TII

MM11

21

ns

ρ⎥⎥⎥

⎦⎢⎢⎢

⎣

=⎥⎥⎥

⎦⎢⎢⎢

⎣TNNN II

MM2

• Solve for as beforen,ρ

Depth From Normals• Photometric Stereo “just” recovers the surface normals• We need to compute the depth!!!

Easy Method: Shape By Integration (1D)• Partial Derivatives: Give the change in surface height with a small step.

f∂0p• We can get the surface by summing the changes.

ixxi x

fp=∂

∂=

0p

dxxfxf ix

i ∫ ∂∂

=0

)(Integrated

orientations

ii pxf += − )( 1

Depth From NormalsShape By Integration (2D)

y( )0000 qp ,

( )qp

y

( )1010 qp ,

x( )ijij qp ,( )ijij qp ,

∫∫∂∂ jiyx ff

Assigned a priori

∑∑∫∫ ++=∂∂

+∂∂

=j

iji

yx

ij qpfdyyfdx

xff ji

0000000

Depth From NormalsShape By Integration (2D)

Limitations:• Integration is good only if object shape is continuous• It is sensitive to noise:

− Solution: Average the results over many different paths.

X X X XX X X X

High cost and still inaccurate!!!!!!

Depth From NormalsFrankot-Chellappa Algorithm

• Best algorithm I found to compute depth from normalsBest algorithm I found to compute depth from normals.

• Very robust to noise.

• Fast and very simple implementation.

• Let be the surface to be reconstructed( )),(ˆ,, yxfyxFormulation of the problem:

• The basic idea is to project the estimated surface slopes p(x,y) and

q(x,y) onto a set of integrable surface slopes and ),(ˆ yxp ),(ˆ yxqsuch that:

1. Integratibility Constraint: xyyxf

yxyxfq

xp

y ∂∂∂

=∂∂

∂⇔

∂∂

=∂∂ ),(),(ˆˆ

22

2. The distance error is minimized:

xyyxxy ∂∂∂∂∂∂

∫∫ −+− dxdyqqpp 22 ˆˆ


Solution:• Expand by Fourier:

Solution:),(ˆ yxf

∑ ωφω= )()(ˆ)(ˆ yxCyxf This expansion ensures∑Ω∈ω

ωφ⋅ω= ),,()(),( yxCyxf

Fourier Coefficients

This expansion ensures integrability (condition 1 before), because Fourier basis functions are integrable

• We need to find the coefficients that minimize the distance error of

condition 2. The paper shows that these optimal coefficients are:

)(ˆ ωC

22yx

yyxx CjCjwC

ω+ωωω−ωω−

=)()(

)(ˆ

)()(C FFT These are just the Fourierh )()( pCx FFT=ω

)()( qCy FFT=ω

These are just the Fourier Transforms of the original estimated slopes p and q !!!!!

where:



fftshiftfft2 fftshift

ifftshift

fft2

selected topics in computer vision - institut de robòtica ... · • this is the lambert’s...

Documents