a convex optimization approach for depth estimation under illumination variation wided miled,...

1

A Convex Optimization Approach for DepthA Convex Optimization Approach for DepthEstimation Under Illumination VariationEstimation Under Illumination Variation

Wided Miled, Student Member, IEEE, Jean-Christophe Wided Miled, Student Member, IEEE, Jean-Christophe Pesquet, Senior Member, IEEE, and Michel ParentPesquet, Senior Member, IEEE, and Michel Parent

2

AbstractAbstract

• Illumination changes cause serious problems in many computer vision applicatIllumination changes cause serious problems in many computer vision applications. Aions. A spatially varying multiplicative model is developed to account for spatially varying multiplicative model is developed to account for brightness changes induced between brightness changes induced between left and right viewsleft and right views..

• ThThee recovery of the depth information of a scene from recovery of the depth information of a scene from stereo images is an activstereo images is an active e area of researcharea of research in computer in computer vision. The need for an accurate and dense deptvision. The need for an accurate and dense depth map arises inh map arises in many applications such as many applications such as autonomous navigation,autonomous navigation, 3-D reconst3-D reconstructionruction and and 3-D television3-D television..

2

3

I. INTRODUCTIONI. INTRODUCTION

• Feature-based methods:Feature-based methods: Extract salient features from both images, such as Extract salient features from both images, such as edgesedges, , segmentssegments, or , or curvescurves. .

An interpolation step is required if a An interpolation step is required if a densedense map is desired, but map is desired, but accurateaccurate..

• Region-based methods:Region-based methods: It have the advantage of directly generating dense disparity estimates by It have the advantage of directly generating dense disparity estimates by

correlation over local windows, but correlation over local windows, but not accuratenot accurate..

Many global stereo algorithms have, therefore, been developed based on Many global stereo algorithms have, therefore, been developed based on dynamic programmingdynamic programming, , graph cutsgraph cuts, or , or belief propagationbelief propagation. . Variational Variational approachesapproaches have also been very effective for solving the matching problem have also been very effective for solving the matching problem globallyglobally

4

II.MODEL FOR ILLUMINATION VARIATIONSII.MODEL FOR ILLUMINATION VARIATIONS

The intensity of an image pixel:The intensity of an image pixel:

IIii(s) = (s) = ρρ(s) R(s) Rii(n(s))(n(s)), for i , for i ∈∈ l,r﹛ ﹜l,r﹛ ﹜。。

Assuming that the stereo images have been rectified, so that the geometry of Assuming that the stereo images have been rectified, so that the geometry of

the cameras can be considered as horizontal epipolar, and using the the cameras can be considered as horizontal epipolar, and using the

Image Irradiance Equation:Image Irradiance Equation:

IIrr( x-u(s), y ) = v(s) I( x-u(s), y ) = v(s) Ill(s)(s)

4

5


The disparity The disparity uu and illumination and illumination vv can be computed by can be computed by minimizingminimizing the the

following cost function based on the following cost function based on the sum of squared differences (SSD)sum of squared differences (SSD) metric: metric:

Ĵ( u, v ) = ∑s D ∈ [ v(s)Il(s) – Ir( x-u(s), y )]2 , D⊂N2

This expression is This expression is nonconvexnonconvex with respect to the displacement field with respect to the displacement field uu. Thus, . Thus,

to avoid a to avoid a nonconvex minimizationnonconvex minimization, we , we assume thatassume that IIrr is a is a differentiabledifferentiable

function and we consider a function and we consider a TaylorTaylor expansion of the nonlinear term expansion of the nonlinear term

IIrr( x-ū, y )( x-ū, y ) around an initial estimate around an initial estimate ūū as follows: as follows:

Ir( x-u, y ) ≈ Ir( x-ū, y ) - ( u-ū ) I∇ rx( x-ū, y )

6


To simplify the notations:To simplify the notations:

Ĵ( u, v ) ≈ ∑s D ∈ [ L1(s)u(s) + L2(s)v(s) – r(s) ]2

where L1(s) = I∇ r

x( x-ū, y ), L2(s) = Il(s), r(s) = Ir( x-ū(s), y ) + ū(s)L1(s)

Our goal is to simultaneously recover u and v. Thus, setting Our goal is to simultaneously recover u and v. Thus, setting w = ( u, v)w = ( u, v)TT and and

L = [ LL = [ L1, 1, LL22]] , we end up with the following quadratic criterion to be minimized: , we end up with the following quadratic criterion to be minimized:

JJDD( w ) = ∑s D ∈ [ L(s)w(s) – r(s) ]2

7

III. SET THEORETIC ESTIMATIONIII. SET THEORETIC ESTIMATION

• FindFind

w S=∩∈ i=1m

Si

such that such that

J(w) = inf J(S)

where

J: H→]-∞,+∞] is a convex function.J: H→]-∞,+∞] is a convex function.

(S(Sii))1≤i ≤m1≤i ≤m are closed convex sets of H. are closed convex sets of H.

Constraint sets can be modelled as level sets :Constraint sets can be modelled as level sets :

∀i { 1,…,m }, S∈ i = { w H | f∈ i(w) ≤ δi }

where where

∀ ∀i { 1,…,m }, f∈i { 1,…,m }, f∈ ii:H →R is continuous convex function:H →R is continuous convex function

(δ(δii) ) 1≤i ≤m 1≤i ≤m are real-valued parameters.are real-valued parameters.

8


• A. Global Objective Function ( 1 / 2 )A. Global Objective Function ( 1 / 2 ) The initial disparity estimate ū:The initial disparity estimate ū:

ū(x,y) = arg minu U ∈ ∑(i,j) β∈ [ βx,y(u) Il(x+i,y+j) – Ir(x+i-u,y+j) ]2

wherewhere

U N is the search disparity set⊂U N is the search disparity set⊂ 。。

βcorresponds to the matching block centered at the pixel (x,y)βcorresponds to the matching block centered at the pixel (x,y) 。。 ββx,yx,y(u) is the following least squares estimate of the illumination factor for (u) is the following least squares estimate of the illumination factor for

block β:block β:

ββx,yx,y(u)=(u)=∑(i,j) β∈ Il(x+i,y+j)Ir(x+i-u,y+j) /∑(i,j) β∈ Il(x+I,y+i)2

The initial illumination field ϋ :The initial illumination field ϋ :

ϋ(x,y) = βϋ(x,y) = βx,y x,y ( ( ū(x,y) ))

9


• A. Global Objective Function ( 2 / 2 )A. Global Objective Function ( 2 / 2 )

JD\O(w) = ∑s D\O ∈ [ L(s)w(s) – r(s) ]2

J(w) = ∑s D\O ∈ [ L(s)w(s) – r(s) ]2 + α∑s D∈ | w(s) - ŵ(s) .2

2

where

ŵ = (ū, ϋ) is an initial estimate as described aboveϋ) is an initial estimate as described above

| .2 denotes the Euclidean norm in R denotes the Euclidean norm in R22

α is a positive constant is a positive constant

10


• B. Convex ConstraintsB. Convex Constraints

1) Constraints on the Disparity Image: 1) Constraints on the Disparity Image: Total Variation Based Regularization:Total Variation Based Regularization:

For a differentiable analog image u defined on a spatial domain For a differentiable analog image u defined on a spatial domain Ω

TV(u) = ∫Ω| u(s) | ds∇

where

∇u denotes the gradient of u

Sa1 = { (u,v) H | TV(u) ≤ T∈ u }

where

a : stands for analog constraint sets .a : stands for analog constraint sets .

11



1) Constraints on the Disparity Image:1) Constraints on the Disparity Image: Disparity Range Constraint:Disparity Range Constraint:

SSaa2 2 == { (u,v) H | u∈ min≤ u ≤ umax }

12



1) Constraints on the Disparity Image:1) Constraints on the Disparity Image: Nagel–Enkelmann Based Regularization:Nagel–Enkelmann Based Regularization:

wherewhere

I denotes the 2 2 identity matrixI denotes the 2 2 identity matrix

r is chosen according to gradient norm value ranger is chosen according to gradient norm value range

| I|<<r:uniform areas, | I|>>r:edge∇ ∇| I|<<r:uniform areas, | I|>>r:edge∇ ∇

13



2) Constraints on the Illumination Field:2) Constraints on the Illumination Field: Tikhonov Based Regularization:Tikhonov Based Regularization:

Illumination Range Constraint:Illumination Range Constraint:

SSaa5 5 == { (u,v) H | v∈ min≤ v ≤ vmax }

where

vmin = 0.8

vmax = 1.2

14

IV. EXPERIMENTAL RESULTSIV. EXPERIMENTAL RESULTS

NNββis the total number of is the total number of

pixels inβpixels inβNNββis the total number of is the total number of

pixels inβpixels inβ

15


(x0, y0) is (128, 128)(x0, y0) is (128, 128)α is the standard deviation of the α is the standard deviation of the illumination change.illumination change.

(x0, y0) is (128, 128)(x0, y0) is (128, 128)α is the standard deviation of the α is the standard deviation of the illumination change.illumination change.

16


δδssis fixed to 1is fixed to 1δδssis fixed to 1is fixed to 1

17


18


19


20


21


22


24

Thank you for your Thank you for your listening ! listening !

The more you The more you learn,learn, the more you the more you know.know. The more you The more you know,know, the more the more

you you forget.forget. The more you The more you forget, forget, the less you knowthe less you know.. 2009.09.012009.09.01

a convex optimization approach for depth estimation under illumination variation wided miled,...

Documents

s i r x

s r i ns

s d vsi

dense depth map

dense map

stereo images

regionbased methods

illumination variations