recovery of three-dimensional shapes by using defocused structured light

Recovery of three-dimensional shapes by using defocusedstructured light

Carlos Hinojosa a, *, Alfonso Serrano-Heredia b, Juan G. Ibarra c

aCentro de Optica, Instituto TecnoloÂgico y de Estudios Superiores de Monterrey, Sucursal de correos J, Monterrey 64849 N.L., MexicobInstituto Nacional de AstrofõÂsica, Optica y ElectroÂnica, Apdo. Postal 216, Puebla, 72000, Pue., Mexico

cBiological & Agricultural Engineering, University of Arkansas, 203 Engineering Hall, Fayetteville, AR 72701, U.S.A.

Received 2 July 1998; accepted 6 August 1998

Abstract

We present a new concept for three-dimensional shape recovery using defocused structured light (DSL) images. The DSLtechnique externally extracts the depth information from the scene by using projections of cylindrical wavefronts on the object.

These projections show di�erent degrees of defocus as a function of the depth. E�cient algorithms for shape recovery aredeveloped taking advantage of the spatial regularity of the projected light and the pro®le of each fringe. In this way a depthmap of the scene is obtained using only one image. Experimental results are shown and we discuss the possibility of a real timeimplementation. # 1998 Elsevier Science Ltd. All rights reserved.

Keywords: 3D-reconstruction; Shape-from-defocus; Structured light; Range imaging; Di�raction grating

1. Introduction

The recovery of three-dimensional shapes using its

projections in two dimensions has generated much

research due to its vast ®eld of applications in robotic

vision and microscopy. Classical methods used for

depth information recovery include shape from

shading, binocular vision, depth recovery by focus-

defocus systems and range imaging from structured

light [1±9]. One of the most attractive characteristics

of the focus±defocus systems and the structured illu-

mination methods is the fact that the image matching

problem can be avoided. Introduction of information

that can be used beforehand by the algorithm design

(line distribution and line orientation control) is an

advantageous feature of structured light methods. On

the other hand, depth from defocus (DFD) and depth

from focus (DFF) methods [4±9] take advantage of

the depth information introduced by the limited depthof ®eld of image acquisition systems.

In this work, we propose a depth recovery methodthat combines the advantages of structured light im-aging with defocused systems. The main feature ofthe method is the treatment of DSL images, whichpresent special advantages. A combination of cylindri-cal and spherical lenses and a di�raction grating areused to project several light fringes on the object.The width and the intensity pro®le of the fringes con-tain the information of the defocus that is used bythe algorithms for the recovery of the depth maps.The fact that the degree of defocus is externallyintroduced makes DSL method di�erent from classicunfocussed light methods, which uses the principle ofa restricted depth of ®eld of the image acquisitionsystem.

One of the main objectives of this work is to obtaina depth map using just one image; this fact has specialsigni®cance since the recovery of depth of a completescene using only one image has always been a di�culttask when structured light images and traditional defo-cus methods are applied. We will explain these di�cul-ties in the following:

Optics & Laser Technology 30 (1998) 281±290

0030-3992/98/$19.00 # 1998 Elsevier Science Ltd. All rights reserved.

PII: S0030-3992(98 )00053 -X

* Author for correspondence: Centro de Optica, Instituto

TecnoloÂ gico y de Estudios Superiores de Monterrey, E. Garza Sada

2501 Sur, Monterrey, 64849, N.L., Mexico. E-mail: chinojos@

campus.mty.itesm.mx

1. A particular structured light image technique con-sists on the projection of a light fringe over anobject of the scene and a registry of the image istaken from an adequate angle. In this special case,the relative displacement of the fringe is a functionof the depth and other known parameters. Thistechnique is commonly used in robotic vision andautomated inspection systems [1±3]. When thestructured light system consists on a simple line, aprocessing of several images is needed to cover thecomplete scene. Nevertheless, when memory or timelimitations exist, it is desirable that the depth mapis acquired by means of the analysis of only oneimage; this can be achieved replicating the lightfringes with a di�raction grating. The problem weface in this case is that the algorithms used for sev-eral fringes in one image have to deal with theprobable intricacy of the fringes caused by disconti-nuities, shades or noise.

2. In depth recovery from defocus, several authorshave reported the advantage of their methods forthe depth information extraction using few images(two or three) of the same scene that were takenwith di�erent camera parameters [4±8]. Others havereported achievements in this ®eld using just oneimage for some speci®c applications [9].Nevertheless, almost all this methods stronglydepend on the natural brightness variations of thescene (high spatial frequency information), whichlimits the application range of the method.

Our method overcomes the mentioned problems in amore complete way. The possibility of a confusion ofdi�erent light fringes disappears since the camera thatgets the images can be situated at an angle such thatthe relative displacements of the fringes are avoided.This fact makes the map generation feasible from anonly image, using not very complicated algorithms andwith short running times. Also, our method does not

need an inherent bright variation of the scene becausethe defocus is externally caused by multiple fringe pro-jection with variable defocus.

2. Defocused structured light generation system

The principal objective of the defocused structuredlight generation system is to produce the necessarydefocus gradient for the depth estimation. In Fig. 1 weshow the basic optical arrangement for the DSL sys-tem. We use a laser beam as a light source, which isexpanded with the spatial ®lter SF. A 1-D binarygrating G di�racts the beam and each di�raction orderis transformed to a line (fringe) using a combinationof a spherical lens EL and a cylindrical lens CL ®xedwith its meridian axis in vertical position. The objectO is illuminated by the di�erent line fronts (DSL wavefronts). Finally, a CCD camera adquires the image.

2.1. Di�raction process

The binary grating can be described as an array ofrectangular apertures of width a with spatial period d(See Fig. 2). The transmittance function of the gratingcan be expressed as

U0 x0; y0ÿ � � rect

y0a

� �Xn�1

n�ÿ1 d y0 ÿ ndÿ �

; �1�

where rect and d are the rectangular and impulse func-tion respectively which are de®ned in Ref. 10. Thesymbol represents the convolution operation and x0,y0 are the grating plane coordinates.

First, we will assume that the only refracting el-ement of the system is the spherical lens, with focaldistance f. The di�raction pattern at the focal distancef is calculated through the 2-D Fourier transform of(1), evaluated at spatial frequencies fx=x/lf andfy= y/lf, where x and y are the di�raction plane coor-

Fig. 1. Defocused structured light image system.

C. Hinojosa et al. / Optics & Laser Technology 30 (1998) 281±290282

dinates and l is the wavelength. The correspondingintensity pattern is given by

I x; yÿ � � 1

l2f2

!Xn�1n�ÿ1 sinc2 an=d� �

d y=lfÿ n=dÿ �

d x=lf� �: �2�In this case, the di�raction pattern consists in a seriesof impulse functions in y-axis with spatial period lf/d,modulated by the sinc function. Fig. 3 schematicallyshows the intensity pattern as a function of n, the dif-fraction order. We can consider the nth di�ractionorder as a light beam with a deviation angle from theoptical system axis proportional to n. Each one ofthese beams will be opened in a fan shape and a factorof defocus will be introduced by the combination ofthe spherical and cylindrical lenses, as we will see next.

If we consider the two refracting elements: thespherical lens (focal distance f) and the cylindrical lens(focal distance fC), the Fresnel ®eld UZ(x, y) at dis-tance z can be written by the product of two 1-DFourier transform (see Appendix A):

Uz x; yÿ � � fact� �z =1D

�U0 y0ÿ �

eÿik2

ÿ1zÿ1

f

�y20

�v�y=lz

=1D�eÿik

2

ÿ1zÿ 1

f L

�x20

�u�x=lz

�3�

where (fact)z is a phase factor that appears in Fresnel

formulation, fL is de®ned by

1

fL� 1

f� 1

fC; �4�

k is 2p/l, and U( y0) is the input ®eld (1), whichdepends only on y0.

The last 1-D Fourier transform in (3) gives [10]

jlzLeÿjplzLu2

; �5�where

zL � 1

zÿ 1

fL

� �ÿ1Expression (5) can be interpreted as a cylindrical wavethat starts in z= fL. For z> fL, the argument of theexponential has a negative value, therefore the wave isdivergent. Expression (5) explain the way the DSLfronts opens in a fan shape.

The intensity pattern at the focal distance f can becalculated using (3), evaluated in z= f and multiplyingby the complex conjugate. The expression is

Iz�f�x; y� � �fact�z�f =1D U0 y0ÿ ��

v�y=lz�� 2: �6�

Substituting (1) in (6) we obtain

Iz�f�x; y� � �fact�Xn�1

n�ÿ1 sinc2 an=d� � d y=lfÿ n=dÿ �

:

�7�The intensity pattern consists in a set of bright lineswhich are parallel to x-axis, with spatial period lf/dand modulated by a sinc function. Notice that thedelta function in x, which appeared in (2), does notexist anymore in (7).

2.2. DSL wave fronts

The principal characteristics of the DSL wavefrontscan be explained using the basic optical unit consistingin a combination of a spherical and a cylindrical lens.In order to explain the basic action of the spherical

Fig. 2. 1-D binary grating and its di�raction pattern.

Fig. 3. Schematic representation of di�raction orders.

C. Hinojosa et al. / Optics & Laser Technology 30 (1998) 281±290 283

and cylindrical unit, let us consider two di�erent casesof incident light orientation as shown in Fig. 4. In casea, a light belt passes through the axis meridian, and incase b, the light belt goes through the power meridian.It can be noted in the ®rst case that the cylindrical lensdoes not contribute in the light refraction because thisone is produced only by the action of the sphericallens, in such a way that the focal point F2 is de®ned.Now, for the second case, we ®nd that both lenscontribute in the refraction, creating a nearer focalpoint F1.

If the incident light has a circular pattern, it willtravel through the ®rst focal point F1 and then it willget the second focal point F2 in the form of an hori-zontal line shape, having the same direction as thepower meridian of the cylindrical lens. It is importantto note that when the light goes through the ®rst focal

point, it has not converged in a point, but in a verticalline. The zone de®ned between the focal points F1 andF2 is known as the Sturm interval. A transversal sec-tion where the light presents a circular pattern can befound in this zone (least confusion circle). In Fig. 5 weshow a combination of a spherical and cylindrical lenswith its two focal points F1 and F2.

The basic analysis unit in DSL images processing isconstituted by the projection of the DSL wavefrontson the object. The projections appears as periodic lightfringes on the object. Each light fringe is really anellipse with great eccentricity that exists in the Sturminterval. The shaded area in Fig. 5 shows the zone weused for the generation of the light fringes.

We can see that the width of the fringes is producedby the spherical lens and it is a function of the depthdistance of the incident plane. The horizontal aperture

Fig. 4. Axis meridian incidence and axis power incidence.

Fig. 5. Light fringe zone.


of the light fringe is produced by the combination ofthe spherical and the cylindrical lenses.

3. Depth recovery

In the preceding section, we have mentioned thatthe spherical lens determines the fringe width. In thissection we will study the relationship of the width s ofthe projected fringe, the focal distance f of the spheri-cal lens and Dz, the distance from the plane of maxi-mum focus and the defocused projection plane.

When the DSL front is not in focus, the di�racted®eld can be obtained by evaluating (3) in z< f. Theimportant expression to evaluate is

=1D�U0 y0ÿ �

eÿik2

ÿ1zÿ1

f

�y20

�v�y=lz

�8�

Using (1) and the convolution theorem, the last ex-pression can be written as

= U0 y0ÿ �� =�eÿik

2

ÿ1zÿ1

f

�y2

0

�v�y=lz

�Xn�1

n�ÿ1 sinc an=d� � d y=lfÿ n=dÿ �

lizf

fÿ z

� �eÿ pif

l2z�fÿz�

� �y2 �9�

The exponential functions in (9) are the terms which

introduce the defocus in the fringes. We see that deltafunctions are convolved by the exponential functions.The result of each of these operations is the same ex-ponential function, displaced to the position of thedelta. In particular, for di�raction order 0, the result is

eÿ pif

l2z�fÿz�

� �y2 �10�

which has the form: eÿy2

Cs, where s is the width esti-mation of the defocused fringe. The interest region isclose to the plane z= f, therefore we conclude

sA fÿ z� � �11�or sADz. Parameter s is proportional to the distancefrom the plane of maximum focus and the defocusedprojection plane. The proportionality constant isadjusted in the calibration process.

In the depth estimation using DSL images, thedepth recovery Dz can be obtained using the relation-ship (11), considering that the parameter s (width ofthe fringe) can be measured or estimated from theDSL image.

4. DSL image characteristics

The defocus structured light images have character-istics that make them convenient for their analysis.Figs 6 and 7 show the fringe projections on di�erent

Fig. 6. DSL image on focal plane.


planes, from which we can observe three importantcharacteristics in a DSL image:

1. The orientation of the fringes is determined andconstant. This fact implies that the gradient direc-tion is known and the derivative operators to beused in the algorithms are more simple and e�-cient.

2. The fringes have a spatial distribution that can be

previously known. This line distribution has a per-

iodic behavior.

3. The three central di�raction orders have the great-

est intensity. Regardless of the intensity variation of

di�erent lines, the signi®cant part is the pro®le uni-

formity for each fringe in particular.

Fig. 7. DSL image on out of focus plane.

Fig. 8. DSL image pro®le.


Fig. 8 shows the fringe pro®le of the DSL image ofFig. 6. An approximated Gaussian shape of the intensityvariations can be noticed. This is due to some smooth-ing factors which are introduced by di�raction e�ects inthe image formation, and by the sampling e�ects intro-duced in the digitalization process [5]. This gaussianmodel will be used in the reconstruction algorithms.

The range of depth in a DSL system has a naturallimit imposed by the overlapping of di�erent fringes.This depth range depends on the DSL system designand a variety of optical factors can be adjusted inorder to optimize the range. Among these factors arethe lens relative positions, the focal lengths, the periodof the grating and its aperture ratio and also, thewavelength of the laser beam.

5. Algorithms for DSL image processing

5.1. Defocus estimation

The defocused structured light images have the ad-vantage of representing the line spread function of theoptical system in a direct way. If we consider an idealimage generator system, we would have an extremelythin line on the focal plane. In a di�erent plane, therewould be a light fringe with a perfect square pro®le.Nevertheless, due to spatial frequency limitation of thesystem (because of its ®nite exit pupil), we have adi�erent response. In this section, it would be assumedthat the projected lines on the object has a Gaussianpro®le, as mentioned in previous section. The esti-mation of the parameter s is based on a least squareerror estimation.

Let us consider an isolated DSL fringe from the restof the fringes. The digitized image values Ci of thefringe represent samples of the ideal Gaussian distri-bution of intensity G( yÿ y0, s),

Ci1G yi ÿ y0; sÿ �

: �12�Substituting the Gaussian function and eliminating thei index, we ®nd

C � 1

s��2pp exp ÿ �yÿ y0�2

2s2

!: �13�

After applying the natural logarithm to (13) we obtain

lnC � ln1

s��2pp

� �ÿ �yÿ y0�2

2s2: �14�

The y0 position of the fringe is known and, for simpli-city, it will be taken as zero. Therefore, the formerequation has the shape of a lineal regression in y 2:

Ay2 � B � D; �15�

with values

A � ÿ 1

2s2; B � ln

1

s��2pp

� �; D � lnC: �16�

The solution [11] is given by

A �Pi

y2i ÿ �y2

� �DiP

i

y2i ÿ �y2

ÿ �2 and B � �Dÿ �y2A: �17�

We can obtain the s estimation from:

s ��

1

ÿ2A� �

s: �18�

5.2. General algorithm

The general algorithm exploits the periodicity of theDSL fringes. The main procedure of the general algor-ithm is the determination of a delimitated region in aspatial period and then the application of the next twoimportant steps:

1. The localization of the central position of a DSLfringe.

2. The estimation of the s value for that particularfringe using the square error estimation.

We point out that the coverage of the complete imagecan be done very e�ciently taking advantage of thepre-determined localization of the regions. A 2-Dmatrix is created with the s values obtained for thewhole DSL image.

With the purpose of discriminating noise a simpleproof called the concavity test is applied each timethat a fringe is detected. This test is based in the pre-determined pro®le form of the fringe.

6. Experimental procedure and results

The DSL system implemented for our reconstruc-tion experiments is shown in Fig. 1. A 632 nmHe Ne laser was used as the light source, the spheri-cal lens has a focal distance of 30 cm and its diam-eter is 10 cm and the 1-D-binary grating had aspatial frequency of 20 lines per mm. A CCD cam-era was employed to detect and digitalize the DSLimage. The camera is positioned in at selected angleof view and is set in its large depth of ®eld mode.Finally, a PC computer was employed to process theDSL image information.

Fig. 9 shows a DSL image of 396� 345 pixels corre-sponding to a rectangular block model in oblique pos-


Fig. 9. DSL image of block model.

Fig. 10. DSL reconstruction, ®rst view.


ition. The block is 18 mm height and its square base is30 mm width.

We applied the tracking algorithm and we deter-mined the s-parameter matrix using the least squareerror method. For the upper block surface we obtaineda correct mean measurement of 18 mm with a standarddeviation of 1 mm. Figs 10 and 11 show the depthmap obtained. Fig. 10 shows the depth coordinate(vertical axis) which corresponds to the height of themodel in mm. Fig. 11 shows the block in its obliqueposition. The image was sampled on the x-axis for aperiod of 4 pixels. The (0±100) scale corresponds tothe image samples on x; the (0±20) scale correspondsto the fringes.

With a standard pentium processor (100 MHz), theexecution speed of the algorithm for an image with thementioned size is less than 1/4 s. This fact permits thepossibility of a real time implementation of the system.

7. Conclusions

The a priori information we can get from the DSLimages can be integrated in a simple and e�cient wayto the 3-D reconstruction algorithms. The depth mapcan be e�ciently obtained using only one image per-mitting the possibility of a real time implementation ofthe system. The applicability of the method is demon-strated for problems that require not more than 1 mmaccuracy.

We conclude that the DSL method for 3-D recon-struction is especially applicable in automatic inspec-tion systems and robot vision.

Acknowledgements

We thank Luis S. Hinojosa and Hector Guzman fortheir helpful assistance.

Appendix A

The Fresnel ®eld at distance z after the aperture isgiven by [10]

Uz�x; y� � eikz

ilz

�UL�x0; y0�eik

2z �xÿx0�2��yÿy0�2� �

dx0 dy0; �A1�

where UL is the ®eld at the aperture plane. The phasesintroduced by the cylindrical and spherical lenses canbe expressed as

UL�x0; y0� � U0�x0; y0�eÿik2f x 2

0�y 2

0� � eÿ ik2fC

x 20 : �A2�

Substituting (A1) in (A2) gives

Uz�x; y� ��fact�z��

U0�y0� eÿik2

ÿ1zÿ 1

fL

�x 20 eÿ

ik2

1zÿ1

f

ÿ �y 20

eÿi2p x0�x=lz��y0�y=lz�� dx0 dy0; �A3�where (fact)z is a phase factor and fL is de®ned by

1

fL� 1

f� 1

fC: �A4�

The integrating term in (A3) can be expressed bythe product of a function depending only on x andanother function depending only on y. Therefore,(A3) can be written as the product of two 1-D

Fig. 11. DSL reconstruction, second view.


Fourier transforms:

Uz�x; y� � �fact�z=1D U0�y0�eÿik2

ÿ1zÿ1

f

�y 20

� �v�u=lz

=1D eÿik

2

ÿ1zÿ 1

fL

�x 20

� �u�x=lz

�A5�

References

[1] Pennington KS, Will PM, Shelton GL. Grid-coded: a technique

for extraction of di�erences from scenes. Optics

Communications 1970;2:113±9.

[2] Will PM, Pennington KS. Grid coding: a pre-processing tech-

nique for robot and machine vision. Arti®cial Intelligence

1971;2(3/4):319±29.

[3] Popplestone RJ, Brown CM, Ambler AP, Crawford GF.

Forming models of plane-and-cylinder faceted bodies from light

strips. Proceedings 4th IJCAI, September 1975. 1975;664(668).

[4] John E, Lawrence P. An investigation of methods for determin-

ing depth from focus. IEEE Transactions on Pattern Analysis

and Machine Intelligence 1993;15(No. 2):97±108.

[5] Pentland AP. A new sense for depth ®eld. IEEE Transactions

on Pattern Analysis and Machine Intelligence 1987;PAMI-9(No.

4):523±31.

[6] Darrel T, Wohn K. Depth from focus using a pyramid architec-

ture. Pattern Recognition Letters 1992;11:787±96.

[7] Shah UB, Nayar SK. Extracting 3-D structure and focused

images using an optical microscope. Proceedings, Fifth Annual

IEEE Symposium on Computer-Based Medical Systems

1992;292±301.

[8] Subbarao M, Choi T. Accurate recovery of three-dimensional

shape from image focus. IEEE Transactions on Pattern

Analysis and Machine Intelligence 1995;17(No. 3):266±74.

[9] JaÈ hne B, Geiûler P. Depth from focus with one image.

Proceedings 1994 Computer Society Conference on Computer

Vision and Pattern Recognition 1994;713±7.

[10] Goodman J.W. Introduction to fourier optics. New York:

McGraw-Hill, 1996.

[11] Papoulis A. Probability, random variables and stochastic pro-

cesses. New York: McGraw-Hill, 1984.


recovery of three-dimensional shapes by using defocused structured light

Documents