image-based modeling of 3d objects with curved surfacesimage.inha.ac.kr/paper/cavw07_park.pdf ·...

COMPUTER ANIMATION AND VIRTUAL WORLDSComp. Anim. Virtual Worlds 2008; 19: 93–109Published online 6 September 2007 in Wiley InterScience(www.interscience.wiley.com) DOI: 10.1002/cav.216...........................................................................................Image-based modeling of 3D objectswith curved surfaces

By Man Hee Lee and In Kyu Park*

..........................................................................

This paper addresses an image-based method for modeling 3D objects with curved surfacesbased on the non-uniform rational B-splines (NURBS) representation. The user fits thefeature curves on a few calibrated images with 2D NURBS curves using the interactive userinterface. Then, 3D NURBS curves are constructed by stereo reconstruction of thecorresponding feature curves. Using these as building blocks, NURBS surfaces arereconstructed by the known surface building methods including bilinear surfaces, ruledsurfaces, generalized cylinders, and surfaces of revolution. In addition to them, we alsoemploy various advanced techniques, including skinned surfaces, swept surfaces, andboundary patches. Based on these surface modeling techniques, it is possible to buildvarious types of 3D shape models with textured curved surfaces without much effort.Copyright © 2007 John Wiley & Sons, Ltd.

Received: 10 August 2006; Revised: 5 July 2007; Accepted: 23 July 2007

KEY WORDS: image-based method; curved surface; NURBS; corresponding curve; interactiveuser interface; stereo reconstruction; surface modeling

Introduction

Recent progress in multimedia technology has beenfocused on the creation, manipulation, and display of3D graphics content, rather than conventional mediasuch as images, videos, and sounds. Therefore, thereis a necessity for advanced technologies which createand manage 3D graphics content more efficiently atreasonable cost in the areas of academia, industry, andconsumer electronics.

The life-cycle of any multimedia content including3D graphics content generally consists of four steps—authoring, storage, transmission, and rendering. Therehave been a lot of previous works involving eachof these pipeline components. In order to efficientlystore and transmit 3D graphic models, the techniquesof data compression and streaming over the networkhave been actively developed as a natural extensionto the corresponding techniques for image and videodata. The 3D graphics community has developed alot of excellent algorithms for real time rendering and

*Correspondence to: I. K. Park, School of Information andCommunication Engineering, Inha University, 253 Yonghyun-dong, Nam-gu, Incheon 402-751, Korea. E-mail: [email protected]

global illumination, using hardware accelerators as theworkhorses.

Among the four ingredients in the life-cycle pipeline,the authoring step remains the most underdevelopedarea. Unlike images and videos, which people cancapture using the common camera and camcorder,creating 3D graphics content still remains an area forspecialized people such as 3D animators and graphicsdesigners. Usually, professional designers create 3Dgraphics content using expensive commercial software,in what amounts to a time-consuming task requiring agreat deal of effort, even for them. In this context, it isnecessary to develop efficient techniques which allowgraphics content to be created more easily, so that regularusers can create and utilize their own content.

In order to overcome this problem, the image-basedapproach known as image-based modeling (IBM) hasattracted a great deal of interest as an alternativemethod of creating 3D content.1–3 Image-based geometrymodeling is based on 3D computer vision technology,4 inwhich only a few images are used as the input and the3D geometry is reconstructed by stereo reconstructionmethods. IBM allows for the fast acquisition of 3D shapeswith a photorealistic texture without an exorbitantcomputational cost.

............................................................................................Copyright © 2007 John Wiley & Sons, Ltd.

M. H. LEE AND I. K. PARK...........................................................................................

However, the traditional IBM algorithms mostlyreconstruct 3D point geometry and its subsequenttriangulation in order to construct the 3D mesh model.This approach has few problems, since it cannot captureprecise curved geometry. That is, it produces a noisytriangulation of the 3D surface even though the surfaceis a mathematically continuous one such as a sphere,cylinder, or other type of surface. It would be much moreprecise and simple if the 3D surface were reconstructedand defined in mathematical form directly. Note that thetraditional approach is more appropriate when modelingpolyhedral shapes rather than curved shapes.

In this paper, we propose novel techniques for model-ing the shape and texture of objects with curved shapes,using an image-based approach and efficient user inter-action. The proposed approach is based on non-uniformrational B-splines (NURBS) surface reconstruction andtexture mapping. Note that the NURBS surface can repre-sent a wide variety of curved surfaces in exact mathemat-ical form and its shape modification is highly interactive,which makes it possible to use it for the geometric prim-itive employed for image-based surface reconstruction.

The proposed surface modeling techniques includebuilding methods for surfaces such as bilinear surfaces,ruled surfaces, generalized cylinders, and surfacesof revolution. In addition, we propose various ad-vanced surface modeling techniques, including skinnedsurfaces, swept surfaces, and boundary patches.† Byreconstructing the materials, such as boundary curvesand spines, in an image-based way and employing oneof the NURBS surface building algorithms, it is possibleto reconstruct the curved surface defined in the 3DNURBS surface. In addition, a view-dependent texturemodeling technique is proposed to capture the texture ofthe NURBS surface consistently. Together with a few re-constructed and textured NURBS surfaces, the process ofmodeling the object is finalized to produce photorealistic3D objects with arbitrarily curved surfaces. Note that theproposed technique can model polyhedral shapes too.

This paper is organized as follows. In the following twosections, previous works on IBM and the overall structureof the proposed approach are introduced. Subsequently,the proposed image-based NURBS surface modelingtechnique is described in detail. Then the experimentalresults are shown and discussed. In the last section, wegive our conclusive remarks.

†In this paper, ‘skinned’ has a different meaning from the sameterm in computer animation. This is one of the surface buildingmethods in NURBS5.

Previous Works

The traditional approach in computer vision and com-putational photometry is based on stereo reconstruction,in which 3D shape is reconstructed from multiple 2Dimages using stereo matching and the triangulationmethod.2 In stereo reconstruction, the correspondencebetween the image features commonly observed intwo or more images is found manually or by usinga matching algorithm. Then, the camera calibrationmatrices are used to estimate the depth of the featurepoint. Most stereo reconstruction algorithms are basedon this framework and the problem of determiningthe feature correspondence robustly and calibrating thecamera accurately remains an open issue. Note that thesame vision technique can be used in the renderingpart, known as image/video-based rendering, which hasbeen investigated more actively in computer graphicscommunity.6–8

Another useful IBM technique is the shape-from-silhouette method, in which the voxel-based geometryis obtained by continuously removing those voxelswhich do not fall into the object foreground in theinput images.9 More advanced techniques are the voxel-coloring and space-carving algorithms,10,11 in which thephoto consistency of the voxel color is employed, ratherthan the simple foreground–background discriminationof the voxels.

Rather than the conventional works described above,which reconstruct the dense point geometry, Debevec’spioneering work in the IBM of the architectural structureemploys the simple user-interactive specification of thecorrespondence between line features.1 In addition,Debevec proposed an optimization technique for placing3D geometric primitives such as cubes and quadrilateralcones based on the semantic information of thearchitectural structure. This strategy is very usefulfor modeling man-made polyhedral structures, and anextended version of this method is used in commercialmodeling software.12

Most commercial software programs in this categorymake use of simple geometric primitives, includingthe point, line, and plane, in their interaction withthe user. For example, in PhotoModelerTM,13 the userspecifies the feature points as well as the connectioninformation between them in order to produce a3D polyhedral model. ImageModelerTM14 includes anautomated module which finds the correspondencebetween the whole feature points out of the few pointsof correspondence specified by the user. In addition, it

............................................................................................Copyright © 2007 John Wiley & Sons, Ltd. 94 Comp. Anim. Virtual Worlds 2008; 19: 93–109

DOI: 10.1002/cav

IMAGE-BASED MODELING OF 3D OBJECTS...........................................................................................

provides a few 3D geometric primitives such as the plane,cube, cylinder, sphere, and disk; and deforms them toreconstruct the 3D shape. In order to represent curvedsurfaces, it provides a triangulation tool using a 3D pointscloud which, although it is not perfect, can be used torepresent a curve very precisely in a mathematical way.Canoma12 has an interesting feature that reconstructsthe 3D structure from a hand-sketched or printed sheetof objects, by placing and deforming the predefined 3Dprimitives manually onto them.

However, since these techniques reconstruct the 3Dshape using simple and predefined 3D geometricprimitives, it is difficult to precisely reconstruct the shapeof objects with a few curved surfaces which have anexact mathematical representation. The reconstructedshape usually suffers from noise if it is reconstructed indense mesh form, whereas the noise could be removedif the surface were reconstructed in mathematical formdirectly.15 Therefore, in this paper, we propose an efficientapproach to reconstruct 3D objects with a curved surfacebased on the NURBS surface, which can represent avariety of shaped surfaces in clear mathematical form.‡

Overview of the ProposedImage-Based Modeling

System

The IBM technique proposed in this paper performsthe reconstruction of 3D shapes with curved surfacesusing a few calibrated images as the input. Once thecorresponding curves in the input images are specifiedby the simple user interface, a 3D curve is reconstructedby employing the stereo reconstruction technique. Thecurves and surfaces used in this paper are modeled ina mathematical way using the NURBS representation,5

which enables us to build arbitrary curves with efficientinteractive modification of their shape. Note that the datapoints to interpolate are easily inserted, removed, andmoved by means of a simple user interface.

The proposed surface modeling algorithms are basedon NURBS curve and surface algorithms. They utilizethe basic and advanced surface modeling techniques,which will be addressed in the ‘NURBS SurfaceModeling’ subsection. As a special type of surface, apolyhedron can also be reconstructed using 1st degreeNURBS. Furthermore, disks as well as concave planar

‡The preliminary version of this work has been presented inReference [16].

objects, which are common shapes in the real-worldenvironment, can also be reconstructed. Photorealistictextures are captured from the input image usingview-dependent acquisition and mapped onto thereconstructed surface.

The input images and reconstructed 3D curvedsurfaces, as well as the intermediate results, are managedin a project workspace, which can export and reuse the3D model in a standard 3D graphic format, i.e., virtualreality modeling language (VRML).

Image-BasedReconstruction of 3D

Curved Surfaces Using theNURBS

In this section, we describe the proposed curve andsurface reconstruction techniques in detail.

Camera Calibration

In order to reconstruct 3D geometry, the cameraparameters of each input image should be estimatedbeforehand. In our approach, it is assumed that theexchangeable image file format (EXIF) information isavailable for the input image. Note that most of recentdigital cameras provide the EXIF information, which isembedded in the image file. For camera calibration, theinternal camera parameters are fetched from the EXIFinformation. The external parameters are computed byemploying the efficient 5-point algorithm.17

Modeling and Representation of 2DCurves Using the NURBS

As a prerequisite to reconstructing a 3D curved surface,3D curves need to be build in which each of them isreconstructed from a pair of corresponding 2D curves.In this paper, we use the NURBS curve to representarbitrary curves. The NURBS curve is defined by thecontrol points, Pi, the weight, wi of Pi, and the pth degreebasis functions, Ni,p(u), as follows.5

C(u) =

n∑i=0

Ni,p(u)wiPi

n∑i=0

Ni,p(u)wi

, 0 ≤ u ≤ 1 (1)


DOI: 10.1002/cav

M. H. LEE AND I. K. PARK...........................................................................................

Figure 1. Approximation of a curved feature using a NURBScurve.

in which the basis function, Ni,p(u), is defined over a knotvector, U (2) where m = p + n + 1.

U = {0, . . . , 0︸︷︷︸p+1

, up+1, . . . , um−p−1, 1, . . . , 1︸︷︷︸p+1

} (2)

In our approach, in order to approximate the curvedfeature of the object in the input images, the datapoints are incrementally added and a NURBS curveis instantly computed to interpolate them. In thisprocedure, local bicubic interpolation5 is performed tocreate the NURBS curve. In Figure 1, it is shown thatnine data points are inserted manually and the NURBScurves are reconstructed to approximate the contour ofthe clock’s side. This procedure is performed on each ofthe corresponding curves, which commonly exist in twoor more input images.

Reconstruction of 3D Curves UsingMultiple Input Images

In stereo reconstruction, the epipolar line simplifies theprocess of finding the pixel correspondence in differentimages, since it reduces the search area confined to theline. In our approach, it is assumed that the camera ispre-calibrated. Therefore, it is natural to use the epipolarconstraint in finding the data points of the corresponding

Figure 2. Finding the corresponding curve using the epipolarconstraint.

curve. Figure 2 shows the proposed procedure, in whichthe yellow line denotes the epipolar line of a data pointprojected onto the other input image. Since the clock’sside contour is being approximated, the next data pointthat should be specified would be the intersection ofthe epipolar line and the visual edge. By repeating thisprocedure, it is possible to obtain the correspondence setof the data points.

From these data points, we use stereo reconstructionto reconstruct the 3D data points. The reconstructed3D data points are used to recover the 3D NURBScurve, which is the 3D reconstruction of the ap-proximated 2D contour, by employing local bicubicinterpolation.5

NURBS Surface Representation

The NURBS surface with degrees p and q inthe u and v directions, respectively, is defined byEquation (3).

S(u, v) =

n∑i=0

m∑j=0

Ni,p(u)Nj,q(v)wi,jPi,j

n∑i=0

m∑j=0

Ni,p(u)Nj,q(v)wi,j

(3)

In Equation (3), Pi,j’s denote the points on the control netand wi,j is the weight of Pi,j . Ni,p(u) and Nj,q(u) refer to


DOI: 10.1002/cav


Surface modeling Inputs Outputstechniques

Bilinear surface Four corner points A surface which interpolates four corner pointsRuled surface Two opposite boundary curves A surface which interpolates the boundary curves

Generalized cylinder Section curve and axis A surface which is made by the section curveswept along the axis direction

Surface of revolution Generatrix curve and axis A surface which is made by the generatrix curverevolved around the axis

Skinned surface A few section curves to interpolate A surface which interpolates the section curvesSwept surface Section and trajectory curves A surface which is made by the section curve

swept along the trajectory curveBoundary patch Four boundary curves A surface which interpolate four curves

Table 1. The NURBS surface modeling methods

Figure 3. The reconstruction process of a bilinear surface. (a) Four corner points shown as two line segments, (b) reconstructedbilinear surface.

the basis functions defined over the knot vectors, U (4)and V (5), where r = p + n + 1 and s = q + m + 1.

U = {0, . . . , 0︸︷︷︸p+1

, up+1, . . . , ur−p−1, 1, . . . , 1︸︷︷︸p+1

} (4)

V = {0, . . . , 0︸︷︷︸q+1

, vq+1, . . . , vs−q−1, 1, . . . , 1︸︷︷︸q+1

} (5)

NURBS Surface Modeling

In this paper, the curved surfaces are reconstructedby choosing an appropriate NURBS surface modelingmethod, according to the surface characteristics. Based

on the surface type observed from the images, a setof proper 3D geometric primitives is determined andreconstructed, from which a selected surface modelingis performed to yield the output surface.

Table 1 shows the various NURBS surface modelingmethods which are employed in this paper.§ The bilinearsurface is the NURBS surface reconstructed by bilinearinterpolation of four corner points. In Figure 3, thereconstruction process of the bilinear surface is shown.The ruled surface is a linearly interpolated surfacedefined over two opposite boundary curves. Figure 4shows the reconstruction process of the ruled surface.

§The mathematical details are described fully in Reference [5].


DOI: 10.1002/cav

M. H. LEE AND I. K. PARK...........................................................................................

Figure 4. The reconstruction process of a ruled surface. (a) Two opposite boundary curves, (b) reconstructed ruled surface.

The generalized cylinder is the surface which isobtained by sweeping the section curve along the axis.Figure 5 shows the reconstruction process of thegeneralized cylinder. The surface of revolution is thesurface which is obtained by revolving the generatrixcurve around the axis. Figure 6 shows the reconstructionprocess of the surface of revolution. The skinned surfaceis an interpolating surface, which interpolates a set ofsection curves. Figure 7 shows the reconstruction processof the skinned surface. The swept surface is constructedby sweeping the section curve along the trajectory curve.Figure 8 shows the reconstruction process of the swept

surface. Finally, the boundary patch is a surface whichinterpolates four boundary curves. Figure 9 shows thereconstruction process of the boundary patch.

In addition to the surface modeling techniquesdescribe above, three additional techniques are proposedin our approach, i.e., a perfect disk, a concave planarface, and a polygonal face. The perfect disk canbe reconstructed from a perfect circle with propertessellation into triangles, as shown in Figure 10.

In order to represent a concave planar face, which isin fact the generalized shape of a disk, we proposed asimple but effective method based on contour partition

Figure 5. The reconstruction process of a generalized cylinder. (a) Section curve and axis, (b) reconstructed generalized cylinder.


DOI: 10.1002/cav


Figure 6. The reconstruction process of a surface of revolution. (a) Generatrix curve and axis, (b) reconstructed surface ofrevolution.

Figure 7. The reconstruction process of the skinned surface. (a) A set of section curves to interpolate, (b) reconstructed skinnedsurface.

Figure 8. The reconstruction process of a swept surface. (a) Section and the trajectory curves, (b) reconstructed swept surface.


DOI: 10.1002/cav

M. H. LEE AND I. K. PARK...........................................................................................

Figure 9. The reconstruction process of a boundary patch. (a) Four boundary curves, (b) reconstructed boundary patch.

and the ruled surface. The boundary of the concave faceis partitioned manually into a couple of separate curves,followed by ruled surface modeling using the separatecurves as input. Figure 11 shows an example of thisprocedure. It is observed that there is no seam at allbetween the clock’s side modeled by the proposedconcave face and the clock’s front and back surfacewhich is modeled by the generalized cylinder. This is

due to the fact that they share a common 3D curve asthe input.

Since a scene usually contains a mixture of objects withcurved and planar surfaces, it is necessary to representthe polygonal face in the same framework as that usedfor modeling a curved surface. Based on the fact that apiecewise polyline can be represented by a NURBS curveof degree 1, it is possible to model a polygonal face or

Figure 10. The reconstruction process of a disk. (a) A 3D perfect circle, (b) reconstructed disk.


DOI: 10.1002/cav


Figure 11. The reconstruction process of a concave planar face. (a) The concave planar curve, (b) reconstructed concave planarface, and (c) the concave planar face joining continuously with a generalized cylinder.

a 3D polyline, as well as to use them as the input forthe NURBS surface modeling techniques. In Figure 12, apolyline is used as the section curve for the generalizedcylinder and then the surface, which resembles a foldingscreen, is reconstructed as a result.

View-Dependent Texture Modelingof the NURBS Surface

In order to reconstruct the photorealistic model, thetexture as well as the geometry should be captured in a

Figure 12. The reconstruction process of a polygonal face. (a) A polyline, (b) reconstructed polygonal face.


DOI: 10.1002/cav

M. H. LEE AND I. K. PARK...........................................................................................

Figure 13. Scene 1 for modeling a few individual objects with different surface types. The total procedure takes 5 minutes for anintermediately-skilled user. (a) 5 input images, (b) reconstructed 3D NURBS curves, (c) reconstructed 3D NURBS surfaces, and

(d) reconstructed 3D models with texture mapping.


DOI: 10.1002/cav


Figure 14. Scene 2 for modeling a single object with different surface parts. The total procedure takes 20 minutes for anintermediately-skilled user. (a) 6 input images, (b) reconstructed 3D NURBS curves, (c) reconstructed 3D NURBS surfaces,

and (d) reconstructed 3D models with texture mapping.


DOI: 10.1002/cav

M. H. LEE AND I. K. PARK...........................................................................................

consistent way. Therefore, for each reconstructed NURBSsurface, photorealistic texture maps should be obtainedfrom the input images. Since the NURBS surface has theinherent u-v parametrization, it is natural to use these pa-rameters as the texture coordinates. In our approach, eachNURBS surface is reprojected onto the individual inputimages and the texel colors of the corresponding texturemap are read from the pixel at the reprojected position.

In this procedure, it is common for each texel tobe reprojected onto more than one image and it isnecessary to determine the texel color among themultiple corresponding pixels. In our approach, a view-dependent texture modeling technique is employed toblend the pixel colors. Let Vi be the viewing vector of theinput image i and Nu,v be the normal vector of the NURBSsurface with the parameters u and v. The weight factor ri,which accounts for the confidence measure of the textureconfidence, is defined as

ri = Vi · Nu,v (6)

The texel color TCu,v at (u, v) is computed by determiningthe weighted average of the pixel colors Ci’s, as definedin Equation (7)

TCu,v =∑

siCiri∑siri

, si =

1 if the image i is selected

0 otherwise

(7)

In some cases, there is occlusion between the objectsin the scene. Therefore, in this case, it is preferable toavoid using the image of the occluded object. In order toaccount for this, in Equation (7), the binary variable si isincluded, which is zero if the image i is excluded by theuser, due to occlusion or other reasons.

User-Interface and SystemFunctionality

All the ingredient modules are integrated in a projectworkspace, which includes the windows and graphicaluser interfaces for 2D images and the reconstructed 3Dscene. The user-interface for the 2D image is designedto edit the NURBS curve easily by adding, moving, anddeleting the data points using computer mouse. The user-interface for the 3D scene allows the standard rotation,translation, and zooming scheme. Overall, the wholesystem takes the set of input images and produces anoutput VRML file which contains the mesh geometry ofthe reconstructed 3D scene plus textures.

Experimental Results

In order to evaluate the performance of the proposedalgorithm, the experiments are carried out on few realtest senses. The experiments were performed on a 64-bitAMD Athlon 2.0 GHz CPU with 2 Gbyte main memoryand an NVIDIA GeForce 6800 GPU.

3D Objet Modeling

In our experiments, three real scenes are tested. Thecurved objects in each test scene are modeled usingthe proposed image-based NURBS surface modelingtechniques.

In Figure 13, a few individual objects with differentshapes are modeled using five input images. The clockis reconstructed with a generalized cylinder and aconcave planar face. The Coke can is reconstructedwith a generalized cylinder and a disk. The bottleof milk is reconstructed with a surface of revolutionand a disk. The bent paper is reconstructed with askinned surface. The Chinese tea cup is reconstructedwith a surface of revolution. Figure 13a shows the inputimages, and Figure 13b and c shows the reconstructedNURBS curves and surfaces, as the intermediate result.And, Figure 13d shows the final NURBS model withtexture. The modeling time is about 5 minutes for anintermediately-skilled user.

In Figure 14, a complex object, i.e., a projector ismodeled, which consists of different shaped surfaceparts. Modeling is performed with 11 ruled surfaces,a generalized cylinder, and a disk. Figure 14b and cshows the reconstructed NURBS curves and surface, andFigure 14d shows the final NURBS model with texture.The modeling time is about 20 minutes.

In order to show the reconstructed result of polyhedralshape, the bottle of milk is reconstructed with ageneralized cylinder in which the section curve is apolyline. Figure 15a shows four input images. Figure 15bshows the reconstructed NURBS curves and surfaces,and Figure 15c shows the final NURBS model withtexture. The modeling time is about 2 minutes.

The reconstructed 3D surface models can be exportedin common graphics formats such as VRML. Figure 16shows the exported 3D models from the reconstructionshown in Figure 13d, which is rendered in a commonVRML viewer.

In Figure 17, a set of the reconstructed textures ofthe test models is shown. Figure 18 shows the texturemapping results using different texture acquisitionmethods. Figure 18a shows the result when the proposed


DOI: 10.1002/cav


Figure 15. Scene 3 for modeling a polyhedral object. The total procedure takes 2 minutes for a intermediately-skilled user. (a) 4input images, (b) reconstructed 3D NURBS curves and surfaces, and (c) reconstructed 3D models with texture mapping.

view-dependent texture modeling technique is used,while Figure 18b shows the result when the nearesttexture is used. It is observed that the texture mappingbased on the proposed technique is better in terms ofsmooth color transition and consistency with the inputimages.

Quantitative Analysis

Although the reconstructed 3D models are observed tobe photorealistic and consistent visually with the inputimages both in shape and texture, cross-checking isperformed to measure the performance quantitatively.


DOI: 10.1002/cav

M. H. LEE AND I. K. PARK...........................................................................................

Figure 16. The rendering result of the reconstructed objects in the VRML viewer.

Figure 17. The reconstructed textures.


DOI: 10.1002/cav


Figure 18. The comparison between texture modeling method. (a) Using view-dependent texture, (b) using the nearest texture.

In this experiment, the test scene shown in Figure 13 isused. First, the reconstructed 3D models are reprojectedonto the input images to evaluate the amount ofobject region overlap. It is observed that the non-overlapping region is 6.79% of the area of the wholeobject region. Next, the difference of the color betweenthe corresponding pixels is computed in PSNR. In Table 2,PSNR scores are shown for a few typical resolutionsof texture. It is observed that it is appropriate toselect the texture resolution similar to the resolutionof the effective segmented region of the object in theinput image.

Once there is occlusion between the objects in the inputimage, such an undesirable image can be excluded fromthe texture modeling by user interaction. Therefore, it ispossible to avoid generating visually inconsistent textureof the occluded object. Table 3 shows the difference ofPSNR when the texture is obtained with or without thisoption.

Texture resolution PSNR (dB)

256 × 256 19.14512 × 512 19.19

1024 × 1024 18.44

Table 2. Comparison of PSNR, different textureresolution

Image selection method PSNR (dB)

All images 19.19Selected images 18.28

Table 3. Comparison of PSNR, different imageselect method. The resolution is 512 × 512

Note that the quantitative error does not seem to benegligible at first glance. However, in most cases, thewhole texture on a single NURBS patch is slightly shiftedon the reprojected image. Therefore, the qualitativevisual quality remains almost same as the original one.

Limitations

The performance of the proposed approach is dependenton the selection of the particular camera calibrationalgorithm. A small error in camera calibration wouldresult in visible mismatch between the reprojectedposition and the original position in the input image.This is exemplified in Figure 14d, in which a small glitchis observed at the right side of the projector model.This is due to the different calibration error for theimages used in modeling the front and the side partof the projector. Note that the 5-point algorithm17 isused in our approach, since there is no need to use thecalibration checker board. However, it is obvious thatthe more accurate is the calibration method used (for


DOI: 10.1002/cav

M. H. LEE AND I. K. PARK...........................................................................................

example Zhang’s algorithm18), the more accurate will bethe surface modeling result.

Another limitation of the proposed approach is that itdepends on the surface texture of the object when fittingfeature curve on the input image. It means that it is notalways possible to find the visible feature curve if there isno apparent texture on the object surface. Because of this,the revolving curve is painted manually on the surface ofthe milk container, as shown in Figure 13a. The problemcan be solved properly if dense stereo matching is appliedon the textures within the boundary curves and the resultis fit with the NURBS surface. Note that it is consideredfor future research.

Conclusions and FutureWorks

In this paper, we proposed an image-based methodfor reconstructing 3D shapes with curved surfacesusing the NURBS representation. Starting from a fewcalibrated images, the user specifies the correspondingcurves by means of an interactive user interface. Then,the 3D curves are reconstructed by employing stereoreconstruction of the control points followed by thelocal interpolation of the NURBS curve. The proposedsurface modeling techniques include surface buildingmethods such as bilinear surfaces, ruled surfaces,generalized cylinders, and surfaces of revolution. Inaddition, we also propose various advanced surfacemodeling techniques including skinned surfaces, sweptsurfaces, and boundary patches. Based on these surfacemodeling techniques, it is possible to build various typesof 3D shape models with curved surfaces without mucheffort. The constructed 3D shape model with curvesand curved surfaces can be exported in VRML format,making possible for it to be used in different 3D graphicssoftware programs. The proposed techniques can be usedin general 3D graphics and CAD authoring tools, as wellas in authoring tools used for augmented reality.

Although the proposed techniques provide an efficientmethod of modeling 3D objects with curved surfaces,they still rely on user interaction. That is, thecorresponding curves have to be specified manually.The future direction of our research will be to focuson the automation of this part of the process. Bydeveloping curve matching algorithms which can dealwith partial curves, the correspondence specification canbe done automatically. In addition to this, the semanticinformation of the objects to be modeled can be usedefficiently, yielding curve matching algorithms based on

the active shape model (ASM)19 and active appearancemodel (AAM)20. The final and ultimate goal will be thedevelopment of a fully automated and more accurateIBM algorithm for more generally shaped object.

ACKNOWLEDGEMENT

This work was supported by Electronics and Telecommunica-tions Research Institute (ETRI) under contract 0701-2005-0064and Inha University.

References

1. Debevec P. Modeling and rendering architecture fromphotographs. Ph.D. Thesis, University of California atBerkeley, 1996.

2. Pollefeys M. 3D models from photos and videos. SIGGRAPH2003, Course 15, 2003.

3. Košecká J, Ma Y, Soatto S, Vidal R. Multiple-view geometryfor image-based modeling. SIGGRAPH 2004, Course 23,2004.

4. Hartley R, Zisserman A. Multiple View Geometry in ComputerVision, 2nd edn. Cambridge University Press: Cambridge,2003.

5. Piegl L, Tiller W. The NURBS Book, 2nd edn. Springer-Verlag:Berlin Heidelberg, 1997.

6. McMillan L, Bishop G. Plenoptic modeling: an image-basedrendering system. In Proceedings of SIGGRAPH 95, 1995; pp.39–46.

7. Matusik W, Buehler C, Raskar R. Image-based visual hulls.In Proceedings of SIGGRAPH 2000, 2000; pp. 369–374.

8. Magnor M, Pollefeys M. Video-based rendering. SIGGRAPH2005, Course 16, 2005.

9. Martin W, Aggarwal J. Volumetric descriptions of objectsfrom multiple views. IEEE Transactions on Pattern Analysisand Machine Intelligence 1983; 71(7): 789–802.

10. Seitz S, Dyer C. Photorealistic scene reconstruction by voxelcoloring. International Journal of Computer Vision 1999; 35(2):151–173.

11. Kutulakos K, Seitz S. A theory of shape by space carving.International Journal of Computer Vision 2000; 38(3): 199–218.

12. http://www.viewpoint.com/products/canoma/13. http://www.photomodeler.com/14. http://www.realviz.com/products/im/15. Park IK, Yun ID, Lee SU. Automatic 3-d model synthesis

from measured range data. IEEE Transactions on Circuits andSystems for Video Technology 2000; 10(2): 293–301.

16. Lee MH, Park IK. Image-based approach for modeling3d shapes with curved surfaces. In Proceedings of PacificGraphics, Taipei, Taiwan, October 2006; pp. 70–75.

17. Nister D. An efficient solution to the five-point relative poseproblem. IEEE Transactions on Pattern Analysis and MachineIntelligence 2004; 26(6): 756–770.

18. Zhang Z. A flexible new technique for camera calibration.IEEE Transactions on Pattern Analysis and Machine Intelligence2000; 22(11): 1330–1334.


DOI: 10.1002/cav


19. Cootes TF, Taylor CJ, Cooper DH, Graham J. Active shapemodels—their training and application. Computer Vision andImage Understanding 1995; 61(1): 38–59.

20. Cootes TF, Edwards GJ, Taylor CJ. Active appearancemodels. IEEE Transactions on Pattern Analysis and MachineIntelligence 2001; 23(6): 681–685.

Authors’ biographies:

Man Hee Lee received his B.S. degree in ComputerEngineering from Inha University in 1999. He is currentlyworking toward the M.S. degree in the Informationand Communication Engineering in Inha University.Since April 2007, he has been visiting Electronicsand Telecommunications Research Institute (ETRI) asa visiting graduate researcher. His research interestsinclude image-based modeling and rendering, sketch-based interface, GPGPU, and 3D game technology. Heis a student member of ACM.

In Kyu Park received his B.S., M.S., and Ph.D. degreesfrom Seoul National University (SNU) in 1995, 1997,and 2001, respectively, all in Electrical Engineering andComputer Science. From September 2001 to March2004, he was a Member of Technical Staff at SamsungAdvanced Institute of Technology (SAIT). Since March2004, he has been with the School of Information andCommunication Engineering, Inha University, wherehe is an Assistant Professor. Since January 2007,he has been visiting Mitsubishi Electric ResearchLaboratories (MERL) as a visiting scholar. Currentresearch interests include the joint area of computergraphics and vision, including image-based modelingand rendering, 3D face modeling, computationalphotography, and GPGPU. He is a member of IEEEand ACM.


DOI: 10.1002/cav

image-based modeling of 3d objects with curved surfacesimage.inha.ac.kr/paper/cavw07_park.pdf ·...

Documents