Responses to the Comments on“Plane-Based Optimization for 3D Object

Reconstruction from Single Line Drawings”

Jianzhuang Liu, Senior Member, IEEE,Liangliang Cao, Zhenguo Li, and

Xiaoou Tang, Fellow, IEEE

Abstract—We disagree with the comments made by Varley [1] on our previous

paper [2]. In this paper, we respond to his comments and show that they are not

correct.

Index Terms—3D reconstruction, degree of reconstruction freedom, line

drawings, singular value decomposition.

Ç

1 INTRODUCTION

VARLEY’S main comments include

1. we oversimplified the 3D reconstruction problem;2. we did not compare our work with some work mentioned

by Varley;3. our Algorithm 1 is not necessary;4. our “minimum variable” idea is misconceived; and5. we did not mention an intractable problem of finding

resolvable sequences.

In this paper, we respond to all of these comments and show that

they are not correct.

2 PROBLEM STATEMENT AND FORMULATION

In Section 2 of [1], Varley considered that our statement, “The

ultimate target of line drawing interpretation is to reconstruct 3D objects

from 2D line drawings,” is wrong and should be changed to “The

ultimate target of line drawing interpretation is to reconstruct those 3D

object which a human would interpret 2D line drawings as portraying.”First, our statement is not a formal definition of line drawing

interpretation. Second, a reconstructed 3D object should be in

accordance with human perception, which is implied obviously. If

the 3D object is not what we expect, it is a failure. Third, in the first

paragraph of our paper, we already explicitly stated that “The

human vision system has the ability to interpret 2D line drawings as 3D

objects without difficulty. Emulating this ability is an important research

topic for machine vision.” This is the objective of our paper.Varley also commented that “The only constraints Liu et al. place

on their variables is the one that their problem statement requires: All

vertices must lie on faces. They ignore other constraints (such as that

faces which appear parallel must be parallel) which are important to

constructing the human-preferred interpretation. This, in turn, affects the

design of their algorithms, which only count the number of degrees of

freedom allowed by the object topology, not those which remain after

important constraints have been satisfied.”Our formulation of the 3D reconstruction problem is given in

Section 5 of [2] as follows:

minimize �ðfÞ; ð1Þ

subject to f 2 NullðPÞ: ð2Þ

The constraint of face planarity is the constraint in (2). However,

other constraints can be put in the objective function �ðfÞ (see (1),(4), and (5) in [2] for more details). In our experiments, to show the

power of our approach, we used only the constraint of minimizingthe standard deviation of the angles (MSDA) in �ðfÞ together with

the constraint of face planarity in (2). Even though only these twoconstraints are used, we did not deny other constraints to be addedinto �ðfÞ. We showed a distorted object in Fig. 12 of [2] and

pointed out that when the constraint of line parallelism is alsoimposed, the distortion can be corrected.

Ignoring the fact that minimizing �ðfÞ can also impose otherconstraints, Varley’s comment is incorrect.

3 PREVIOUS WORK

Varley listed several papers [3], [4], [5], [6], [7] and claimed that“Failure to take proper account of previous work leads them to compare

their method, not with the true state of the art, but with older, less

successful approaches.” Among these papers, [3] only shows twovery simple objects (a truncated pyramid and an L-block) in the

experiments; [4] and [5] are not related because they do not discuss3D reconstruction from single 2D line drawings (we cannot even

find the keyword “reconstruction” or “line drawing” in the twopapers); [6] is about 3D reconstruction from 3D range data but not

from line drawings; and [7] does not give any 3D reconstructionresults.

In our experiments, we compared our method with two mostrelated ones in [8] and [9]. The method in [8] also uses only two

constraints (MSDA and face planarity) and [9] can handle thewidest range of planar objects. Note that both our work [2] and [9]

aim at the 3D reconstruction of general objects that can bemanifolds, nonmanifolds, and nonsolids, where the line drawingsare with all edges and vertices visible. Considering the range and

complexity of the objects handled by the method in [9] and itsrobustness to human input error, we believe that it is the state of

the art.

4 ALGORITHM 1

In Section 4 of [1], Varley thought that “There is a minor bug in

Algorithm 1 as described in the paper: If the object framework is not

graph-connected (see Fig. 2 for an example of a valid but not graph-

connected framework), it loops forever as the set F of unprocessed faces

remains nonnull but none of them is a neighboring face of LD, the set of

processed faces.”We do not agree that there is a bug in Algorithm 1. Our paper

assumes that the correct face topology of a line drawing is known,

as stated in the second paragraph of Section 3 in [2]. Therefore, forthis object with a hole (Fig. 2 in [1]), each of the top and bottom

faces is represented by two cycles. With this information,Algorithm 1 does not loop forever. It is possible to have other

interpretations of the face topology from this object (e.g., the objecthas no hole), but, if the correct face topology is given, Algorithm 1

always converges.Varley also thought that the degree of freedom of an object

translating along the z-axis should be ignored. In fact, we alreadydiscussed this issue in the footnote of Section 5 in [2].

1726 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 31, NO. 9, SEPTEMBER 2009

. J. Liu, Z. Li, and X. Tang are with the Department of InformationEngineering, The Chinese University of Hong Kong, Hong Kong.E-mail: {jzliu, zgli, xtang}@ie.cuhk.edu.hk.

. L. Cao is with the Department of Electrical and Computer Engineering,University of Illinois at Urbana-Champaign, Urbana, IL 61801.E-mail: cao4@uiuc.edu.

Manuscript received 30 Mar. 2009; accepted 6 May 2009; published online 15May 2009.Recommended for acceptance by R. Zabih.For information on obtaining reprints of this article, please send e-mail to:tpami@computer.org, and reference IEEECS Log NumberTPAMI-2009-03-0200.Digital Object Identifier no. 10.1109/TPAMI.2009.118.

0162-8828/09/$25.00 � 2009 IEEE Published by the IEEE Computer Society

The most serious comment Varley made on Algorithm 1 is: “Is

the algorithm necessary at all?” The purpose of Algorithm 1 is to findan upper bound ub of the degree of reconstruction freedom (DRF)for a line drawing. Varley used the following two special cases toclaim that this algorithm is not necessary at all: 1) “for triangulated

mesh models ub ¼ v, where v is the number of vertices,” and 2) “for most

graph-connected wireframes which represent solid polyhedra, ub ¼ nþ 3

where n is the number of discrete subgraphs which remain if we delete all

nontrihedral vertices and edges touching them.”Case 1 is actually rare where a line drawing consists of only

triangulated faces. For Case 2, the scheme Varley suggested oftenfails. One simple example is given in Fig. 1a, which is a graph-connected line drawing representing a solid polyhedron, alsoshown in [2]. The DRF of this line drawing is 5 as discussed in [2].However, according to Varley’s scheme, the remaining subgraph isshown in Fig. 1b after deleting all non-trihedral vertices and edgestouching them, and ub ¼ 1þ 3 ¼ 4, which is obviously wrong.

It should be emphasized that our method handles generalplanar objects that can be manifolds, nonmanifolds, and nonsolids.A simple nonsolid example is shown in Fig. 1c. The DRF of it isclearly 5 and our Algorithm 1 also outputs 5. We cannot use ub ¼nþ 3 (¼ 4 for this example) to obtain the correct answer.

Varley claimed that Algorithm 1 is unreliable and used the linedrawing shown in Fig. 3 in [1] as an example. Algorithm 1 finds theupper bound of its DRF ub ¼ 5, based on the face topology foundby our face identification algorithm in [10]. (In [2], it is stated that,to find an upper bound as close to the DRF as possible, Algorithm 1is run multiple times, say 10, and the smallest upper bound ischosen.) Varley argued that it should be 4 with “the fact that the

object is a quasi-normalon, with all vertices lying on the intersections of

axis-aligned faces.”The core of this argument is: Should much geometric informa-

tion be used to find ub? Algorithm 1 mainly uses topologicalinformation (the face topology and the graph of a line drawing) tofind ub. Geometric information is not reliable from freehand linedrawings and there has not been an algorithm that can find theDRF from a general line drawing. Algorithm 1 gives a reliablesolution to finding the upper bound of the DRF. Note that ub ¼ 5

obtained by Algorithm 1 is correct for the object in Fig. 3 in [1].In summary, Varley’s claim that Algorithm 1 is unnecessary is

invalid. One scheme he suggested is limited to only triangulatedmesh line drawings. The other scheme is suitable for only partialsolid polyhedra. Instead, our Algorithm 1 can always give correctresults for general line drawings that can be manifolds, nonmani-folds, and nonsolids.

5 MINIMUM NUMBER OF VARIABLES

Varley pointed out that the main problem in some previousmethods, which create a minimum-variable system to derive thez coordinates of the vertices of a line drawing, is that they may givea mediocre solution for an imperfect line drawing. Since we also

used this minimum variable idea, he believed that this ismisconceived.

Different from the previous methods, this minimum variableidea in our method is not used to derive the z coordinates but tofind the upper bound of the DRF of a line drawing (Algorithm 1).This upper bound is then employed to determine the dimension ofthe initial searching space, which is then expanded to toleratesketch errors in the line drawing. The 3D reconstruction is carriedout by Algorithm 2 that searches for the optimal face parametervector f � in the expended space such that the objective function�ðfÞ is minimized. Another way of allowing freehand sketch errorsin our algorithm is that the faces of the reconstructed object are notrequired to be strictly planar (see Step 7 of Algorithm 2 in [2]). Ourexperimental results have shown many successful 3D reconstruc-tion results from complex practical line drawings that do not needto be drawn precisely.

6 RESOLVABLE REPRESENTATION

Varley thought that we should mention an intractable problem offinding resolvable sequences and claimed that “Liu et al.’s overallmethod will in any case fail for these objects because of the resolvablerepresentation problem.” This claim is not true.

Resolvable representation is a boundary representation for a3D polyhedron, which consists of the vertices and faces of theobject. From a resolvable representation, the polyhedron can bereconstructed in a step-by-step incremental manner. It is proventhat a polyhedron homeomorphic to a sphere has a resolvablerepresentation [11].

Our 3D reconstruction method has nothing to do with finding aresolvable sequence from a line drawing. It can be summarized as:1) finding an upper bound ub of the DRF from a line drawing,2) using singular value decomposition (SVD) to obtain a low-dimensional space based on ub, and 3) searching for the optimalface parameter vector f � in the space with a common optimizationalgorithm (quasi-Newton in [2]) such that �ðfÞ is minimized. Notethat none of these steps involves finding a resolvable sequence.Our algorithm can tackle objects without resolvable sequences. Infact, most of the objects in our paper [2] have no resolvablesequences but can be reconstructed successfully.

Varley showed two examples in Figs. 3 and 4 in [1] and thoughtthat our method could not handle them. In fact, our algorithm cangenerate correct 3D models from them, as shown in Fig. 2.

7 CONCLUSION

We have responded to the comments raised by Varley. Ourformulation of the 3D reconstruction problem is general. Althoughonly two constraints (face planarity and MSDA) were used in theexperiments, other constraints can be easily added to the objectivefunction. In [2], we compared our algorithm with Lipson andShpitalni’s [9], which was the state of the art before our paper [2]was published, in terms of the range and complexity of objects itcan handle and its robustness to human input error. Varley’s claimthat our Algorithm 1 is not necessary is invalid because he did notprovide a way that can deal with general objects but gave two

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 31, NO. 9, SEPTEMBER 2009 1727

Fig. 1. (a) A simple solid line drawing. (b) The remaining subgraph after deleting all

non-trihedral vertices and edges touching them from (a). (c) A simple nonsolid line

drawing. (d) The remaining subgraph after deleting all nontrihedral vertices and

edges touching them from (c).

Fig. 2. Our 3D reconstruction results from the two line drawings in Figs. 3 and 4 in

[1], each displayed in two viewpoints.

schemes that are limited to two special cases. Instead, ourAlgorithm 1 can always output correct results for general objects.We point out that our 3D reconstruction method does not have theproblems existing in previous methods that create a minimum-variable system or depend on a resolvable sequence. From theexperiments in [2], it is clear that our algorithm can handle generalobjects that can be manifolds, nonmanifolds, and nonsolids, withor without a resolvable sequence, from practical line drawings. Inthis paper, we also show successful reconstruction results fromtwo line drawings Varley gave in [1], which he thought ourmethod could not tackle.

REFERENCES

[1] P.A.C. Varley, “Comments on ‘Plane-Based Optimization for 3D ObjectReconstruction from Single Line Drawings’,” IEEE Trans. Pattern Analysisand Machine Intelligence, vol. 31, no. 8, pp. , Aug. 2009.

[2] J. Liu, L. Cao, Z. Li, and X. Tang, “Plane-Based Optimization for 3D ObjectReconstruction from Single Line Drawings,” IEEE Trans. Pattern Analysisand Machine Intelligence, vol. 30, no. 2, pp. 315-327, Feb. 2008.

[3] I.J. Grimstead and R.R. Martin, “Creating Solid Models from Single 2DSketches,” Proc. Third Symp. Solid Modeling and Applications, pp. 323-337,1995.

[4] J.X. Ge, S.C. Chou, and X.S. Gao, “Geometric Constraint Satisfaction usingOptimization Methods,” Computer-Aided Design, vol. 31, pp. 867-879, 1999.

[5] A.V. Kumar and L. Yu, “Sequential Constraint Imposition for Dimension-Driven Solid Models,” Computer-Aided Design, vol. 33, pp. 475-486, 2001.

[6] F.G. Langbein, A.D. Marshall, and R.R. Martin, “Choosing ConsistentConstraints for Beautification of Reverse Engineered Geometric Models,”Computer-Aided Design, vol. 36, pp. 261-278, 2004.

[7] Y.T. Li, S.M. Hu, and J.G. Sun, “On the Numerical Redundancies ofGeometric Constraint Systems,” Proc. Ninth Pacific Conf. Computer Graphicsand Applications, pp. 118-123, 2001.

[8] Y. Leclerc and M. Fischler, “An Optimization-Based Approach to theInterpretation of Single Line Drawings as 3D Wire Frames,” Int’l J.Computer Vision, vol. 9, no. 2, pp. 113-136, 1992.

[9] H. Lipson and M. Shpitalni, “Optimization-Based Reconstruction of a 3DObject from a Single Freehand Line Drawing,” Computer-Aided Design,vol. 28, no. 8, pp. 651-663, 1996.

[10] J. Liu and Y. Lee, “A Graph-Based Method for Face Identification from aSingle 2D Line Drawing,” IEEE Trans. Pattern Analysis and MachineIntelligence, vol. 23, no. 10, pp. 1106-1119, Oct. 2001.

[11] K. Sugihara, “Resolvable Representations of Polyhedra,” Discrete andComputational Geometry, vol. 21, pp. 243-255, 1999.

. For more information on this or any other computing topic, please visit ourDigital Library at www.computer.org/publications/dlib.

1728 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 31, NO. 9, SEPTEMBER 2009