IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 13, NO. 4, JULY 2002 835

Shape Indexing Using Self-Organizing MapsP. N. Suganthan, Senior Member, IEEE

Abstract—In this paper, we propose a novel approach togenerating topology preserving mapping of structural shapesusing the self-organizing maps (SOM). The structural informationof the geometrical shapes is captured by the relational vectors.These relational attribute vectors are quantised using an SOM.Using this quantization SOM, a histogram is generated forevery shape. These histograms are treated as inputs to trainanother SOM which yields a topology preserving mapping ofthe geometric shapes. By appropriately choosing the relationalvectors, it is possible to generate the mapping invariant to somechosen transformations such as rotation, translation, scale, affine,or perspective. Experimental results using trademark objectsare presented to demonstrate the performance of the proposedmethodology.

Index Terms—Attributed relational graphs, pairwise geometrichistograms, relational attribute vectors, self-organizing maps,shape indexing, shape recognition, shape retrieval, structuraldatabases, topology conserving mapping.

I. INTRODUCTION

DUE to the recent advances in storage, communications,image compression, and internet technologies, multi-

media information has become more popular. Consequently,more and more multimedia documents containing video clips,images and audio are being generated in diverse applicationareas including education, medicine, entertainment, sport,remote sensing, and online information services. With thisexplosive growth in the volume of multimedia informationarchives, the effective management of these multimediaarchives for efficient browsing and retrieval of desired infor-mation is of paramount importance. In recent years, severalintelligent techniques have been developed to perform thesetasks, as evidenced by the recent publications [1]–[3] on thesubject of content-based image retrieval.

The most commonly used properties of images for visualcontent-based retrieval are color [4], texture [5], shape [3],[6]–[9], spatial relationships between various properties [10], ora combination of these properties [1], [2], [11], [12]. Althoughseveral methods have been proposed to retrieve images, themost popular approach for indexing into image databases hasbeen the histogram indexing using the above listed properties[4]. In this paper, we propose a novel shape indexing schemeusing a structural histograming technique and the SOM algo-rithm. Prior to presenting our approach, we briefly review somerelevant image retrieval techniques using shape information.

The edge pixels in the images have been used by someresearchers to perform the shape-based similarity search. Hirata

Manuscript received April 28, 2001; revised November 5, 2001.The author is with the School of Electrical and Electronic Engineering,

Nanyang Technological University, Singapore 639798, Singapore (e-mail:epnsugan@ntu.edu.sg).

Publisher Item Identifier S 1045-9227(02)04420-X.

et al. [13] computed the correlation between query sketch anddatabase edge images. They accounted for small shifts anddistortions by shifting the local correlation between the twoblocks of query sketch and database edge images. Jainet al.[11] constructed global 72-bin shape histograms using edgedirections. They also made use of invariant moments. Theshape similarity is performed by computing a weighted sumof the Euclidean distances. Another global shape matchingapproach was to employ deformable models such as the modaldeformation [14] method. In these approaches, user sketchesare aligned with the edge contour shapes in the database imagesusing some energy minimization techniques. Mokhtarianet al.[15] used the curvature scale space (CSS) method to representtwo-dimensional (2-D) shapes at different resolutions. Maximaof the CSS image are used to represent the shape. The matchingscheme was made to retrieve shapes invariant of translation,rotation, and scale.

Another common approach to shape-based indexing andretrieval is to use segmented boundary curves instead of theedge pixels or the complete closed curves. Petrakiset al.[16] approximated shapes into a sequence of concave andconvex segments and then a dynamic programming-basedshape matching scheme was employed to establish the corre-spondences between curve segments over different resolutions.They made use of an R-tree to perform the indexing in a lowerdimensional space. Berrettiet al. [6] proposed a shape retrievalscheme for generic shapes using a metric tree based indexingscheme. They also decomposed the shapes according to theshapes’ protrusions and organized the token attributes intoan M-tree to perform the shape similarity computation andretrieval.

The trademark image databases have been commonly used totest image retrieval and in particular several shape retrieval sys-tems. Kato [17], in his system, normalized the trademark imagesto an 8 8 pixel grid and computed shape features from the re-sulting pixel frequency distributions to be used for retrieval. Wuet al. [9] developed a system for trademark archiving and re-trieval (STAR) making use of text and images. In their retrievalsystem, they use moment invariants and Fourier descriptors.Eakinset al. [7] also investigated the problem of shape-basedtrademark retrieval. They use regions boundaries extracted frombinary images and approximated by straight lines and circulararc segments. These primitive boundary descriptors are groupedinto families to obtain various global shape features. They con-ducted experiments on a collection of over 10 000 images fromthe UK Trade Mark Registry.

In this paper, we employ the SOM to organize structuralshapes in a topographical manner for efficient shape retrieval.In the past, the SOM has been applied to solve several complexproblems including vector quantization, pattern recognition,

1045-9227/02$17.00 © 2002 IEEE

836 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 13, NO. 4, JULY 2002

organization of documents, and image retrieval [18], [19]. Theconcept of mapping structural shapes in a topology conservingmanner is novel. The structural information contained ingeometrical shape is extracted using the pairwise relational at-tribute vectors. These vectors are quantised using an SOM [20],as the SOMs offer a number of advantages such as the abilityto quantize adaptively depending on the dynamic ranges of theattributes and the ability to deal with the curse of dimensionalityin the histograms-based methods more efficiently. For example,Huet et al. [8] define a number of relational attributes but usethe best two attributes in their histogram based line patternsretrieval system. If they were to use all five or so attributes,then the line patterns retrieval will become a five-dimensional(5-D) histogram matching problem. Further, if the problem [8]did not require scale invariance, then quantization of absolutelengths and distances would become a difficult problem. Allthese difficulties can be resolved satisfactorily by employingthe SOM to perform the quantization. As the SOM preservesthe topological relationships in the data (i.e., similar relationalattribute vectors are mapped to neighboring SOM units), wecan combine neighboring elements in the histogram, if wedesire to perform a coarse-to-fine search.

Using this trained quantization SOM referred to as ,a global histogram of relational attribute vectors is generatedfor every structural shape. These histograms are treated asinput vectors to another SOM referred to as . As theseglobal histograms capture the shape properties of the objects,the trained using these histograms naturally generatesa topology conserving mapping for the structural shapes. Thisstructural topology preserving maps can be made invariant tochosen transformations such as the similarity or affine transfor-mations by choosing relational attribute vector appropriately.Although we employ a single layer SOM, it is possible toemploy a tree-structured SOM [21], [22] to perform the searchefficiently to identify the winner neuron or to have structureadaptive models to facilitate flexible inclusion or removal ofshapes [23], [24].

Efficient organization and indexing of model objects areimportant steps in the field of model-based structural objectrecognition [25]–[27] too. Specifically, in the graph-basedobject recognition [28], it is critically important to effectivelyorganize and index into the databases in order to confine thegraph matching operation to some selected models, as graphmatching is a hard combinatorial optimization problem. For in-stance, the handwritten Chinese character recognition problem[29] may also be formulated as a line patterns matchingproblem. As there are more than 10 000 distinct Chinesecharacters, organizing them using the SOM would enable usto retrieve a limited number of potential model candidates forfurther rigorous matching.

Thispaper is organized as follows. In Section II,we present therelational attributes used to characterize a pair of line segments.In Section III, the SOM algorithm as well as the applications ofthe SOM algorithm for generating one-dimensional histogramsand organization and indexation of databases are described.Experimental results are presented in Section IV; and the paperis concluded in Section V.

Fig. 1. An illustration on the computation of directed relational attributes.

II. RELATIONAL ATTRIBUTE VECTORS

We have chosen to represent the shapes by line segments.Depending on the requirements, the relational attributes betweena pair of line segments may be chosen to possess variousinvariance properties such as invariance to translation, rotation,scale, and more general affine transformations [8], [30]. In thisstudy, we consider two cases namely, invariance to translation-rotation and invariance to translation, rotation, and scale. Priorto computing the attributes, the intersection point between thetwo lines are computed as shown by “” in Fig. 1. The endpoint of the first line also known as the reference line closer tothe intersection point is labeled as “”. The other end point ofthe first line is labeled as “.” Likewise the end points of thesecond line are also labeled as “” and “ ” as shown in Fig. 1.Therefore, given a pair of line segments, all four end points canbe unambiguously labeled. In the first set of experiments, thefollowing seventranslation and rotation invariantrelationalattributes are used: 1) . Theangle returned is between zero and. However, if we identifythe rotation from to as clockwise or counter-clockwiseby evaluating the vector product between vectorsand ,then we can compute the angle attribute betweento inorder to improve the discrimination quality of this attribute.2) Length of the reference line . 3) Length of the secondline . 4) Distance . 5) Distance . 6) Distance . 7)Distance .

In the second set of experiments the following fivetransla-tion, rotation, and scale invariantrelational attributes are used:1) . The angle computed is be-tween to . 2) Relative position ratio: .3) Line length ratio: . 4) End pointratio: . 5) Cross end point ratio:

. Hence, by choosing the relationalattributes appropriately, we can ensure that the topological map-ping possesses the required invariance properties.

As there are thousands of lines in the trademark images, therewould be a huge number of line pairs. Further, if the query linepatterns are corrupted by noise, then making use of every linepairs may not be beneficial to the performance of the system.In such a situation, local neighborhood graphs with the neigh-borhood degree of 6 are known to yield the best performance[8] for translation, rotation and scale invariant retrieval. In thisstudy too, the pairwise relational vectors are computed up tosix nearest neighbor line segments of every line segment in

SUGANTHAN: SHAPE INDEXING USING SELF-ORGANIZING MAPS 837

TABLE ITHE SOM ALGORITHM

the trademark model base. Every pair of line segments yieldsa seven-dimensional (7-D) in the translation and rotation invari-ance case or a 5-D vector in the translation, rotation, and scaleinvariance case. These vectors are used to train two SOMs, onefor the 5-D and the other for the 7-D vectors, as explained inSection III to perform an adaptive quantization.

III. QUANTIZATION OF RELATIONAL VECTORS

In this section, first the SOM algorithm [20], [25] is brieflyexplained. Then the procedure for generating a 1-D relationalhistogram for every shape using the trained quantization map

and developing the for topologically mapping theshapes are described.

A. Self-Organizing Maps

In this application, it is desirable to have an equiprobablemap. In other words, it is desirable to have each neuron to be thewinner with the probability where is the total numberof nodes in the SOM. Although the usage of topological neigh-borhoods attempts to provide a uniform utilization of all nodes,it does not completely resolve the problem. There are severalapproaches proposed in the literature [20]. Three of theseapproaches, namelyconvex combination, competitive learningwith conscienceand competitive learning with attention, are

reviewed and evaluated recently by Bebiset al.[25]. Accordingto their findings, thecompetitive learning with conscienceappears to yield the best performance. In our experiments, wemade use of the same approach to train SOMand SOM.

The SOM algorithm withconscienceis summarized in Table I[20], [25]. The dimensions of the input vectors are five and sevenfor SOM . The number of output neurons will be identical to thenumber of bins that we wish to have in the 1-D histogram.

B. Shape Histograms and Indexing

After completing the training of the SOM, we can extractthe 1-D histogram by performing the steps outlined in Table II.At the completion of executing the steps in Table II, we have a1-D histogram for every shape in the database. These histogramsare treated as the input vectors to construct the SOMusing thesame self-organizing map algorithm in Table I. Hence, the inputfeature vectors’ dimension of SOMis identical to the numberof nodes in SOM. Having trained the SOM, we associate theshapes with the neurons. Every shape is associated with threebest matching neurons. With this, the organization phase is com-plete. During the application phase, we perform the steps out-lined in Table III to extract potentially similar model candidatesto the query shape.

In order to identify the best matching unit in step 3 inTable III, the histogram intersection similarity measure isused. It was shown that the histogram similarity measure [4] is

838 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 13, NO. 4, JULY 2002

TABLE IICONSTRUCTION OFPAIRWISE RELATIONAL VECTORHISTOGRAMS

TABLE IIIINDEXING DATABASES USING SOM

superior when there are multiple shapes in the query or only apartial shape is presented in the query.

where , , and are the histogram intersection similarity,number of bins in the histogram (or the number of nodes in

), histogram of the query shape, and an arbitrary featurevector of , respectively.

IV. EXPERIMENTAL RESULTS

Experiments were conducted using a part of the trademarkdatabase.1 The number of line segments in each trademarkobject may vary approximately between 100 and 2000. Weconducted two experiments using the two sets of relationalattribute vectors defined in Section II. In the translation androtation invariant experiments (EXP1), 330 objects are usedand they are scaled by 2 and 0.5 to generate scaled versionsof the objects to yield a total of 990 objects with scalings of0.5, 1, and 2. As the attributes are scale sensitive, indeed thereare 990 distinct objects. In the other experiment (EXP2), all

1About 990 trademark objects approximated by line segments were kindlygiven by Dr. Huet [8].

Fig. 2. Some trademark objects extracted for the perfect query object in thetop left corner of the figure.

990 objects are used with the 5-D translation, rotation, andscale invariant attributes. In our experiments, the number ofneurons in the is 1600. The has 225 neurons.Although it is hard to present the complete 1515 mappinggenerated by the , some typical retrieval results arepresented to illustrate the retrieval performance as well as thetopology preserving nature of the mapping. Fig. 2 shows someof the objects retrieved in EXP1 when “CNBC” is used as the

SUGANTHAN: SHAPE INDEXING USING SELF-ORGANIZING MAPS 839

query object. The “CNBC” object was also retrieved but notshown in the figure. As naturally expected, the system wasable to always retrieve the original shape when noiseless queryobjects were presented to the system in both experiments.

Further experiments were conducted to study the perfor-mance of the system when the query object is corrupted bynoise. In these experiments, some lines in the prototype objectare removed and the end points of some of the remaininglines are perturbed randomly using a uniform distribution. Theresults are summarized in Table IV. From the table, it can beseen that both systems perform acceptably up to about 30%percentage of missing lines. Figs. 3 and 4 show some retrievedobjects in EXP2 when the query objects with a fraction ofmissing lines are presented to the system. The query image isshown at the top left corner and the clean version of the objectis shown in the second column of the first row. As we use onlysix nearest neighbor graphs in both experiments, the conceptof the structural similarity is enforced only locally. Hence,some of the objects retrieved may not obviously resemble thequery object. If we make use of a much larger neighborhoodgraph, the structural similarity would be more apparent, but thecomputational cost would increase substantially. From theseexperiments, it can be concluded that the 5-D rotation, scale,and translation invariant attributes are more robust than the 7-Dtranslation and rotation invariant attributes.

The reasons for the rapid deterioration in the performancein EXP1 is due to the usage of six nearest neighbor graph andscaled versions of the same objects. As line segments are grad-ually removed from the smaller scaled objects, the computedhistogram resembles a larger scaled version of the same object.In order to improve the performance in EXP1, a larger nearestneighbor graph should be used. However, this will significantlyincrease the computation cost. Hence, the translation, scale androtation invariant approach with a limited neighborhood graphwould be more suitable in these situations.

On average the extracted about 40 shapes for furtherrigorous matching. This is about 4% of the total shapes in thedatabase. The final retrieval can be performed using a moreaccurate matching method [28] on the 40 or so potentialcandidates. From the experimental results in Figs. 2–4, it is clearthat the was able to retrieve the similar shapes using thehistogram intersection similarity measure. From the retrievedobjects, it is also clear that similar shapes are mapped to thesame neuron or the neighboring neurons. Our inspection of theneurons at other positions also showed that the topologicallysimilar shapes are mapped to the same neuron or neighboringneurons. The results clearly show the ability of the proposedapproach to map the shapes in a topology conserving mannerand to retrieve structurally similar shapes for a given partiallydistorted query shape.

V. CONCLUSION

In this paper, we proposed a novel topology preservingmapping scheme for geometric structural objects using theSOM. We made use of 7-D and 5-D relational vectors fortranslation-rotation invariance and translation, rotation, andscale invariance mappings, respectively. These relational vectors

TABLE IVSHAPE RETRIEVAL ERRORRATE RESULTS

Fig. 3. Some trademark objects extracted for the query object with about 30%missing lines as shown in the top left corner of the figure.

Fig. 4. Some trademark objects extracted for the query object with about 40%missing lines as shown in the top left corner of the figure.

are quantized using two s and subsequently convertedinto a 1-D histogram for every structural shape. These 1-Dhistograms are then used to train the s. Thecan be used to retrieve structural objects from a database fora given query shape. We presented experimental results todemonstrate the reliable performance of the proposed system.The can also be used to organize structural objectsin a model-based object recognition system and when a testscene is given, models which are probably present in the testscene can be retrieved efficiently.

The proposed approach offers a number of advantages suchas the ability to make use of several relational attributes (asopposed to the limited number of attributes that can be usedin the histogram-based relational indexing methods [8], [30]),the ability to perform a dynamic quantization, the flexibility inincluding and removing model objects and database images,possibility for efficient implementation such as the tree-struc-tured SOM [20], [21], [22] and the ability to handle otherattributes like color and texture in a homogeneous mannerby the SOM [18], [19]. We have also demonstrated that theproposed approach is capable of generating the topologicalmapping with the desired invariance properties.

840 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 13, NO. 4, JULY 2002

REFERENCES

[1] L. Guan, S.-Y. Kung, and J. Larsen,Multimedia Image and Video Pro-cessing, ser. Image Processing. Boca Raton, FL: CRC, 2000.

[2] A. W. M. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain,“Content-based image retrieval at the end of early years,”IEEE Trans.Pattern Anal. Machine Intell., vol. 22, pp. 1349–1380, Dec..

[3] R. C. Veltkamp and M. Hagendoorn, “State-of-the-art in shapesearching,” in Multimedia Search: State of the Art. New York:Springer-Verlag, 2000.

[4] M. Swain and D. Ballard, “Color indexing,”Int. J. Comput. Vision, vol.7, no. 1, pp. 11–32, 1991.

[5] B. S. Manjunath and W. Y. Ma, “Texture features for browsing and re-trieval of image data,”IEEE Trans. Pattern Anal. Machine Intell., vol.18, pp. 837–841, Aug. 1996.

[6] S. Berretti, A. D. Bimbo, and P. Pala, “Retrieval by shape similarity withperceptual distance and effective indexing,”IEEE Trans. Multimedia,vol. 2, pp. 225–239, 2000.

[7] J. P. Eakins, J. M. Boardman, and M. E. Graham, “Similarity retrievalof trademark images,”IEEE Multimedia, vol. 2, pp. 53–63, April–June1998.

[8] B. Huet and E. R. Hancock, “Line pattern retrieval using relationalhistograms,”IEEE Trans. Pattern Anal. Machine Intell., vol. 21, pp.1363–1369, Dec. 1999.

[9] J. K. Wu, C. P. Lam, and B. Mehtre, “Content-based retrieval fortrademark registration,”Multimedia Tools Applicat., vol. 3, no. 3, pp.245–267, 1996.

[10] S.-K. Chang, Q. Y. Shi, and C. W. Yan, “Iconic indexing by 2D strings,”IEEE Trans. Pattern Anal. Machine Intell., vol. 9, pp. 413–428, 1987.

[11] A. K. Jain and V. Vailaya, “Image retrieval using color and shape,”Pat-tern Recognition, vol. 29, no. 8, pp. 1233–1244, 1996.

[12] W. Niblacket al., “The QBIC project: Querying images by content usingcolor, texture and shape,”Image and Visual Storage and Retrieval, pp.173–187, 1993.

[13] K. Hirata and T. Kato, “Query by visual examples: Content-based imageretrieval,” inAdvances in Database Technology, EDBT-92.

[14] A. P. Pentland, R. W. Pickard, and S. Scarloff, “Photobook: Content-based manipulation of image databases,”Int. J. Comput. Vision, vol. 18,no. 3, pp. 233–254, 1996.

[15] F. Mokhtarian, “Silhouette-based isolated object recognition throughcurvature scale space,”IEEE Trans. Pattern Anal. Machine Intell., vol.17, pp. 539–544, 1995.

[16] E. Petrakis and E. Milios, “Shape retrieval based on dynamic program-ming,” IEEE Trans. Image Processing, vol. 9, pp. 141–147, Jan. 2000.

[17] T. Kato, “Database architecture for content-based image retrieval,” inProc. SPIE 2185, Image Storage Retrieval Syst., A. A. Jambardino andW. R. Niblack, Eds., 1992, pp. 112–123.

[18] E. Oja, J. Laaksonen, M. Koskela, and S. Brandt, “Self-organizing mapsfor content-based image database retrieval,” inKohonen Maps, E. Ojaand S. Kaski, Eds. Amsterdam, The Netherlands: Elsevier, 1999.

[19] H. Zhang and D. Zhong, “A scheme for visual feature based image in-dexing,” in Proc. SPIE Conf. Storage Retrieval Image Video Databases,San Jose, CA, Feb. 1995.

[20] T. Kohonen, Self-Organizing Maps, 2nd ed. New York:Springer-Verlag, 1997.

[21] J. Laaksonen, M. Koskela, and E. Oja, “Application of tree structuredself-organizing maps in content-based image retrieval,” inProc.ICANN’99, pp. 174–179.

[22] P. Koikkalainen and E. Oja, “Self-organizing hierarchical feature maps,”in Proc. IJCNN-90, Int. Joint Conf. Neural Networks, vol. II, 1990, pp.279–285.

[23] B. Fritzke, “A growing neural gas network learns topologies,” inAd-vances in Neural Information Processing Systems, G. Teasauro, D. S.Touretzky, and T. K. Keen, Eds. Cambridge, MA: MIT Press, 1995,pp. 625–632.

[24] P. N. Suganthan, “Hierarchical overlapped SOM’s for pattern classifica-tion,” IEEE Trans. Neural Networks, vol. 10, Jan. 1999.

[25] G. Bebis, M. Geoorgiopoulos, and N. d. da V Lobo, “Using self-orga-nizing maps to learn geometric hash functions for model-based objectrecognition,”IEEE Trans. Neural Networks, vol. 9, pp. 560–570, May1998.

[26] L. G. Shapiro and R. M. Haralick, “Organization of relational modelsfor scene analysis,”IEEE Trans. Pattern Anal. Machine Intell., vol. 4,pp. 595–602, Nov. 1982.

[27] L. Xu and I. King, “A PCA approach for fast retrieval of structural pat-terns in attributed graphs,”IEEE Trans. Syst., Man. Cybern. B, vol. 31,pp. 812–817, Oct. 2001.

[28] P. N. Suganthan, E. K. Teoh, and D. P. Mital, “Hopfield network withconstraint parameter adaptation for overlapped shape recognition,”IEEE Trans. Neural Networks, vol. 10, Mar. 1999.

[29] P. N. Suganthan and H. Yan, “Recognition of handprinted chinese char-acters by constrained graph matching,”Image and Vision Comput., vol.16, no. 3, Mar. 1998.

[30] N. A. Thacker, P. A. Riocreux, and R. B. Yates, “Assessing the com-pleteness of properties of pairwise geometric histograms,”Image VisionComput., vol. 13, no. 5, pp. 423–429, 1995.

P. N. Suganthan(S’91–M’92–SM’00) received theB.A. degree, Postgraduate Certificate, and M.A.degree in electrical and information engineeringfrom the University of Cambridge, Cambridge, U.K.,in 1990, 1992, and 1994, respectively. He receivedthe Ph.D. degree from the School of Electricaland Electronic Engineering, Nanyang TechnologicalUniversity, Singapore, in 1996.

From 1995 to 1996, he was a PredoctoralResearch Assistant in the Department of ElectricalEngineering, University of Sydney, Sydney, Aus-tralia. From 1996 to 1999, he was a Lecturer in the

Department of Computer Science and Electrical Engineering, University ofQueensland, Australia. Since July 1999, he has been an Assistant Professorin the School of Electrical and Electronic Engineering, Nanyang Technolog-ical University, Singapore. His research interests include neural networks,pattern recognition, image/video retrieval, genetic algorithms, support vectormachines, and bioinformatics.

Dr. Suganthan is an Associate Editor of thePattern Recognition Journalandan Associate Member of the Institution of Electrical Engineers.