A region-dependent image matching methodfor image and video annotation

Golnaz Abdollahian, Murat Birinci †, Fernando Diaz-de-Maria ‡, Moncef Gabbouj †, Edward J. Delp∗ Video and Image Processing Laboratory (VIPER ) † Department of Signal Processing ‡ Department of Signal Theory

School of Electrical and Computer Engineering Tampere University of Technology and CommunicationsPurdue University Tampere, Finland Universidad Carlos III

West Lafayette, Indiana, USA Leganes (Madrid),Spain

Abstract

In this paper we propose an image matching approachthat selects the method of matching for each region in theimage based on the region properties. This method canbe used to find images similar to a query image from adatabase, which is useful for automatic image and video an-notation. In this approach, each image is first divided intolarge homogeneous areas, identified as “texture areas”, andnon-texture areas. Local descriptors are then used to matchthe keypoints in the non-texture areas, while texture regionsare matched based on low level visual features. Experi-mental results prove that while exclusion of texture areasfrom local descriptor matching increases the efficiency ofthe whole process, utilization of appropriate measures fordifferent regions can also increase the overall performance.

1 Introduction

Recently, a number of methods have been proposed toexploit Web-based tags for image and video annotation[11, 1]. Unlike the more traditional annotation approaches[10], these techniques do not require training and the vocab-ulary used for annotation is diverse. These methods exploitthe user-generated information from online image databasessuch as Flickr. The photos on these databases are often as-sociated with various tags including information about thecontent, time, date, location and camera used. The Web-based annotation methods process the tags collected fromthe most similar images to the query image or video framein order to find the relevant tags for annotation.

Thus, a major step in these annotation methods is to de-termine the similarity between images or “image matching.”Image matching in this case is challenging due to the large

∗The second and third authors performed the work while at Purdue Uni-versity

variation in the images taken from the same scene. Theglobal and local characteristics of a scene can change dras-tically depending on the viewpoint, lighting condition, andresolution of the camera.

During the last decade, most research on image match-ing and retrieval has been inspired by the seminal paper bySchmid and Mohr [8], and is based on local descriptors ofinterest points [6]. Compared to global approaches, thesemethods are more robust to clutter or partial occlusions;nevertheless, many of the identified interest points are un-reliable or their descriptors do not match. Moreover, thecomputational cost due to the one-to-one matching is sig-nificant for large scale images. A number of methods havebeen proposed to deal with these problems to some extent[3, 9]. In 2006, a global approach was proposed in parallelthat aims to describe the gist of a scene [7]. However, as theauthors mention, their approach is not an alternative to localimage analysis, but would be a complement. In this workwe propose combining low level features, typically used tomatch the general attributes of an image, with local descrip-tors to determine visual similarity between images.

Figure 1. An example where the majority ofKeypoints found by SIFT are located on thetexture area (tree branches).

Large homogeneous texture areas such as grass, trees,bricks, sky or street asphalt are common in outdoor imagesand videos. These regions can affect the global visual sim-

11111111111111111111111111111111111111111111111111111111111 CBMI’20111111111111111111111111111111111111111111111111111111111111 CBMI’2011

978-1-61284-433-6/11/$26.00 ©2011 IEEE 121 CBMI’2011

1111111111111111111111111111111111111111111111111111111111 CBMI’2011

ilarity between the images significantly. In addition, sometexture areas such as tree branches or bricks can decreasethe performance of the keypoint matching methods such asSIFT [6]. These texture areas result in the detection of alarge number of keypoints that are not useful for the purposeof object matching. However, both detection and matchingof those points increase computational complexity signifi-cantly. For example, in Figure 1 SIFT detects 6735 key-points, but more than 60% of these keypoints are located onthe tree branches, which are of no significant value in theprocess of object matching. Nonetheless, these keypointsconsiderably increase the computational complexity.

In this paper we identify the large homogeneous areasin the images as “texture areas” (Section 2.1), and ex-clude them from the keypoint detection and matching pro-cess (Section 2.2). The texture areas are matched betweenimages based on their overall color and texture proper-ties (Section 2.3). The matching scores from different ar-eas of the images (texture and non-texture areas) are thencombined to obtain an overall matching score for an im-age(Section 2.4). Figure 2 shows an overview of our ap-proach.

Figure 2. A block diagram of the proposed ap-proach.

2 A region-dependent image matchingmethod

2.1 Texture area detection

In order to identify the texture regions, a method pro-posed in [12], which is a combination of quadtree segmen-tation and region growing algorithm, is used. In this methodthe image is divided into non-overlapping macroblocks. Inorder to determine homogeneity, each macroblock is fur-ther decomposed into four non-overlapping regions. Themacroblock is considered to be homogeneous if its four re-gions have pairwise similar statistical properties. A regiongrowing algorithm is then used to merge the similar homo-geneous macroblocks. Each homogeneous area that covers

at least five percent of the image area is considered to be atexture region. Examples of the identified texture regionsusing this method are shown in Figure 3.

Figure 3. Left: the results of the split andmerge algorithm for identifying the homoge-neous regions (the white lines show the re-gion boundaries). Right: the identified tex-ture masks. Each shade of grey is a separatetexture region. The black areas indicate non-texture regions.

2.2 Non-texture area matching: local de-scriptors

Non-texture regions usually contain important detailsthat are useful for recognizing significant objects and land-marks in a scene. Therefore, we use methods that are basedon local descriptors to identify the keypoints and matchthem between images. Among the several local descriptorsthat have been proposed in the literature, Scale InvariantFeature Transform (SIFT) [6] is one of the most success-ful descriptors and has been widely used for many applica-tions. A number of variations of SIFT has been proposedto improve the efficiency performance of this method (e.g.,SURF [2]).

One disadvantage of SIFT is the heavy computationalcomplexity that it adds to the system. For high resolutionoutdoor images, SIFT can find thousands of keypoints. Inmany cases only a small percentage of the obtained key-points are actually useful for the purpose of matching. Cer-tain texture areas such as trees, grass and bricks, containkeypoint-like patterns that can result in detection of a highnumber of irrelevant keypoints and decrease the efficiencyof matching. Therefore, we avoid the calculation of key-points in the texture areas using the texture masks obtained

1111111111111111111111111111111111111111111111111111111112 CBMI’20111111111111111111111111111111111111111111111111111111111112 CBMI’2011

978-1-61284-433-6/11/$26.00 ©2011 IEEE 122 CBMI’2011

1111111111111111111111111111111111111111111111111111111112 CBMI’2011

in the previous section. In the SIFT method, after findingthe local extrema in the scale space, the points that are lo-cated inside a texture mask are discarded from further cal-culations, namely descriptor calculation and matching. Fig-ure 4 illustrates the obtained SIFT keypoints from an imagewith and without using a texture mask. As we will showin Section 3, the exclusion of texture areas can significantlydecrease the algorithm run time for obtaining local descrip-tors as well as the matching process.

(a) (b)

Figure 4. (a) Keypoints detected by SIFT fromthe original image (10,577 keypoints), (b)Keypoints detected by SIFT using the texturemask (3,723 keypoints).

2.3 Texture area matching: low level fea-tures

In order to find similar texture areas between differentimages, each texture area is represented by a color andtexture feature vector. The color feature vector is a 256-bin histogram of the chroma components, Cb and Cr ina Y CbCr color space. Haralick’s texture features [4], de-rived from the graylevel co-occurrence matrices (GLCM),are used to form the texture feature vector. These featuresinclude angular second moment, dissimilarity, correlation,entropy and contrast, derived from co-occurrence matricesusing 12 different distance vectors. This will result in a 60dimensional feature vector.

The Bhattacharyya distance is used to measure the dis-tance between the color histograms. The Haralick’s texturefeatures are first normalized to their standard deviation andthen compared using the Euclidean distance. The standarddeviation for each feature is obtained from the images inthe database. The two distances (color and texture) are thenaveraged to obtain the total distance, Di,j , between textureregions Ti and Tj . The larger the matched area, the more itaffects the visual similarity between the two images. Thus,between the two regions being matched, we incorporate thearea of the smaller of the two in the matching score,

s (Ti, Tj) =min(Ai, Aj)

Di,j + ε(1)

where s (Ti, Tj) is the matching score between regionsTi and Tj , and Ai and Aj are the corresponding areas ofthose regions. The parameter ε is added to the denominatorto address the scenarios where the feature vectors of the tworegions are exactly the same.

To obtain the overall texture matching score,Stexture(n,m), between images n and m, each tex-ture region, Ti, in image n is matched to a region in imagem with highest matching score,

sm (Ti) = maxTj∈Rm

s(Ti, Tj). (2)

Then,

Stexture(n,m) =∑

Ti∈Rn

sm (Ti) (3)

where Rm and Rn are the set of texture regions in imagem and n, respectivly. Figure 5 shows the results of texturematching for three pairs of images.

2.4 Obtaining the overall image matchingscore

Matches from either the local descriptors or the textureareas can have higher priority depending on the application.For example, in the case of Web-based annotation, if the cat-egories of the tags are known (name of unique objects suchas “Eiffel Tower” versus a common name such as “ocean”or “flowers”), the scores can be weighed differently for eachtag. Here we consider a tag-independent approach to com-bine the scores from the local descriptor matching and tex-ture matching.

The local descriptor matching score from image m to n

is obtained as

Slocal(m,n) =KM (m,n)

K(m)(4)

where KM (m,n) is the number of matched keypoints fromimage m to image n and K(m) is the total number of key-points in image m. Both Stexture(m,n) and Slocal(m,n)are normalized so that for each image m, the average ofthese scores over n is zero and the variance is 1. The com-bined similarity score is then defined as

SC(m,n) = Stexture(m,n) + Slocal(m,n) (5)

where Stexture and Slocal are the normalized scores. Foreach query image, images with the highest combined simi-larity scores are retrieved and the tags associated with themare used for annotation. More details are given in Section 3.

1111111111111111111111111111111111111111111111111111111113 CBMI’20111111111111111111111111111111111111111111111111111111111113 CBMI’2011

978-1-61284-433-6/11/$26.00 ©2011 IEEE 123 CBMI’2011

1111111111111111111111111111111111111111111111111111111113 CBMI’2011

Figure 5. Examples of the proposed texture area matching. In the bottom row, the texture areas areoverlaid on the original images by increasing the intensity in those regions. The red lines indicatematches between the texture areas.

3 Experimental results

To compare our methods with the ones using only localdescriptors, two image sets are used. The first set is fromthe database by Li [5] with images of size 384 × 256 or256×384 pixels, that are manually annotated. Most imagesin this database contain large texture areas. We selected 85images that contained at least one specific object or clearregion boundaries to be suitable for SIFT matching. Thereare multiple shots from each scene with different cameraviews in this database. For each scene, we included all thesimilar images.

We have also collected a set of 45 images of size 768 ×

1028 pixels from Purdue Campus and the neighboring ar-eas. These images contain several static objects in the back-ground, such as buildings, monuments, towers and statues,and also a variety of different types of textures, e.g. grass,sky, tree branches, tiles and bricks.

3.1 Time analysis

Texture masks prevent the local descriptor algorithmfrom calculating several unnecessary descriptors andmatching them, resulting in higher efficiency of the match-ing process. The total run time for the extraction of SIFTdescriptors and matching them across the set of 85 imagesfrom Li’s database was 1276.65s without the texture masks.The total run time for extracting and matching the SIFT de-scriptors from the non-texture regions, and extracting andmatching the texture regions was 777.5s. Thus, our methodresults in 40% reduction in the run time by using the texturemasks. The execution times were measured on a PC with a2.66GHz Core2Duo processor and 1.96GB of RAM.

Table 1 shows the average number of keypoints obtainedby SIFT and SURF, and the average run time for obtainingthe keypoint descriptors, with and without using the tex-ture masks for the Purdue Campus database. This resultshows that the average run time was reduced by 28.4% for

SIFT, and by 29% for SURF when using the texture masks.The number of identified keypoints was 47.5% decreasedfor SIFT, and 38.9% decreased for SURF, after using themasks.

Without Texture Mask With Texture MaskNK σNk t (ms) σt NK σNk t (ms) σt

SIFT 3437.9 1681.3 2791.6 756.2 1805.6 903.3 1998.5 443SURF 5159.8 2327.8 436.4 179.8 3153.7 1593.5 309.90 62.0

Table 1. The statistics of the SIFT and SURFkeypoint descriptors with and without usingthe texture masks for the Purdue Campusdatabase. In this table NK is the averagenumber of keypoints, t is the average runtime, and σNk and σt are the correspondingstandard deviations.

The reduced number of keypoints also accelerated theprocess of matching since less number of keypoints neededto be compared. Table 2 shows the average run time ofthe matching process using SIFT and SURF for the Pur-due Campus database, with and without the texture masks.Our results showed that the matching time was reduced by63.6% for SIFT and by 52% for SURF when using the tex-ture masks.

Without Texture Mask With Texture MaskAvg # Matched Points t (ms) SEt Avg # Matched Points t (ms) SEt

SIFT 101.3 2128.9 168.8 83.1 775.8 87.1SURF 64.0 345.5 12.4 43.4 165.5 14.4

Table 2. The statistics for the keypoint match-ing using SIFT and SURF, with and with-out texture masks for the Purdue Campusdatabase, where SEt is the standard error ofthe average time.

1111111111111111111111111111111111111111111111111111111114 CBMI’20111111111111111111111111111111111111111111111111111111111114 CBMI’2011

978-1-61284-433-6/11/$26.00 ©2011 IEEE 124 CBMI’2011

1111111111111111111111111111111111111111111111111111111114 CBMI’2011

3.2 Annotation performance

To evaluate the annotation performance, images fromLi’s database [5] were used. Images in the database aremanually annotated with 1 to 4 tags, with the average of2.6 tags per image. The annotation performance compari-son was done between three different approaches: 1) SIFTOnly, where the SIFT descriptors of each image are ex-tracted from the entire image area. The descriptors arethen matched between the images in the database. For eachimage, images with the highest number of matched key-points are retrieved; 2) Texture Only, where the matchingscores between the texture areas are obtained as describedin Section 2.3. These scores are used for the retrieval; 3)Combined Similarity Score, where the SIFT descriptors areextracted only from the non-texture areas. The combinedmatching scores are then obtained using Equation 5.

For every image in the database, the L most similar im-ages are retrieved using each of the three approaches men-tioned above (here L = 4). The tags from these L imagesare extracted and ranked based on their associated scores.The score of a tag is defined as the sum of the similarityscores of the retrieved images that contain that tag. Our sys-tem returns 4 tags with the highest scores as the annotationoutput.

To evaluate the results, precision, recall and F measureare obtained as follows. Each tag in a query image is con-sidered to be missed if it does not appear in the returnedtags. Recall is defined as 1 − #Missed Tags

Total#Tags. For each im-

age that is manually annotated with N tags, each tag in thefirst N output tags is considered to be correct if it is in-cluded in the manual annotation. Precision is then definedas #Correct Tags

Total#Tags. As Table 3 shows, the annotation results

are improved when the SIFT matching results from the non-texture areas are combined with texture area matching re-sults. Although this improvement is small but the overallprocessing time has significantly decreased.

Precision % Recall % F %SIFT Only 80.72 85.20 82.90

Texture Only 64.57 69.06 66.74Combined

Similarity Score 82.06 86.55 84.25

Table 3. Evaluation of the annotation results.

3.3 Image matching

In addition to images that have high number of keypointsmatches, our method is able to retrieve images that are vi-sually similar but contain only a few reliable keypoints. Forexample, in Figure 6 SIFT performed poorly in matching

two similar shots of the same scene. Even though, SIFT lo-cated 6735 keypoints in one image and 6388 keypoints inthe other, only 13 matches were found. However, based ontexture matching, 44% of the image area in the left imagematched the image on the right (Figure 6). Thus, the pro-posed method is able to identify the image on the right as aclose match.

Figure 6. Left: matching based on originalSIFT method. Right: texture area matching.

4 Conclusions

In this paper we proposed a region-dependent methodfor image matching that can be used for automatic imageand video annotation, especially for approaches that ex-ploit user-tagged image databases for this purpose. In ourmethod the texture areas are identified and matched usinglow level visual features. The non-texture areas are handledby local descriptor matching. This approach has several ad-vantages: 1) the exclusion of texture areas from the key-point matching accelerates this process. 2) It prevents mis-matches that happen in certain texture areas with keypoint-like patterns (e.g., trees and bricks). 3) The combinationof texture and local descriptor matching can help recog-nize both region-based tags (e.g., “ocean” and “sunset”) andlandmarks that are associated with specific landscapes.

References

[1] G. Abdollahian and E. J. Delp. User generated video an-notation using geo-tagged image databases. In Proceedingsof IEEE International Conference on Multimedia and Expo,July 2009.

[2] H. Bay, T. Tuytelaars, and L. V. Gool. Surf: Speeded uprobust features. In Proceedings of European Conference onComputer Vision, page 404417, 2006.

[3] J. Cech, J. Matas, and M. Perdoch. Efficient sequential cor-respondence selection by cosegmentation. In CVPR, pages1–8, June 2008.

[4] R. Haralick, K. Shanmugan, and I. Dinstein. Textural fea-tures for image classification. IEEE Transactions on Sys-tems, Man and Cybernetics, 3(6):610–621, November 1973.

[5] J. Li. Photography image database.[6] D. Lowe. Distinctive image features from scale-invariant

keypoints. International Journal of Computer Vision,60(2):91–110, 2004.

1111111111111111111111111111111111111111111111111111111115 CBMI’20111111111111111111111111111111111111111111111111111111111115 CBMI’2011

978-1-61284-433-6/11/$26.00 ©2011 IEEE 125 CBMI’2011

1111111111111111111111111111111111111111111111111111111115 CBMI’2011

[7] A. Oliva and A. Torralba. Building the gist of a scene: Therole of global image features in recognition. Progress inBrain Research: Visual perception, 155:23–36, 2006.

[8] C. Schmid and R. Mohr. Local greyvalue invariants for im-age retrieval. IEEE Transaction on Pattern Analysis andMa-chine Intelligence, 19(5):530–535, 1997.

[9] J. Sivic and A. Zisserman. Video google: A text retrievalapproach to object matching in videos. In Proceedingsof the International Conference on Computer Vision, page14701477, October 2003.

[10] C. G. Snoek1 and M. Worring. Multimodal video indexing:A review of the state-of-the-art. Multimedia Tools and Ap-plications, 25(1):5–35, January 3005.

[11] X.-J. Wang, L. Zhang, F. Jing, and W.-Y. Ma. Annosearch:Image auto-annotation by search. IEEE Computer SocietyConference on Computer Vision and Pattern Recognition,2:1483–1490, 2006.

[12] F. Zhu, K. Ng, G. Abdollahian, and E. J. Delp. Spatialand temporal models for texture-based video coding. InProceedings of the SPIE International Conference on VideoCommunications and Image Processing (VCIP 2007), Jan-uary 2007.

1111111111111111111111111111111111111111111111111111111116 CBMI’20111111111111111111111111111111111111111111111111111111111116 CBMI’2011

978-1-61284-433-6/11/$26.00 ©2011 IEEE 126 CBMI’2011

1111111111111111111111111111111111111111111111111111111116 CBMI’2011