supporting efficient multimedia database exploration

Download Supporting efficient multimedia database exploration

Post on 15-Jul-2016




2 download

Embed Size (px)


  • The VLDB Journal (2001) 9: 312326 / Digital Object Identier (DOI) 10.1007/s007780100040

    Supporting efcient multimedia database explorationWen-Syan Li, K. Selcuk Candan, Kyoji Hirata, Yoshinori HaraC&C Research Laboratories, NEC USA, 110 Rio Robles, M/S SJ100, San Jose, CA 95134, USA;E-mail: {wen,candan,hirata,hara}

    Edited by S. Christodoulakis. Received: 9 June 1998/ Accepted: 21 July 2000Published online: 4 May 2001 c Springer-Verlag 2001

    Abstract. Due to the fuzziness of query specication andme-dia matching, multimedia retrieval is conducted by way of ex-ploration. It is essential to provide feedback so that users canvisualize query reformulation alternatives and database con-tent distribution. Since media matching is an expensive task,another issue is how to efciently support exploration so thatthe system is not overloaded by perpetual query reformulation.In this paper, we present a uniform framework to represent sta-tistical information of both semantics and visual metadata forimages in the databases.We propose the concept of query veri-cation, which evaluates queries using statistics, and providesusers with feedback, including the strictness and reformula-tion alternatives of each query condition as well as estimatednumbers of matches. With query verication, the system in-creases the efciency of the multimedia database explorationfor both users and the system. Such statistical informationis also utilized to support progressive query processing andquery relaxation.

    Key words: Multimedia database Exploration Query re-laxation Progressive processing Selectivity statistics Hu-man computer interaction

    1 Introduction

    Query processing in multimedia databases is different fromquery processing in traditional database systems. Contentsstored in traditional database systems are generally preciseand, as a result, query processing answers are deterministic.On the other hand, in both document retrieval and image re-trieval, results are based on similarity calculations. In docu-ment retrieval, documents are represented as keyword lists. Toretrieve a document, information systems compare keywordsspecied by users with the documents keyword lists. Images,

    Correspondence to: K.S. Candan, Computer Science and Engineer-ing Department, College of Engineering and Applied Sciences, Ari-zona State University, Box 875406 Tempe, AZ 85287-5406, USA;E-mail:

    This work was performed when K.S. Candan visited NEC, CCRL.

    in a similar manner, are usually represented as media features.However, image matching is carried out through comparingthese feature vectors. Comparison of these three types of queryprocessing are illustrated at the top of Fig. 1.We see that mul-timedia database query processing is truly an integration ofthese three types of query processing.

    An image consists of three types of information repre-senting its contents: visual features, structural layout (spatialrelationships), and semantics of image objects, as shown at thebottom of Fig. 1. A multimedia database query requires simi-larity measures in all these aspects. For example, a query maybe posed as retrieve images in which there is a woman to theright of an object and the object is visually similar to the pro-vided image". This query results in a list of candidate imagesranked by their aggregated scores with degrees of uncertaintybased on all above three aspects. Thus, we view multime-dia database query processing as a combination of: (1) infor-mation retrieval notions described in [1] (exploratory query-ing, inexact match, query renement); and (2) ORDBMS orOODBMS database notions (recognition of specic concepts,variety of data types).

    In most multimedia applications, supporting partial matchcapability in contrast to supporting only exact match func-tionalities in relational DBMSs is essential and desirable.There are two major reasons:1. There may not be a reasonable number of images which

    match with the user query. Figure 2 (Query) shows theconceptual representation of a query. Figures 2ad showexamples of candidate images that may match this query.The numbers next to the objects in these candidate imagesdenote the similarity values for the object-level matching.The candidate image in Fig. 2a satises object matchingconditions but its layout does not match user specication.Figure 2b and d satisfy image layout conditions but objectsdo not perfectly match the specication. Figure 2c hasstructural and object matching with low scores.Note that in Fig. 2a, the spatial constraint, and in Fig. 2c,the image similarity constraint for the lake completely fail(i.e., the match is 0.0). Such candidate images in gen-eral would not be returned by SQL/DBMS-based imageretrieval systems. Query relaxation is needed to includesuch types of partially matched images.

  • W.-S. Li et al.: Supporting efcient multimedia database exploration 313

    Fig. 1. Multimedia databases: integration of tradi-tional query processing, IR, and image matching

    2. The users may not be able to specify queries preciselyor correctly due to the so-called word mismatch prob-lem" described and studied in the eld of informationretrieval[2]. Another reason for such misspecied imagequeries is that it is difcult for users to describe a color ora shape without ambiguity.

    Due to the large volume of data, the unstructured or semi-structured nature of information, and the vagueness in the con-cept of amatch, queryingmultimedia databases should be con-ducted in an exploratory fashion guided by system feedback.In other words, image retrieval should be performed throughcomputer human interaction (CHI). We dene a CHI explo-ration cycle as a user query followed by a sequence of queryreformulations guided by the system for honing target images.In the design of a multimedia database system, along with theoptimization of query processing, we concentrate on the im-provement of the speed of the image retrieval in a whole CHIexploration cycle.

    We view media retrieval as transitions of correspondingquery result space. By reformulating a query, the result spacethat a user sees is shifted from one to another until the targetresult space is reached.We believe that themedia retrieval pro-cess can be improved through well-designed computer humaninteraction. Li et al.[4] introduced the concept of system facil-itated exploration, in which users interact with themultimediadatabase system to reformulate queries. This approach is be-

    yond browsing which is supported by most existing systems.However, some drawbacks observed are that the system canonly provide feedback after query processing and feedback islimited to semantics-based query criteria.

    In this paper, we present a new approach to media retrievalby systematically facilitating users to reformulate queries inthe scope of the SEMCOG Multimedia Database System[3]developed at NEC C&C Research Laboratories in San Jose.SEMCOG provides query verication functionalities, whichpresents users with various types of feedback regarding thequery and the database content to assist users in exploringmultimedia databasesmore efciently. The feedback includes:(1) query reformulation alternatives for both semantics- andvisual characteristics-based query criteria; (2) the estimatednumber of matched images; and (3) database content distribu-tion related to query conditions. With this information, usersare facilitated to navigate" from the current result space to-ward the target result space, rather than ad hoc trial and erroron an image-by-image basis. The facilitation allows the usersto reformulate queries interactively based on system feedbackand query analysis.

    To support such facilitation efciently, the system utilizesmultimedia content statistics. Using content statistics for re-sult estimation and query optimization has been studied fordecades in the scope of traditional databases. However, to ex-tend it to multimedia contents is not an easy task. We develop

  • 314 W.-S. Li et al.: Supporting efcient multimedia database exploration

    Fig. 2. Query image and candi-dates of partial matches

    Fig. 3. Using multimedia statistics for facilitat-ing query and exploration

    a uniform framework for representing statistics for both tex-tual and media data. SEMCOG classies media objects intoclusters based on semantics and visual characteristics. Thenumber of images in each cluster is viewed as the selectivityfor such criteria, which can be color, shape, or object position.In Fig. 3, a user issues a query image and species one of theimage objects as Pro Golfer. Based on the selectivity values ofthemedia and semantics specied in the query, SEMCOGesti-mates the number ofmatching images and provides alternativeconditions, such as related semantic terms or similar colors,for the user to reformulate the query. Based on the indexingscheme, SEMCOG also supports automated query relaxationand progressive processing to generate the top ranked resultsincrementally.

    The rest of the paper is organized as follows: we rstpresent background information about our system. In Sects.3 and 4, we describe the schemes to compute the selectivityfor semantics and visual characteristics of multimedia con-tents. In Sect. 5, we present the usage of these indices andselectivity values for: (1) estimation of matching images; (2)automated query relaxation; and (3) progressive processing.We also present experimental results to estimate the efciencyof image retrieval by clustering. In Sect. 6, we review relatedwork. In Sect. 7 we offer our concluding remarks.

    2 Object-based modeling and query languageIn this section, we give a summary of the modeling and lan-guage design in SEMCOG. We only describe the material re-lated to the work presented in this paper. Additional informa-

    tion about the system architecture, language syntax, an


View more >