integrated multimedia indexing and retrieval

Multimedia Database Management System - Chapter 8

Integrated MultimediaIndexing and Retrieval

Rachmat Wahid Saleh Insani, S.Kom

Multimedia Database Management System - Chapter 8

General Approaches Integrated Multimedia IR

• Based on low level audio, image, and video features. It is called content-based IR.

• Based on structured attributes. For examples: title, date creation, filetype, etc.

• Uses annotation. Unstructured free text or structured description.

Multimedia Database Management System - Chapter 8

Why Integrate the Approaches?

First, media do not normally appear in isolation. They usually coexist within an information unit to reinforce each other

to convey a message.

Multimedia Database Management System - Chapter 8

Why Integrate the Approaches?

Different indexing and retrieval approaches have different strengths and weaknesses and complement each other. The content-based approach captures low-level features and it cannot

capture high-level abstraction and concepts.

Multimedia Database Management System - Chapter 8

Why Integrate the Approaches?

There are different levels of difficulty in using different retrieval approaches.

Queries in text are much easier than queries by sketches.

Multimedia Database Management System - Chapter 8

Integrated IR Techniques

Individual media such as structured data, text,

audio and image appear in combinations.

Some media types are facilitate the indexing and retrieval of other more difficult media.

Multimedia Database Management System - Chapter 8

Multimedia Objects (MMOs)

MMOs Type 1 Combination of structured attributes, free text and audio.

Multimedia Database Management System - Chapter 8

Multimedia Objects (MMOs)

MMOs Type 2 Combination of structured attributes, free text, audio and image.

Multimedia Database Management System - Chapter 8

Multimedia Objects (MMOs)

MMOs Type 3 Combination of structured data, free text, audio and video.

Multimedia Database Management System - Chapter 8

Integrated Audio IR

Content based audio IR can be integrated with IR based on structured data and free text annotation. • Structured data. Structured data examples: file name,

file format, music type. The user can then search the audio based on pitch or spoken words.

• Annotation. Each audio segment is annotated with free text. Time information included in the annotation.

Multimedia Database Management System - Chapter 8

Integrated Audio IR

• Segmentation Only relevant audio segments are retrieved and played.

• Segment Each segment is annotated with free text.

• Time information Time information encoded in the annotation.

Multimedia Database Management System - Chapter 8

Integrated Image IR

Integrated image IR is similar with integrated audio IR. Structured attributes and free text annotation are used for integrated image IR.

Multimedia Database Management System - Chapter 8

Integrated Video IR

Annotation describes object spatial and temporal relationships within a video. Annotations are obtained in three ways: • Transcripts in a video. • Automatic speech and music recognition on the

video soundtrack. • Entering annotation manually by a domain expert.

Time consuming but useful and comprehensive.

Multimedia Database Management System - Chapter 8

Merging of Results based on Individual Features

How to obtain a ranked return list based on each combined features? How to obtain the final list? • Different similarity used for different features. • Similarity values from different return list have different ranges and

meanings. • Different features have different significances to the user.

What if user chooses to have different weights for different features?

Multimedia Database Management System - Chapter 8

Merging of Results based on Individual Features

Merging Process. Suppose the following are two ranked lists based on feature 1 and feature 2. Feature 1 (C:0.8, G:0.6, A:0.5, M:0.4, H:0.2)

Feature 2 (A:0.7, C:0.5, N:0.4, D:0.3, P:0.1)

Letters denote retrieved items and the associated numbers are the normalized similarity measures of the item for the given feature. • What is the ranked list if user chooses the same weight of

50% for each the two features? • What is the ranked list if user chooses 30% of first feature

and 70% of second feature?

Multimedia Database Management System - Chapter 8

Media TranslationUser is interested in relevant information regardless of the

media types. All relevant information, whether

it is in the form of text, audio, image and video should be presented to the user even

though the query is in the form of structured attributes, free text,

image, audio or video or their combination.

Retrieval works in one direction in the media hierarchy.

Multimedia Database Management System - Chapter 8

A General Architecture of Multimedia Information Management

In the architecture of multimedia information management, the basic information unit is modeled by a multimedia object that consist of a number

of different media with semantic, spatial, and temporal relationship.

Multimedia Database Management System - Chapter 8

Database Population Phase• Features of each multimedia items are analysed and

extracted. • The extracted feature vectors are stored in

appropriate feature indexes. Each feature vectors has a pointer to original multimedia items.

• The different media types of a multimedia objects are stored in different types of storage or same storage.

• They have to be stored in way such that continuous multimedia presentation is supported.

Multimedia Database Management System - Chapter 8

Text Index• After feature extraction, structured data or attributes

are stored and managed by a relational database. • The text index stores and manages textual

information using chosen IR techniques such as those based on inverted files and the vector space model.

• Textual information is derived from various sources such as text documents, free text annotations of audio, image and video, and recognized spoken words from speech.

Multimedia Database Management System - Chapter 8

Spatial Relationship

• The spatial relationship index will store and manage names of spatial objects and their locations, and is organized in an enhanced inverted file.

• Spatial information is specified by the coordinates of each object's centroid.

• For example: Object 3: MMO2(40, 70), MMO7(80, 120)

Multimedia Database Management System - Chapter 8

Temporal Relationship

• Similarly, the temporal relationship index stores and manages the names of temporal objects and their number of occurrences and can be organized in an enhanced inverted file.

• Temporal information about each object is specified by the appearance and disappearance times of the object relative to the beginning of the MMO.

• For example: Object 4: MMO5(2, 9), MMO9(7, 15)

Multimedia Database Management System - Chapter 8

User Interface

The user interface has three main functions: • To support multimedia object authoring,

composition, annotation and insertion into the database,

• To facilitate searching and browsing, • To present search results.

Multimedia Database Management System - Chapter 8

Multimedia Authoring and Annotation

• Preferred tool for multimedia authoring is graphic based. It let users easily specify spatial and temporal relationships among multimedia object components.

• Tool for multimedia annotation let users easily enter structured attributes and free text description, and annotation on spatial and temporal relationships among multimedia object components.

Multimedia Database Management System - Chapter 8

Search and BrowsingTypes of query which are supported with the user interface: • SQL queries. • Queries specified by certain query languages with multimedia

capabilities. • Queries based on keywords with connected logical operators. • Queries with free text. • Queries by example of multimedia objects. • Queries by humming. • Queries by sketch. • Queries by specifying feature values.

Multimedia Database Management System - Chapter 8

Result Presentation and Relevance Feedback

• Retrieval results should be presented in such a way that the user can quickly see if an item is relevant by using visual icons and information compaction.

• More importantly, the system should allow the user to refine or reformulate queries based on the initial returned items.