Multimedia Search and Retrieval: New Concepts, System Implementation, and Application

Qian Huang, Atul Puri, Zhu Liu

IEEE TRANSACTION ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, AUGUEST 2000

Outline Introduction Media Types and Their Properties Multimedia Data Abstraction and Retrieval System Architecture System Implementation An Application: Multimedia Broadcast

News Summary

Introduction MPEG-7 focuses on the standardization of a

common interface for describing multimedia materials,supporting pull and push information.

Our proposed multimedia information retrieval system cover system architecture, data storage and representation, database and its remote access, and human-machine interfaces.

The concept of physical and logical layers and events.

Media Types and Their Properties Audio

Audio events can be defined as continuous audio segments that have some distinct acoustic characteristics

Visual Text

Multimedia Data Abstraction and Retrieval

A physical layer addresses the media events that are time continuous and better described using the specific characteristics of a media signal.

A logical layer addresses multimedia events that present integrated behavior of other events across media.

Multimedia Data Abstraction and Retrieval(cont.) Physical Program and Physical Event

Physical program a continuous stream of multimedia data that

begins and ends due to the physical scene edits and cuts decided by human operators

Physical event A continuous segment of multimedia data,

representing a coherent and meaningful event within a physical program

Multimedia Data Abstraction and Retrieval(cont.)

Logical Event A virtual view of the multimedia data constr

ucted from a set of physical programs or events, or recursively from other physical or logical events.

Application dependent Semantic Content Structure

Conceptual and virtual Provide a seamless interface to user Table of Content (ToC)

Relationship among different event layers

System Architecture

System Implementation Server Implementation

Content Hierarchy Extraction Layered Content Storage

Store lower level information in a file system and higher level information in standard database systems

When raw data is stored in a file system, different events in the data stream can be accessed using time based indexing

Database Content Access Employ JDBC to provide connectivity to our

database for query and update

Multimedia data storage in combined file system and database

Example of event indexing from database to a file

Database access

System Implementation(cont.)

Client Browser Implementation Java applet-based interface

Platform independence support for a variety of coded media

Remote Database Query RMI (Remote Method Invocation)

Browsing Interface Table of Content: a conceptual content

structure TimeLine: a content structure in time Event Presentation: presentation and playback

Content Playback with a JMF Player

Remote database access

Browser interface for a news program with both ToC and timeline representations of the semantic structure within the program

An Application: Multimedia Broadcast News Server End

Content Hierarchy Extraction Broadcast news data is well structured. Lower levels – extract primitive physical

events in each individual media stream. Higher semantic level – merge different

events into semantically meaningful logical events. ex: Headline stories and news summaries.

Content hierarchy of broadcast news programs

Content Storage and Access In the file system, for each physical

broadcast news program, store the timetables for audio, video,and text,individually.

The tables in the database system will index into these timetables for the actual retrieval and data access.

Physical events in each media are recorded in separate database tables at the physical event layer.

Relationship between logical event and physical event objects

Database design in broadcast news application

Client End Remote Database Query

The physical program ID is uniquely determined by the channel and the date of the program.

Content Presentation Content Structure Presentation

ToC and TimeLine Static(single click) and playback(double click)

Story Presentation News Summary Presentation

Streaming Playback

Browser interface for a news program with both ToC and timeline representations of the semantic structure within the program

Visual representation for story “El Nino”

Representation for news summary of the day

Playback interface with synchronized audio/video/text

Summary We have introduced a new layered

approach to multimedia content representation and storage for search and retrieval.

The concepts are implemented in a client/server multimedia information retrieval system designed for broadcast news search and retrieval.