Content-based Video Indexing and Retrieval

Motivation

There is an amazing growth in the amount of digital video data in recent years.

Lack of tools for classify and retrieve video content

There exists a gap between low-level features and high-level semantic content.

To let machine understand video is important and challenging.

Motivation

Necessity of Video Database Management System Increase in the amount of video data captured. Efficient way to handle multimedia data.

Traditional Databases Vs Video Databases Traditional Databases has tuple as basic unit of data. Video Databases has shot as basic unit of data.

Video Management

Video consists of:

Text Audio Images

All change over time

Video Data Management

Metadata-based method Text-based method Audio-based method Content-based method Integrated approach

Metadata-based Method

Video is indexed and retrieved based on structured metadata information by using a traditional DBMS

Metadata examples are the title, author, producer, director, date, types of video.

Text-based Method

Video is indexed and retrieved based on associated subtitles (text) using traditional IR techniques for text documents.

Transcripts and subtitles are already exist in many types of video such as news and movies, eliminating the need for manual annotation.

Text-based Method

Basic method is to use human annotation Can be done automatically where subtitles /

transcriptions exist BBC: 100% output subtitled by 2008

Speech recognition for archive material

Text-based Method Key word search based on

subtitles Content based

Live demo: http://km.doc.ic.ac.uk/vse/

Text-based Method

Audio-based Method

Video is indexed and retrieved based on associated soundtracks using the methods for audio indexing and retrieval.

Speech recognition is applied if necessary.

Content-based Method

There are two approaches for content-based video retrieval:

Treat video as a collection of images Divide video sequences into groups of

similar frames

Integrated Approach

Two or more of the above techniques are used as a combination in order to provide more flexibility in video retrieval.

Video Data Management

1. Video Parsing• Manipulation of whole video for breakdown into key

frames.

2. Video Indexing• Retrieving information about the frame for indexing

in a database.

3. Video Retrieval and browsing• Users access the db through queries or through

interactions.

Video Parsing

Scene: single dramatic event taken by a small number of related cameras.

Shot: A sequence taken by a single camera Frame: A still image

Detection and identification of meaningful segments of video.

Video

Scenes

Shots

Frames

Key Frame Analysis

Shot Boundary Analysis

Obvious Cuts

Video Parsing

Video Parsing

System overview

Query

Feature vectors

Feature generation

Distances Retrieved results

Weighted sum of

distances

Shot boundary detection

Key frames

Test data

.

.

.

45217853

ΣwD

Relevance feedback

k-nn (boost, VSM)

.

.

. 45217853

45217853

45217853

Video Shot Definition

A shot is a contiguous recording of one or more video frames depicting a contiguous action in time and space.

During a shot, the camera may remain fixed, or may exhibit such motions as panning, tilting, zooming, tracking, etc.

Video Shot Detection

Segmentation is a process for dividing a video sequence into shots.

Consecutive frames on either side of a camera break generally display a significant quantitative change in content.

We need a suitable quantitative measure that captures the difference between two frames.

Video Shot Detection

Use of pixel differences: tend to be very sensitive to camera motion and minor illumination changes.

Global histogram comparisons: produce relatively accurate results compared to others.

Local histogram comparisons: produce the most accurate results compared to others.

Use of motion vectors: produce more false positives than histogram-based methods.

Use of the DCT coefficients from MPEG files: produce more false positives than histogram-based methods.

Frame Dissimilarity Normalized color histogram difference is

adopted Measure of dissimilarity, or distance

Shot Dissimilarity Minimum dissimilarity between any two

frames of two shots

Shot Boundary Detection

Distance of FramesD(ƒi , ƒj) = ∑b|hib – hjb|/N

hib , a given bin in the histogram of frame i,

N , the total number of pixels in each frame.

Measure of Shot DissimilarityD(Si , Sj) = mink,l D(ƒki , ƒlj)

ƒki , frame k of shot i.

Shot boundary detection

Split video into meaningful segments Traditionally look at inter-frame differences Common problems

Gradual changes Rapid motion

Our solution Inspired by Pye et al, Zhang et al Moving average over greater range

Shot boundary detection

At each frame, compute 4 distance measures – d2 , d4, d8, d16 across ranges of 2,4,8,16 frames respectively

Coincident peaks indicate shot boundaries d4 difference used to find transition

start/end times

SBD examples

Cut Gradual

Video Indexing and Retrieval

Based on representative frames Based on motion information Based on objects

Representative Frames

The most common way of creating a shot index is to use a representative frame to represent each shot.

Features of this frame are extracted and indexed based on color, shape, texture (as in image retrieval)

Representative Frames

If shots are quite static, any frame within the shot can be used as a representative.

Otherwise, more effective methods should be used to select the representative frame.

Representative Frames

Two issues:

How many frames must be selected from each shot.

How to select these frames.

Representative Frames

How many frames per shot? Three methods:

One frame per shot. The method does not consider the length and content changes.

The number of selected representatives depends on the length of the video shot. Content is not handled properly.

Divide shots into subshots and select one representative frame from each subshot. Length and content are taken into account.

Representative Frames

Now we know the number of representative frames per shot.

The next step is to determine HOW to select these frames.

Representative Frames

Definition:

SEGMENT is

a shot, a second of a video, a subshot

Representative Frames

Method I

The first frame is selected from the segment. This is based on the observation that a segment is usually described by using the first few frames.

Representative Frames

Method II

An average frame is defined so that each pixel in this frame is the average of pixel values at the same grid point in all the frames of the segment.

Then the frame within the segment that is most similar to the average frame is selected as the representative.

Representative Frames

Method III

The histograms of all the frames in the segment are averaged. The frame whose histogram is closest to this average histogram is selected as the represenative frame of the segment.

Representative Frames

Method IV

Each frame is divided into background and foreground objects. A large background is then constructed from the background of all frames, and then the main foreground objects of all frames are superimposed onto the constructed background.

Foreground and Background Variance Method Overview

Videos are divided into categories along with their shots. We calculate the Foreground Variance,Background Variance and

Average Color of shots, and store them in the database. Shots are retrieved by comparing the Foreground Variance,

Background variance and Average Color values.

What is Background and Foreground? Background is the area that is outside the primary object. Foreground is the area where the primary object can be found.

Foreground and Background Variance Method

Choosing Foreground and Background. W = C*(1/10).

c

w

w

w

Foreground and Background Variance Method Actual Method

Steps for calculating Foreground Variance values Take each pixel of the Foreground area and access its individual

Red, Green, and Blue values. Calculate Average Red, Average Green, and Average Blue

Color values for Foreground.Repeat the above process for all the frames of the shot and calculate Average Red, Average Green, and Average Blue Color values.

Using the above Foreground values of all the frames of a shot, we calculate the Variance of Red, Green, and Blue.

Foreground and Background Variance Method Actual Method

Steps for calculating Foreground Variance values The Formula for calculating the Variance of Red Color for

Foreground is VFgRed = ∑ (Xi – Mean)2/(N-1), where Xi = Average values of Red Color of all the frames for Foreground, and N = Total No of frames.

The same process in the above step is repeated for Green and Blue and we find VFgGreen and VFgBlue

In the similar lines we find the Background Variance values VBgRed, VBgGreen, and VBgBlue.

Foreground and Background Variance Method Actual Method

Steps for calculating Average Color values. Access each pixel of each frame and calculate individual color values. Add up all the individual Red, Green and Blue values of each pixel

separately. For calculating Average Red Color for one frame, divide the sum of

all the red pixel values by the total No of pixels in one frame. For calculating the Average Red Color values for the entire shot, we

divide the sum of all the Red Color values of individual frames by the total No of frames. Thus we will be getting AvgRed.

Foreground and Background Variance Method Actual Method

Steps for calculating Average Color values. Similarly we calculate the AvgGreen, and AvgBlue

values for the entire shot. We have a total of nine different variables and we

store these values in the Database. For retrieving similar shots, we compare the

above nine values in the Database to the corresponding values of the query shot.

Foreground and Background Variance Method Actual Method

We compare the Foreground, Background, and the Average Color values using the formula

Ri = √((∆1-∂1)2 + (∆2-∂2)2 + (∆3-∂3)2) where ∆1, ∆2, and ∆3 are Database values. where ∂1, ∂2, and ∂3 are query shot values. We add up all the Ri values after comparing the Foreground,

Background and Average Color values. If that values is less than 100, then that shot is retrieved and

displayed. The shots are displayed in the increasing order of their closeness to the query shot.

Motion Information

Motivation

Indexing and retrieval based on representative frames ignores the motion information contained in a video segment.

Motion Information

The following parameters are used:

Motion content Motion uniformity Motion panning Motion tilting

Motion Information (content)

This a measure of the total amount of motion within a given video. It measures the action content of the video.

For example, a talking person video has a very small motion content, while a violent car explosion typically has high motion content.

Motion Information (uniformity)

This is a measure of the smoothness of the motion within a video as a function of time.

Motion Information (panning)

This measure captures the panning motion (left to right, right to left motion of a camera).

Motion Information (tilting)

This is a measure of the vertical motion component of the motion within a video. Panning shots have a lower value than a video with large amount of vertical motion.

Motion Information

The above measures are associated either with the entire video or with each shot of the video.

Object-based Retrieval

Motivation

The major drawback of shot-based video indexing is that while the shot is the smallest unit in the video sequence, it does not lend itself directly to content-based representation.

Object-based Retrieval

Any given scene is a complex collection of parts or objects.

The location and physical qualities of each object, as well as the interaction with others, define the content of the scene.

Object-based techniques try to identify objects and relationships among these objects.

Object-based Retrieval

In a still image object segmentation and identification is normally a difficult task.

In a video sequence, an object moves as a whole. Therefore, we can group pixels that move together into an object.

Object segmentation is quite accurate by using the above idea.

Object-based Retrieval

Object-based video indexing and retrieval can be performed easily when video is compressed using the MPEG-4 object-based coding standard.

An MPEG-4 video is composed of one or more VOs. A VO consists of one ore more video object layers (VOLs).

RawVideo

Database

RawImage

Database

PhysicalObject

Database

Frame

Object Description

Image Features

Spatial-Semanticsof Objects

(human,building,…)

Sequence of Frames(indexed)

Object Identification and Tracking

Intra/Inter-Frame Analysis

(Motion Analysis)

Inter-Object Movement(Analysis)

Semantic Association(President, Capitol,...)

Spatio-Temporal Semantics:Formal Specification of

Event/Activity/Episode forContent-Based Retrieval

Spatial Abstraction

TemporalAbstraction

Object Definitions(Events/Concepts)

An Architecture for Video Database System

Conclusion

Video indexing and retrieval is very important in multimedia database management.

Video contains more information than other media types (text, audio, images).

Methods: representative frames, motion information, object-based retrieval.