Video summarization by Video summarization by vvideo ideo sstrtructure analysis and graph optimizucture analysis and graph optimiz

ationation

M. Phil 2M. Phil 2ndnd Term Presentation Term Presentation

Lu ShiLu Shi

Dec 5, 2003Dec 5, 2003

OutlineOutline

MotivationMotivation Video structureVideo structure Video skim length distributionVideo skim length distribution Spatial-temporal graph modeling Spatial-temporal graph modeling Optimization based video shot selectionOptimization based video shot selection Experimental resultsExperimental results

MotivationMotivation

Huge volume of video data are distributed over the Huge volume of video data are distributed over the WebWeb

Browsing and management in the huge video Browsing and management in the huge video database are time consumingdatabase are time consuming

Help the user to quickly grasp the content of a videoHelp the user to quickly grasp the content of a video

Two kinds of applications:Two kinds of applications: Video skimming (dynamic)Video skimming (dynamic) Video static summary (static)Video static summary (static)

GoalsGoals

ConcisenessConciseness Content coverageContent coverage

Spatial and temporalSpatial and temporal CoherencyCoherency

Not too jumpyNot too jumpy

FlowchartFlowchart

Video structureVideo structure

Video narrates a story just like an article doesVideo narrates a story just like an article does Video (story)Video (story) Video scenes (paragraph)Video scenes (paragraph) Video shot groups Video shot groups Video shots (sentence)Video shots (sentence) Video framesVideo frames

Video structure Video structure Graphical exampleGraphical example

Video structureVideo structure

Can be built up in a bottom-up mannerCan be built up in a bottom-up manner Video shot detectionVideo shot detection Video shot groupingVideo shot grouping Video scene formation Video scene formation

Video structureVideo structure

Video shot detectionVideo shot detection Video slice image [1]Video slice image [1] Column - pairwise distanceColumn - pairwise distance Filtering and thresholdingFiltering and thresholding

… …… …

Video structureVideo structure

Video shot groupingVideo shot grouping Window-sweeping algorithm [2]Window-sweeping algorithm [2] Spatial similaritySpatial similarity Temporal distanceTemporal distance Intersected video shot groups form loop scenesIntersected video shot groups form loop scenes

Video structureVideo structure

Summarize each video scene respectivelySummarize each video scene respectively Loop scenes and progressive scenesLoop scenes and progressive scenes

Loop scenes depict an event happened at a placeLoop scenes depict an event happened at a place Progressive scenes: “transition” between events or Progressive scenes: “transition” between events or

dynamic eventsdynamic events

Video structureVideo structure

Scene importance: length and complexityScene importance: length and complexity Content entropy for loop scenesContent entropy for loop scenes Measure the complexity for a loop sceneMeasure the complexity for a loop scene

)log()(i

j

i

j

Sc

Sg

j Sc

Sg

i l

l

l

lScEntropy

Video structureVideo structure

Determine each video scene’s target skim lengthDetermine each video scene’s target skim length Determine each progressive scenes’ skim lengthDetermine each progressive scenes’ skim length

If , discard it, else If , discard it, else

Determine each loop scenes’ skim lengthDetermine each loop scenes’ skim length If ,discard itIf ,discard it

Redistribute to remaining scenesRedistribute to remaining scenes

1tL

Ll

v

vsSci

v

vsScSc L

Llskim

ii

2)(

)(' t

ScEntropyl

ScEntropylLskim

jjSc

iScvsSc

j

i

i

vsL'

Graph modelingGraph modeling

Spatial-temporal dissimilarity functionSpatial-temporal dissimilarity function Linear with visual dissimilarityLinear with visual dissimilarity Exponential with temporal distanceExponential with temporal distance

)),((),(1),( ji shshsTemporalDikjiji eshshVisualSimshshDis

Graph modelingGraph modeling

The spatial temporal relation graph The spatial temporal relation graph Each vertex corresponds to a video shotEach vertex corresponds to a video shot Each edge corresponds to the dissimilarity function betweeEach edge corresponds to the dissimilarity function betwee

n shotsn shots Directional and completeDirectional and complete

Skim generationSkim generation

The goal of video summarizationThe goal of video summarization Conciseness: given the target skim lengthConciseness: given the target skim length Content coverageContent coverage The spatial temporal dissimilarity functionThe spatial temporal dissimilarity function

The spatial temporal relation graph The spatial temporal relation graph A path corresponds to a series of video shotsA path corresponds to a series of video shots Vertex weight summationVertex weight summation Path length is the summation of the dissimilarity between Path length is the summation of the dissimilarity between

consecutive shot pairsconsecutive shot pairs

vsL

Skim generationSkim generation

Objectives:Objectives: Search for a path in the graph such that:Search for a path in the graph such that: Maximize the path length (dissimilarity Maximize the path length (dissimilarity

summation)summation) Vertex weight summation should be close to Vertex weight summation should be close to

but not exceed itbut not exceed it The objective function The objective function

vsL

vssvsspvssobj LpVWSLpVWSwLLpfs

)(),)((),(

sp

Skim generationSkim generation

Global optimal solutionGlobal optimal solution Let denote the paths begin with , whose Let denote the paths begin with , whose

vertex weight summation is upper bounded byvertex weight summation is upper bounded by The optimal path is denoted by The optimal path is denoted by

The target is The target is )( ,0

ovobj

vsLpf

}{ ,i

Lv rxp

rL

xv

)(max)( ,,i

Lvxobjio

Lvobj rrxpfpf

Skim generationSkim generation

Optimal substructureOptimal substructure

Dynamic programmingDynamic programming Effective way to compute the global optimal Effective way to compute the global optimal

solution solution Trace back to find the optimal pathTrace back to find the optimal path Time complexity , space complexity Time complexity , space complexity )( 2

vsLnO

nxlwshshDispfpfishiri

n

xirx shixo

lLvobjv

vvo

Lvobj ),),()((max)( ,1,

)()( , snrn vsho

Lvobj Llwpf

)( vsLnO

ExperimentsExperiments Key frames of selected video shotsKey frames of selected video shots

ExperimentsExperiments There is no ground truth so that it is hard to objectively evaluate a video There is no ground truth so that it is hard to objectively evaluate a video

skimskim Subjective experimentSubjective experiment Parameters:Parameters: 250,01.0sec,4sec,3 21 kwtt

ConclusionConclusion

Video structure analysisVideo structure analysis Scene boundaries, sub-skim length determinationScene boundaries, sub-skim length determination

Graph scene modelingGraph scene modeling Optimization based sub skim generationOptimization based sub skim generation Generate a video skimGenerate a video skim

ReferenceReference

[1] C. W. Ngo, Analysis of spatial temporal sli[1] C. W. Ngo, Analysis of spatial temporal slices for video content representation, Ph. D theces for video content representation, Ph. D thesis, HKUST, Aug.2000sis, HKUST, Aug.2000

[2] [2] Y. Rui, T.S. Huang, and S. Mehrotra, Constructing table-of content for videos, ACM Multimedia Systems Journal, Special Issue Multimedia Systems on Video Libraries, vol. 7, no.5, pp. 359~368, Sept 1999.

Q & AQ & A

Thank you!!Thank you!!