a robust scene-change detection method for video segmentation chung-lin huang and bing-yao liao ieee...

A Robust Scene-A Robust Scene-Change Detection Change Detection Method for Video Method for Video

SegmentationSegmentation

Chung-Lin Huang and Bing-Yao Chung-Lin Huang and Bing-Yao LiaoLiao

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY

OutlineOutline

• IntroductionIntroduction• Abrupt Scene-Change DetectionAbrupt Scene-Change Detection• Gradual Scene-Change DetectionGradual Scene-Change Detection• Experimental ResultsExperimental Results• ConclusionConclusion

IntroductionIntroduction

• The main problem of segmenting a The main problem of segmenting a video sequence into shots is the video sequence into shots is the ability to distinguish between ability to distinguish between scene scene breaks and normal changesbreaks and normal changes that that happen in the scenehappen in the scene

• This paper combines the This paper combines the intensityintensity and and motion informationmotion information to detect to detect scene changes such as abrupt scene scene changes such as abrupt scene changes and gradual scene changeschanges and gradual scene changes

Previous ProblemsPrevious Problems

• The two main problems in most existing The two main problems in most existing algorithmsalgorithms– they are they are threshold-dependent algorithmsthreshold-dependent algorithms– they they suffer false detectionsuffer false detection with scenes involving with scenes involving

fast camera or object motion.fast camera or object motion.

• This paper proposes a scene-change This paper proposes a scene-change detection algorithm with three contributionsdetection algorithm with three contributions– Relaxing thresholdRelaxing threshold selection problem selection problem– higher detection rate (scene change should not be higher detection rate (scene change should not be

missed)missed)– lower false alarm ratelower false alarm rate

ABRUPT SCENE-CHANGE ABRUPT SCENE-CHANGE DETECTIONDETECTION

• MethodMethod– Measurement of the Changes Between Measurement of the Changes Between

FramesFrames• Pixel-Based DifferencePixel-Based Difference• Histogram-Based DifferenceHistogram-Based Difference

– Static Scene TestStatic Scene Test– Scene Transition ClassificationScene Transition Classification– Detection AlgorithmDetection Algorithm

Detection AlgorithmDetection Algorithm• First phaseFirst phase

– locate the highest and the second highest peaks of DClocate the highest and the second highest peaks of DCimage difference in the midst of the sliding window, image difference in the midst of the sliding window, and then and then calculate the ratio ncalculate the ratio n between the first and se between the first and second peakscond peaks

• Second phaseSecond phase– Histogram MeasureHistogram Measure– Static Scene Test Static Scene Test (Most of the false alarms declared by the hi(Most of the false alarms declared by the hi

stogram detector are due to sudden light changes, while the edgstogram detector are due to sudden light changes, while the edge information is more or less invariant to these changes)e information is more or less invariant to these changes)

– Scene TransitionScene Transition

genuine AmbiguousNo Scene Change

Nhigh Nlow

Measurement of the Measurement of the Changes Between FramesChanges Between Frames

• Pixel-Based Difference:Pixel-Based Difference:–

• Histogram-Based Difference: (XHistogram-Based Difference: (X2 2 Test)Test)–

Where Cx and Cy are the DC image of frames X and Y

Color HistogramColor Histogram

• Efficient representationEfficient representation– Easy computationEasy computation– Global color distributionGlobal color distribution

• Insensitive toInsensitive to– RotationRotation– ZoomingZooming– Changing resolutionChanging resolution– Partial occlusionsPartial occlusions

• DisadvantageDisadvantage– Ignore spatial relationshipIgnore spatial relationship– Sensitive in illumination changesSensitive in illumination changes

• Choose illumination-insensitive color channelsChoose illumination-insensitive color channels

ExampleExample

• Color space selection & quantizationColor space selection & quantization– Use RGB channelsUse RGB channels

• Each channel is divided into 2 intervalsEach channel is divided into 2 intervals• Total number of bins = 2Total number of bins = 233 = 8 = 8

– HH((II): Color histogram for Image ): Color histogram for Image II

– HH11 = (7, 7, 7, 7, 9, 9, 9, 9) = (7, 7, 7, 7, 9, 9, 9, 9)

• Image 1 has 7 pixels in each Image 1 has 7 pixels in each CC11 to to CC44, and 9 , and 9 pixels in each pixels in each CC55 to to CC88

Static Scene TestStatic Scene Test

• Define Define – all objects present in the scene exhibit rather all objects present in the scene exhibit rather

small motion compared to the frame size, and small motion compared to the frame size, and global movement caused by the camera is slow global movement caused by the camera is slow and smooth.and smooth.

• MethodMethod– Edge DetectionEdge Detection– Edge DilationEdge Dilation

• ResultResult– The transition of two consecutive frames with The transition of two consecutive frames with

covering ratiocovering ratio larger than a larger than a predefined thresholdpredefined threshold is considered as a non-static or dynamic scene.is considered as a non-static or dynamic scene.

Example of edge Example of edge detectiondetection

Edge Detection

Edge Dilation ( r=3 )

Scene Transition Scene Transition ClassificationClassification

• Transition TypeTransition Type– 1) static scene to static scene1) static scene to static scene– 2) dynamic scene to static scene or vice 2) dynamic scene to static scene or vice

versaversa– 3) 3) dynamic scene to dynamic scenedynamic scene to dynamic scene

• Dynamic-to-dynamic transition usually Dynamic-to-dynamic transition usually indicates continuous object or camera indicates continuous object or camera motion, rather than a real scene changemotion, rather than a real scene change

Gradual Scene-Change Gradual Scene-Change DetectionDetection

• Gradual Scene-ChangeGradual Scene-Change– DissolveDissolve– Fade-InFade-In ( X = 0 )( X = 0 )– Fade-OutFade-Out ( Y = 0 )( Y = 0 )

• Why not easy to detectWhy not easy to detect– Camera and object motions always introduce Camera and object motions always introduce a a

larger variationlarger variation than a gradual transition. than a gradual transition.

Scene X to Y In Duration T

Intensity Statistics Intensity Statistics ModelModel

• Normal CaseNormal Case– For any frames near the reference frame, For any frames near the reference frame,

their their dissimilarity measure almost dissimilarity measure almost increases exponentiallyincreases exponentially with their with their distancedistance

• Gradual TransitionGradual Transition– The dissimilarity The dissimilarity increases linearlyincreases linearly with with

their distance during the transitiontheir distance during the transition– After the transition is over, the difference After the transition is over, the difference

measures are measures are randomly distributedrandomly distributed

Normal Sequence

Gradual Transition

Seed : the beginning frame of a gradual

transition

• N-distance measureN-distance measure– the difference measure generated by comparing a the difference measure generated by comparing a

frame with itself and its successive ( N – 1 ) framesframe with itself and its successive ( N – 1 ) frames– – Ideal model of the N-distance measureIdeal model of the N-distance measure

Gradual Scene-Change-Gradual Scene-Change-Detection AlgorithmDetection Algorithm

• 1) 1) N-distance measureN-distance measure• 2) Difference operation2) Difference operation

• 3) 3) Low-pass filtering: Implement a simple low-pass filter to keep the low frequency segments and to remove the high-frequency segments

Gradual Scene-Change-Gradual Scene-Change-Detection AlgorithmDetection Algorithm

• If the number of zero crossings between frame k and frIf the number of zero crossings between frame k and frame l (k and l is zero crossing frame)ame l (k and l is zero crossing frame)– Zero-crossing rate calculation:Zero-crossing rate calculation:

– larger than a threshold : larger than a threshold : high frequencyhigh frequency fragment fragment– else : else : low frequencylow frequency fragment (gradual scene-change segment) fragment (gradual scene-change segment)

• a local “score” record mechanisma local “score” record mechanism– ScoreScoreii(q)(q) q=0, 1, 2 …, N-1q=0, 1, 2 …, N-1

• High frequency fragment : ScoreHigh frequency fragment : Scoreii(q) = 0(q) = 0• Low frequency fragment : ScoreLow frequency fragment : Scoreii(q) = 1(q) = 1

0

50

100

150

1 3 5 7 9 11 13 15 17 19 21 23 25

-10

0

10

20

30

40

1 3 5 7 9 111315171921 232527

0

0.5

1

1.5

1 3 5 7 9 11131517192123 2527

N-Distance Measure

Difference operation

Local Score

Gradual Scene-Change-Gradual Scene-Change-Detection AlgorithmDetection Algorithm

• a global “track” record mechanisma global “track” record mechanism– Track(p)Track(p) p = 1, 2, 3, …,Lp = 1, 2, 3, …,L

• After every N-distance Measure of frameAfter every N-distance Measure of frame i i ,, we can get the local Score we can get the local Scoreii(q)(q)– Accumulate the Score record to the Track recordAccumulate the Score record to the Track record

the total number of frames in the video sequence

Improve Gradual Scene-Improve Gradual Scene-Change-Detection Change-Detection

AlgorithmAlgorithm• To develop a fast seed-searching processTo develop a fast seed-searching process

– we select one from every S consecutive frames for N-distance measure.

• Since gradual scene change does occur in Since gradual scene change does occur in segment 3 only, we need to ignore the segment 3 only, we need to ignore the scores in segment 1 due to correlation scores in segment 1 due to correlation behavior of the reference frame and its behavior of the reference frame and its neighboring frames.neighboring frames.

• The correlated distance in segment 1 is C

EXPERIMENTAL EXPERIMENTAL RESULTS (1)RESULTS (1)

EXPERIMENTAL EXPERIMENTAL RESULTS (2)RESULTS (2)

Performance Measure

Performance Result

EXPERIMENTAL EXPERIMENTAL RESULTS (3)RESULTS (3)

ConclusionConclusion

• This method avoided the false alarms This method avoided the false alarms by using the by using the validation mechanismvalidation mechanism..

• It also proves that the It also proves that the statistical statistical model-based approachmodel-based approach is reliable for is reliable for gradual scene-change detection.gradual scene-change detection.

• Experimental results show that a Experimental results show that a very very high detection ratehigh detection rate is achieved while is achieved while the false alarm rate is comparatively the false alarm rate is comparatively low.low.