1

Video Frames Interpolation Using Adaptive Warping

Ying Chen LouMajor Advisor: M.J.T. Smith

Co-advisor: Edward DelpNov. 15, 2010

2

Outline

• Background

• Generic motion model

• Video spatial interpolation

• Video compression

• Video frame rate up-conversion

• Summary and future work

3

Motivation• Spatial interpolation

– Conversion from SDTV to HDTV

– Zooming of region of interest (ROI)• Surveillance/forensics

• Medical imaging

• Satellite imaging

• Temporal interpolation (Frame rate up-conversion)– 3:2 pull down 24Hz -> 30Hz

– Avoid flicker and blurring on LCD

– Loss of frames in transmission

Frame RateUp-Converter

4

Challenges and Goal• Core: motion estimation

• To derive a generic motion model which can be used for different applications

– Motion needs to be accurate• Ill-posed problem (aperture problem)

– Suitable for different types of motion• Translational (panning)

• Zoom in/out

• Rotational

– Low to moderate computational

intensive

Local window used for ME

5

Illustrative ExampleBlock Matching vs. Optical Flow

Motion

Block Matching

Block Matching

Optical Flow

Optical Flow

6

Motion Estimation Method (Warping)

I[n1, n2, k] = I[n1 + d1[n1, n2, k], n2 + d2[n1, n2, k], k + δk]

OFE

where

( ( ), ( ), ) ( ( ), ( ), ) ( ( ), ( ), )0x y

I x t y t t I x t y t t I x t y t tv v

x y t

1 2( , , ) ( , , ) ( , , ) ( , , ) ( , , ), ,x y

d x y k d x y k I x y t I x y k I x y k kv v

k k t k

1 2

( , , ) ( , , )( , , ) ( , , ) ( , , ) ( , , ) 0

I x y k I x y kd x y k d x y k I x y k k I x y k

x y

7

• Assumes that the pixel displacement functions within some region R of an image I[n1,n2,k] can be written as:

• and are vectors composed of the displacement parameters to be estimated

• The bilinear displacement parameter are computed by minimizing the mean squared error function (MSEF)

1

Td n n

��������������������������������������� ��� 2 n n��������������������������������������� ��� T

d

��������������

��������������

21 1 2 1 2( [ , ] [ ( ), ( )])

������������������������������������������������������������������������������������ T T

k kn R

Y n n Y n n n n

, ����������������������������

Warping Method (cont’d)

8

Quad-tree Divisions Makes the algorithm adaptive and more efficient

Quad-tree Uniform

9

Application I Spatial Interpolation Motivation

• Characteristics of lecture videos– Large static backgrounds

– Little to medium motion of foreground

• Possible to store several high resolution frames and retrieve them later

• Simplest scenario– periodically transmit one high resolution frame and the remaining

frames are in the form of low resolution

• Extend the proposed method to other types of video

1

2 3 4 N1 2 3 N

10

• Interpolation– Bilinear

– Adaptive Synthesis

Filter Banks

• Warping– Block matching-based and

optical flow-based

– Quadtree splitting

High quality full resolution frame

Half resolution frames

1

2 3 4 N1 2 3 N

High quality full resolution frame Bilinear interpolated

full resolution frame

1 2 3 N

High quality full resolution frame Bilinear interpolated

full resolution frame

warping

warping

Proposed method I: Full-band Warping (FWWA)

11

High Motion Breakdown

12

Robustness Issue

• Address the robustness issue– Obtain reliable motion vectors

– Maintain the sharpness

• Challenges– No corresponding pixels in the

reference frame

– Ambiguity of improvement in

sharpness and distortion objectively

13

Proposed method IIThe Composite Algorithm

• Incorporate advanced spatial interpolation algorithms

• Bidirectional warping – Forward and backward warping

– To solve “no corresponding pixels in the reference frame”

• Hierarchical motion structure– To solve “ambiguity of improvement in sharpness and geometric

distortion ”

1 2 3 N

High quality full resolution

frame

Bilinear interpolated full resolution frame

Forward warping

Backwardwarping

ZF3

ZB3

Original Low

resolution frame

Down-sampled

Down-sampled

ZB3'

X3

ZF3'

N+1

14

Hierarchical Motion Structure

ZF3 – forward warped frame ZF3’ – downsampled forward warped frame

ZB3 – backward warped frame ZB3’ – downsampled backward warped frame

X3 – original MxN low resolution frame

diff <thresh

ZF3' X3

Compare ZB3' and

XI3

Compare ZF3' and

XI3

ZB3' X3

diff <thresh

Y

N

Y

N

4 pixels from ZF3

4 pixels from ZB3

XI3

4 pixels from

15

Experiment Setup• Assessment of the composite algorithm

– Compare with bilinear, bicubic, NEDI, VA, and the full-band warping algorithm (FWWA)

• Investigate the impact of different spatial interpolation methods on the composite warping algorithm

– Incorporate different spatial interpolation methods in the first step

– The remaining stages are the same

• 3 sets of video sequences, CIF, 50 frames– Low motion (‘talking head’ lecture video )

– High detail

– High motion

16

Assessment of the Composite Algorithm• Every 5th frame as a high resolution reference frame

• Decimate frames to get low resolution frames – Use a 21-tap lowpass filter

– Downsample factor 2

• Use original frames as ground truth

• Competing methods– Bilinear, Bicubic

• simple image interpolation methods

– New-edge Directed Interpolation (NEDI)

• an advanced image interpolation method

– A super resolution method proposed by Patrick Vandewalle (VA) 1

– Full band warping (FWWA)[1] P. Vandewalle, S. Susstrunk, and M. Vetterli, “A frequency domain approach to registration of aliased images with

application to superresolution”, EURASIP Journal on Applied Signal Processing, 2006

17

Objective ResultsAkiyo Carphone News Silence Mother

Bilinear 34.23 32.3 28.86 32.99 36.46

Bicubic 35.08 32.92 29.85 33.64 37.45

NEDI 34.98 32.64 28.7 32.76 36.58

VA 35.08 32.91 29.86 33.65 37.47

FWWA 41.84 32.79 33.34 34.51 40.74

Comp 43.15 35.08 37.04 38.34 40.89

Sales Bus Flower Tempete

Mobile

30.07 25.84 23.07 26.66 22.39

30.63 26.55 23.47 27.23 22.94

30.05 25.35 22.71 26.34 22.11

30.63 26.56 23.47 27.23 22.94

32.13 18.85 26.11 27.28 24.42

35.31 25.85 27.92 31.15 25.79

Stefan Table Football

26.56 29.2 28.47

27.45 29.87 29.54

25.91 29.25 28.14

27.45 29.88 29.54

20.40 27.41 19.98

26.69 30.75 27.07

18

Talking Head VideoOriginal frame Bilinear interpolation

FWWA Comp

19

High Detail VideoOriginal frame Bilinear interpolation

FWWA Comp

20

High Motion VideoOriginal frame Bilinear interpolation

FWWA Comp

21

(a) (b)

(c)

(c) (d)

(a) Bilinear (b) NEDI (c) VA (d) Bicubic + Comp

22

Demo

23

Different Spatial Interpolation

Akiyo Carphone News Silence Mother

Bilinear 34.23 32.3 28.86 32.99 36.46

Bicubic 35.08 32.92 29.85 33.64 37.45

NEDI 34.98 32.64 28.7 32.76 36.58

VA 35.08 32.91 29.86 33.65 37.47

Bi-Comp 42.09 34.84 36.17 38.2 40.54 Cu-Comp 43.15 35.08 37.04 38.34 40.89

NEDI-Comp 41.24 34.14 35.23 36.98 39.53 VA-Comp 43.13 35.08 37.04 38.24 40.68

24

Conclusion

• The composite methods achieved good spatial interpolation results

• Accommodate complex motion

• Outperform competing methods subjectively and objectively– Improvement comes mostly from the warping process

– A combination of bicubic interpolation and warping results in best overall performance

• Complexity not too high

• Subjective and objective results are satisfactory

• Particularly perform well for lecture videos and high detail videos

25

Application II Video Compression H.264/AVC

• Retain edges but remove texture at low bitrates

26

Goal and Proposed Method• Goal

– Propose a coding method which keeps the high frequency components

– Achieve as high visual quality as possible

– Maintain integrating of H.264/AVC coder which is well engineered

– Be robust

• Three assumptions– Smaller resolution requires fewer bits

– Sequences with low motion don’t need full resolution coding

– Key frames more bits

• Proposed method– Adaptive warping

– Spatio-temporal

27

Overall System

Downsample to

M/2xN/2

H.264 Intra

Coded

H.264 Inter

Coded

Reference Frames

MxN

LL FramesM/2xN/2

Downsample To

M/2xN/x

Input Sequence

Pre-processing

Downsample to

M/2xN/2

H.264 Intra

Deoded

H.264 Inter

Decoded

Compression/Decompression

Interpola-tion

Warping

Reconstructed Frames

Selection

Post-processing

encoder decoder

28

Experiment Setup

• New algorithm compared against H.264 in the following setup• H.264 codec

– Every Nth frame as an I frame, the rest coded as P or B frames

• Proposed method– Every Nth frame used as the reference frame– Other frames are decimated (LL subband) and coded – Total bit rate is sum of full resolution reference frame and the quarter

resolution LL subbands

• Bit rates are the same in both cases

29

Result (1)

30

Subjective Result

(a) H.264 (b) proposed method 3rd frame for Salesman sequence @ 80kbits/s

31

Conclusion• The proposed method achieve better visual quality at low

bitrates

• The gap decreases as the bitrates increase

• At high bitrates, H.264 has more bits to spend on high frequency components and thus achieves better quality

• The nature of the proposed method works better for sequences with more details

• Room remained to be improved• Explore tradeoffs in spatio-temporal decimation rates• More frequently for video with large motion and less often for video with

small motion

• For long lecture video, we can choose full coverage of reference frame and no anymore later

32

Application III Video Frame Rate Up-Conversion

• Overview of FRUC– No motion vectors

• Frame repetition, frame average

– Use motion vectors• Use motion vectors from the decoder directly

• Advantage: Low complexity

• Disadvantage: Not true motion

• Perform motion estimation again• Advantage: true motion

• Disadvantage: High computational complexity

Frame RateUp-Converter

33

Goal and Proposed Method• Goal

(1) True motion vectors (2) Relative low complexity

• Challenges(1) Highly accurate MVs (2) Low percentage of MV re-

estimation

(3) Occlusion (4) Blocking artifacts

• Approach– Decoded video sequences from the decoder

– Additional information from the decoder

• System diagramPrevious Reconstructed

frame X1

Current residual frame,

reconstructed frame X2, and its MVs

MV Reliability

Check

MV Re-estimation

Motion CompensatedInterpolation

Interpolated FrameSmall

BlockMerging

34

MV reliability check• Categoried into 3 groups

35

Small Block Merging

• Avoid broken edges and want to maintain object structure

36

MV Re-estimation

• Key in the system

• Accurate MVs are required for FRUC

• Low complexity

• A combination of Optical Flow-based and Block Matching-based motion estimation (Warping method)

37

Motion Compensated Interpolation

N=2, k=1

38

Occlusion

• Uni-directional interpolation

N=2, k=1

)( )( if ]2

1[

)( )( if ]2

1[

][

2

1

BFB

BFF

m

vmBADvmBADvmxX

vmBADvmBADvmxXxX

39

Overlapped Boundary Motion Compensation (OBMC)

Goal:• To reduce the blocking artifacts.• Selectively perform OBMC to reduce the computational complexity (BAD > T)

40

Experiment Setup• JM 11.0

• GOP: IPP…P, 15th I frame, fixed QP

• Code odd frame and skip every other frame

• 15fps

• Transform 8x8=2 mode on

• Search range = 16

• Standard CIF Video sequences used:– Akiyo, News, Salesman, Foreman, Carphone

– Flower Garden, Tempete

– Football, Table Tennis

• Test against DMCFI, correlation-based motion selection1

[1] Ai-Mei Huang and T. Nguyen, “Correlation-based motion vector processing for motion compensated interpolation”,

ICIP 2008

41

Visual Result (1) 384kb/s

(a) Orig

(b) DMCFI

20.54dB

(c )Correlat

ion-based

20.48dB

(d)Proposed24.19dB

42

Visual Result (2) 512kb/s

(a) Orig

(b) DMCFI

20.01dB

(c )Correlat

ion-based

20.17dB

(d)Proposed20.13dB

43

Conclusion

• Proposed a FRUC method that combines optical flow and block matching-based motion estimation

• Reduced computational complexity

• Reduced blocking artifacts

• Achieve better visual quality for low motion video sequences and perform on par with other methods for high motion video sequences

44

Summary• Provide a generic framework to achieve

– Spatial enlargement of video frames

– Video compression

– Frame rate up-conversion

• Achiever higher objective and subjective results

• Improve the robustness by using FW, BW and hierarchical motion structure

45

Future Work

• Continue to refine the model

• Apply to higher resolution video

• Incorporate Subjective Video Quality Analysis

• Reference frame recycling– Adaptively select the position of high quality reference frame

46

Q & A

47

Application I Spatial InterpolationRelated Work

• Frame restoration

• Frame interpolation– Bilinear, bicubic, spline, …

– Adaptive Synthesis Filter Banks

– New edge-directed interpolation

• Superresolution (SR)

48

Different Spatial InterpolationAkiyo Carphone News Silence Mother

Bilinear 34.23 32.3 28.86 32.99 36.46

Bicubi 35.08 32.92 29.85 33.64 37.45

NEDI 34.98 32.64 28.7 32.76 36.58

VA 35.08 32.91 29.86 33.65 37.47

Bi-Comp 42.09 34.84 36.17 38.2 40.54

Cu-Comp 43.15 35.08 37.04 38.34 40.89

NEDI-Comp 41.24 34.14 35.23 36.98 39.53

VA-Comp 43.13 35.08 37.04 38.24 40.68

Sales Bus Flower Tempete Mobile

30.07 25.84 23.07 26.66 22.39

30.63 26.55 23.47 27.23 22.94

30.05 25.35 22.71 26.34 22.11

30.63 26.56 23.47 27.23 22.94

34.96 25.7 27.38 30.28 25.66

35.31 25.85 27.92 31.15 25.79

34.21 25.25 27.13 30.15 25.79

35.3 25.85 27.9 31.15 26.93

Stefan Table Football

26.56 29.2 28.47

27.45 29.87 29.54

25.91 29.25 28.14

27.45 29.88 29.54

26.59 30.55 27.26

26.69 30.75 27.07

26.12 30.15 26.48

26.69 30.73 27.08