Video Coding with Spatio-temporal Texture Synthesis and Edge-based inpainting

Chunbo Zhu , Xiaoyan Sun , Feng Wu , and Houqiang Li

ICME 2008

Outline

•Introduction•Framework

▫Previous work▫Encoder▫Decoder

•Experiment Result•Conclusion

Introduction

•Texture synthesis•Image/video inpainting

Introduction

Encoder

Decoder

[1] D. Liu, X. Sun, F. Wu, S. Li, and Y.-Q. Zhang, “Image compression with edge-based inpainting,” IEEE Transactions on Circuits and Systems for Video Technology, Oct. 2007.

Original image

Edge map Removed blocks

Structure propagation

Texture synthesis

JPEG

Introduction

•Texture regions can be well synthesized.▫Selectively removed during encoding.▫Restored by texture synthesis and edge-

based inpainting in the decoder.•Assistant information can be extracted to

guide restoration.▫Region types, edge information, motion

parameters. Keep spatio-temporal consistency

Framework—previous work

[2] C. Zhu, X. Sun, F. Wu, and H. Li, “Video coding with spatio-temporal texture synthesis,” IEEE International Conference on Multimedia and Expo, 2007.

I B PBB …

Framework—encoder

I B PBB …

Local Motion Region

Global Motion Region

Same as previous work

Framework—encoder

Structure blocks

Textureblocks

Block Categorization

Edge detection

Block categorizati

on

Exemplar selection

examplars

Check connective degreeremoved blocks do not constitute a large region

Framework—encoder

Motion Threading

[3] L. Luo, F. Wu, S. Li, Z. Xiong, and Z.Q. Zhuang, “Advanced motion threading for 3D wavelet video coding,” Signal Processing: Image Communication, Vol. 19, Issue 7, pp. 601-616, Aug. 2004.

•Spatial and temporal consistency▫Block-based backward motion estimation▫Motion threads in texture regions are pruned

B0 B2 B3 B4 B5B1

Exemplar Selection• All the motion threads are 3-D exemplar

candidates• Spatio-temporal variation(SVT)

N : length of a thread which consists of N blocks Bi(i=1…N) : spatio-temporal 6-neighboring blocks of each block : variance of a blockE() : mean pixel value of a block

Choose the threads with higher variation

Output a sequence of binary masks indicating which blocks are removed

Framework—encoder

Sprite Generation•GMR

▫pixel motion can be described by a parametric geometrical model

•Sprite (background mosaic)▫A sprite for a GOP

[4] Y. Lu, F. Wu, S. Li, and Y.-Q. Zhang, “Efficient background video coding with static sprite generation and arbitrary-shape spatial prediction techniques,” IEEE Transactions on Circuits and Systems for Video Technology, May 2003.

(x, y) and (x’,y’) are coordinates of pixels in original frame and reference image

Framework—encoder

removable blocksunremovabl

e blocks

Region removal in GMR

• Mapping into original frame inverse warping

• if one 8x8 block in an original frame contains more than half of the removable pixels, it is selected as a removable block and then skipped when encoding. Otherwise, it is selected as an exemplar and coded with H.264 encoder.

Edge Detectio

n

Region Classificatio

n

Mapping

Exemplar SelectionR

SRF

SPT

RS: Removable blocks in spriteRF: Removable blocks in original frame

Framework—decoderMask1: region mapMask2: removing map

Decoder—Spatio-temporal texture synthesis in LMR•Texture synthesis module

▫Arbitrary-shaped regions▫Perform frame by frame▫Square patch▫Confidence map

Similarity of candidate patch and target patch𝑆=𝑆𝑆𝐷 (𝑊 𝑡 ,𝑊𝑐 )+𝛼 ∙𝑆𝑆𝐷 (𝑊 𝑡 ′ ,𝑊 𝑐)

Squared difference of known pixels

Framework—decoder

Decoder—Edge-based inpainting in GMR

Warping

Blending

Edge inpaintin

g

Texture Synthesis

GMR

GMR parameter

SPT edges

SPTH

SPT

Experimental Results• Implemented into JM 10.2 of H.264• YUV 4:2:0 sequence format with CIF

resolution(352x288)• rate distortion optimization (RDO) and CABAC were

turned on

Experimental Results

Conclusion• In this paper, we present a video coding

scheme in which some regions are removed in the encoder and restored in the decoder by spatio-temporal texture synthesis and edge-based image inpainting.

• To keep temporal consistency, different types of motion have been considered in region removal and restoration for both textural and structural regions.