MRF-BASED TRUE MOTION ESTIMATION USING H.264 DECODING INFORMATION

Yung-Lin Huang, Yi-Nung Liu, and Shao-Yi Chien

Media IC and System Lab

Graduate Institute of Networking and Multimedia

National Taiwan University

  Signal Processing Systems (SIPS), 2010 IEEE

Outline

IntroductionMarkov Random Field

Motion Vector Analysis Motion Vector Pre-processing Predictor Selection Simplified Belief Propagation Experimental Results Conclusion

Introduction (1/4) Instead of heuristic approaches, TME can be

formulated as a pixel-labeling problem:

Introduction (2/4) Markov Random Field :

Given an undirected graph G = (V, E) A set of random variables(label) X = (Xv)v   ∈ V

Markov properties:

Introduction (3/4) Assigning each pixel a label, can be justified

in terms of maximum a-posterior estimation of a MRF model:

posterior likelihood * prior∝

With negative log probabilities, where the max-product becomes a min-sum.

The max-product algorithm can be used to find an approximate minimum cost labeling of energy functions.

Ed (the data term) & Es (the smoothness term)

Introduction (4/4) The cost energy function of a Markov Random Field

model to estimate the optimal labels { lp } of corresponding pixels:

Ed: the data term that measures the penalty between the labels and the data

Es: the smoothness term that penalizes the coherence between labels

P : the set of all pixelsN : the 4-nearest neighbor pixels

Motion Vector Analysis(1/3) Optical flow datasets are used here because

the ground truth (GT) MV maps are provided:

Check the existence of true MV by similarity:

The existence of TMV:W,H: the width, height of the test sequence

In Fig. 4. Both THx and THy are set to 1, and PSR ranges from 0 to 64.

Motion Vector Analysis(2/3)

The ME strategy(FastFS, FS or EPZS) has little effect on the experimental results.

There are still MVs with true motion trajectory in the H.264-coded MV field.

Motion Vector Analysis(3/3)

Proposed Algorithm

Motion Vector Pre-processing

In the proposed algorithm, the block size is fixed in each scale, so the MVs of variable block sizes must be split and merged.

The block merging method takes not only the macroblock types (from H.264) but also neighboring MVs into consideration.

Although the global optimization might modify these bad MVs, the pre-processing costs less efforts.

Proposed Algorithm

Predictor Selection(1/2)

In Fig. 4, the probability that true MV exists is high with enough PSR.

We choose PSR=32, when the block size is 16, the range of ±32 pixels ±2 blocks.

The strategy of predictor selection and the MRF model of the proposed algorithm are shown in Fig. 5(a). Nine predictors are selected.

Predictor Selection(2/2)

Proposed Algorithm

Simplified Belief Propagation (1/3) The multi-scale concept from [7],instead of

pixel-based operation, 4x4 block is taken as the smallest unit.

The belief propagation isoperated from the highest scale (16x16 block) to thelowest scale (4x4 block).

ft: video frame at time t

The basic concept of belief propagation is to perform message passing operation iteratively and approximate global minimum by local messages.

Each pixel requires O(k2) computation for full-search candidates.

The proposed algorithm requires only O(k) computation with predictor selection [7] for each pixel.

Simplified Belief Propagation (2/3)

Simplified Belief Propagation (3/3)[7] Pedro F. Felzenszwalb and Daniel P. Huttenlocher, “Efficient belief propagation for early vision,” Int. J. Comput.Vision, vol. 70, no. 1, pp. 41–54, 2006.

Loopy Belief Propagation approach for MRF:

Messages with the truncated linear model:

Time complexity: O(nk2T)n: the number of pixelsk: the number of possible labelsT: the number of iterations

Time complexity: O(k)n: the number of pixelsk: the number of possible labelsT: the number of iterations

Experimental Results Frame Rate Up Conversion compare with:

Bidirectional overlapped block motion estimation (OBME), MV field smoothing with median filter.

Proposed algorithm has higher PSNR aboutthe camera motion video (mobile calendar) because of the global MV field optimization.

OBME requires full search(FS) with an enlarged search range.

The proposed algorithm has relative lower computational complexity.

Motion Vector Field:

Experimental Results

Motion Vector Field:

Experimental Results

Conclusion

In this paper, a MRF-based true motion estimation obtained from H.264/AVC scheme is proposed.

The MV field of H.264/AVC is optimized using belief propagation efficiently.

In the future works, more reusable decoding information and hardware implementation will be involved.