![Page 1: Spatial-Temporal Consistency in Video Disparity Estimation](https://reader035.vdocuments.net/reader035/viewer/2022062520/56816230550346895dd262ac/html5/thumbnails/1.jpg)
Spatial-Temporal Consistency in Video Disparity Estimation
ICASSP 2011
Ramsin Khoshabeh , Stanley H. Chan,
Truong Q. Nguyen
![Page 2: Spatial-Temporal Consistency in Video Disparity Estimation](https://reader035.vdocuments.net/reader035/viewer/2022062520/56816230550346895dd262ac/html5/thumbnails/2.jpg)
Outline
• Introduction• Proposed Method– Image-based disparity map estimation– Temporal consistency
• Experimental Result
![Page 3: Spatial-Temporal Consistency in Video Disparity Estimation](https://reader035.vdocuments.net/reader035/viewer/2022062520/56816230550346895dd262ac/html5/thumbnails/3.jpg)
Introduction
• Stereo disparity estimation is an integral problem associated with 3D content delivery
• Two type of existing algorithm – Local – Global : minimizing energy function
• Even applying the best of existing methods to individual frames of stereo sequences yields temporally inconsistent disparity maps
(fast, lack the accuracy)
(slow)
![Page 4: Spatial-Temporal Consistency in Video Disparity Estimation](https://reader035.vdocuments.net/reader035/viewer/2022062520/56816230550346895dd262ac/html5/thumbnails/4.jpg)
Introduction
• The goal is to present a method to generate accurate and spatio-temporally consistent disparity maps from complex stereo video sequences.
![Page 5: Spatial-Temporal Consistency in Video Disparity Estimation](https://reader035.vdocuments.net/reader035/viewer/2022062520/56816230550346895dd262ac/html5/thumbnails/5.jpg)
Proposed Method
• Use Image-based technique– Video disparity problem in space-time is
computationally impractical– But we lose the consistency between consecutive
frames noisy• Improve the temporal consistency
![Page 6: Spatial-Temporal Consistency in Video Disparity Estimation](https://reader035.vdocuments.net/reader035/viewer/2022062520/56816230550346895dd262ac/html5/thumbnails/6.jpg)
Proposed MethodImage-based disparity map estimation
• In this step, disparity maps are computed for each frame individually.
• Use a global method using Hierarchical Belief Propagation (HBP) for inferencing.
• Energy function : P : set of pixels in an imageL : finite set of labelsA labeling f assigns a label fp ϵ L to each pixel p ϵ P
Data cost Discontinuity costHow well the labeling fit the node
![Page 7: Spatial-Temporal Consistency in Video Disparity Estimation](https://reader035.vdocuments.net/reader035/viewer/2022062520/56816230550346895dd262ac/html5/thumbnails/7.jpg)
Proposed MethodImage-based disparity map estimation
• Discontinuity cost enforces the assumption that labels should vary slowly.
• Except for significant changes along object boundaries
![Page 8: Spatial-Temporal Consistency in Video Disparity Estimation](https://reader035.vdocuments.net/reader035/viewer/2022062520/56816230550346895dd262ac/html5/thumbnails/8.jpg)
Proposed MethodImage-based disparity map estimation
• Data cost is computed over a large window for each pixel using locally adaptive support weights [11]
[11] K. J. Yoon and I. S. Kweon, “Locally Adaptive Support-Weight Approach for Visual Correspondence Search,” in CVPR, 2005.
Strength of grouping by similarity
Strength of grouping by proximity
: Color difference: Spatial distance
only points with a high probability of belonging to the same object contribute significantly to the cost calculation
![Page 9: Spatial-Temporal Consistency in Video Disparity Estimation](https://reader035.vdocuments.net/reader035/viewer/2022062520/56816230550346895dd262ac/html5/thumbnails/9.jpg)
Proposed MethodImage-based disparity map estimation
• use the method of [5] to minimize the energy over the entire image in a coarse-to-fine manner.
• Use the hierarchy to reduce the number of message passing iterations .
[5] P. Felzenszwalb and D. Huttenlocher, “Efficient Belief Propagation for Early Vision,” in CVPR, 2004, pp. 261–268.
![Page 10: Spatial-Temporal Consistency in Video Disparity Estimation](https://reader035.vdocuments.net/reader035/viewer/2022062520/56816230550346895dd262ac/html5/thumbnails/10.jpg)
Proposed Methodtemporal consistency
• Disparity should be a piecewise smooth function in time, except for discontinuities at object borders
• Consider the sequence of disparity maps as a space-time volume
x
yt
![Page 11: Spatial-Temporal Consistency in Video Disparity Estimation](https://reader035.vdocuments.net/reader035/viewer/2022062520/56816230550346895dd262ac/html5/thumbnails/11.jpg)
Proposed Methodtemporal consistency
TV-norm
Add (βx, βy, βt) so that we can control the relative emphasis
Total variation
Forward difference
[14] S. H. Chan, R. Khoshabeh, K. B. Gibson, P. E. Gill, and T. Q. Nguyen,“An augmented lagrangian method for total variation video restoration,” in ICASSP, May 2011
allows us to handle both spatial and temporal consistency simultaneously,by tuning the parameters (βx, βy, βt)
![Page 12: Spatial-Temporal Consistency in Video Disparity Estimation](https://reader035.vdocuments.net/reader035/viewer/2022062520/56816230550346895dd262ac/html5/thumbnails/12.jpg)
Experimental Result
Spatial noise Temporally inconsistencies
Remove errorPreserve object edges
![Page 13: Spatial-Temporal Consistency in Video Disparity Estimation](https://reader035.vdocuments.net/reader035/viewer/2022062520/56816230550346895dd262ac/html5/thumbnails/13.jpg)
Experimental Result
![Page 14: Spatial-Temporal Consistency in Video Disparity Estimation](https://reader035.vdocuments.net/reader035/viewer/2022062520/56816230550346895dd262ac/html5/thumbnails/14.jpg)
Experimental Result
• Add Gaussian noise to simulate real sequences• Bad pixel is defines as any pixel that has an estimated
disparity that |Dest – Dreal | > threshold(set at 1)
![Page 15: Spatial-Temporal Consistency in Video Disparity Estimation](https://reader035.vdocuments.net/reader035/viewer/2022062520/56816230550346895dd262ac/html5/thumbnails/15.jpg)
Experimental Result
Evaluate the efficacy of proposed TV method in improving arbitrary disparity estimates.