1

M15338: Depth Map Estimation

Software version 2

April, 27th 2008, Archamps

Olgierd StankiewiczKrzysztof Wegner

team supervisor: Marek DomańskiChair of Multimedia Telecommunications and Microelectronics

Poznan University of Technology, Poland

2

Outline Depth map quality measurement

Ground-truth map View resynthesis

View synthesis tool Depth map estimation tools

Belief Propagation based estimation Accuracy refinement by mid-level

hypothesis Summary

3

Depth map quality

Commonly used: ‘Bad-Pixels’

Miss information about error magnitude and energy

Requires ground-truth disparity map

thresholdyxdyxG

thresholdyxdyxGyxpixelbad

),(),(0

),(),(1),(

4

Depth map quality NBP-SAD (Normalized Bad Pixel SAD)

NBP-SSD (Normalized Bad Pixel SSD)

Still, requires ground-truth disparity map

pixelsbadyx

yxdyxGpixelsbadofcount

SADNBP,

),(),(1

pixelsbadyx

yxdyxGpixelsbadofcount

SSDNBP,

2),(),(

1

5

Depth map quality measurement by view resynthesis

End-user never sees depth-map Resynthesis

No standarized method Tool employs straight-forward method

PSNR (Peak Signal-to-Noise Ratio)of resynthesized view as quality measure

6

Bad-Pixels vs PSNRBad-pixels vs PSNR of resynthesis

HSABM+OF [1]ThreeViewBP [9]

AdaptingBP [4]

SubPixDoubleBP [6]

AdaptOvrSegBP [7]

PlaneFitBP [8]

Our proposal BP

Double BP[5]

SSD+MF [3]

29

30

31

32

33

34

35

36

0,00% 1,00% 2,00% 3,00% 4,00% 5,00% 6,00% 7,00% 8,00% 9,00%

Bad-pixels [%]

PS

NR

[d

B]

7

View synthesis tool

Simple and straight-forward For linearly positioned stereo pairs

only Two disparity maps and

corresponding reference views Weighting of pixels from side-views,

translated according to their disparity

8

View synthesis tool

9

Belief Propagation based depth estimation tool

Alternative for Hierarchical-Shape Adaptive Block Matching

Employs message passing for optimization of disparity map

hierarchical processing in layers

Pixel differences (1-point SAD) used as observations

10

Message passing in Belief Propagation

mt s→d – message passed in t-th iteration from node s to node d, V(fp,fq) – cost of belief change from disparity fp to disparity fq.

11

Message in Belief Propagation

Single message contains information about all possible disparities

12

Hierarchical processing in Belief Propagation

Higher resolution

Lower resolution

from the lowest resolution to the full resolution in

coarse-to-fine manner

13

Belief propagation

Vpq(xp,xq) – transition cost in node q between disparity xp and xq

insisted by nodeł p

Vp(xp) – observation in node p about disparity xp (SAD value)mpq(xq) – message from node p to q about disparity xq

14

Belief propagation

Pot model Simpleand computationally efficient .

Stable beliefs are prefered

15

Belief propagation results

1th iteration 20 iterations 300 iterations

Middlebury test results – 1,65% of bad-pixelsBest Middlebury algorithm – 0,88% of bad-pixels

16

Bad-Pixels vs PSNRBad-pixels vs PSNR of resynthesis

HSABM+OF [1]ThreeViewBP [9]

AdaptingBP [4]

SubPixDoubleBP [6]

AdaptOvrSegBP [7]

PlaneFitBP [8]

Our proposal BP

Double BP[5]

SSD+MF [3]

29

30

31

32

33

34

35

36

0,00% 1,00% 2,00% 3,00% 4,00% 5,00% 6,00% 7,00% 8,00% 9,00%

Bad-pixels [%]

PS

NR

[d

B]

17

Accuracy refinement by mid-level hypothesis

Low computational cost Improves accuracy of disparity

map (number of disparity levels) Spatial resolution unchanged Focuses on unit-step edges in

disparity map

18

Unit-step edges

19

Mid-level hypothesis

Hypothesis spread along unit-step edges

20

Refinement by mid-level hypothesis

Pixel accurate disparity (1x) After refinement (8x)

21

Works over untextured regions

22

ResultsChange of bad-pixels relative to (x1) during iterative refinement

0,00%

20,00%

40,00%

60,00%

80,00%

100,00%

120,00%

x1 x2 x4 x8 x16

Bad

-pix

els

chan

ge

23

Summary

New version of experimental Depth Estimation software

Quality measurement problem with respect to multi-view applications

Simple view resynthesis tool Belief Propagation depth estimation

tool Novel technique for accuracy

refinement