inferring the cyclopean image · 2018. 1. 29. · inferring the cyclopean image a forward-backward...
TRANSCRIPT
1
Trajectory estimation problems in vision
–temporal tracking• deterministic: Kalman Filter• stochastic: Particle Filter
–curve tracing• deterministic: LiveWire (Mortensen&Barrett, 95)• stochastic: JetStream (Perez et al. 01)
–stereo vision• deterministic: Dynamic Programming• stochastic: ??
Inferring the cyclopean image A forward-backward algorithm for stereo matching
• Stereo vision– seeing in depth
– cyclopean vision (cf. double vision)
• Applications to – teleconferencing
– virtual reality
collaborators: P. Torr, I.Cox, A. Criminisi
2
Virtual reality
(Anandan, Criminisi, Kang, Szeliski, Uyttendale, Microsoft Research)
Teleconferencing: failure of eye contact
screencamera
camera
3
Problem: estimate cyclopean intensity
Estimate parallax:
Cyclopean image:
L(x+d) = R(x)Parallax:
?? Occlusion?? Prior on d i) Epipolarity ii) Ordering
Epipolargeometry
(tutorial by Andrea Fusiello )
4
Stereo disparity on epipolar lines
Epipolar matchingas optimal path finding
m
n
1 2 3 4
1
2
3
4zero parallax
Mapping:
where
minimisingFind (Dijkstra)
(Ohta & Kanade, 1985;Cox, Hingorani & Rao, 1996)
5
Ordering constraint
m
n
Nail illusion
Gaze correction: virtual cyclopean camera
Dijkstra solution (10 fps)
6
Robust virtual cyclopean camera
• Family of paths
• Dijkstra framework won’ t do it
• Viterbi framework could work.
JetStream: particle filter(Perez et al.)
7
Stereo disparity in cyclopean coordinates
Cyclopean Coordinates
m
nk
d
1 2 3 4
1
2
3
4
12
3
-1-2
-3
2
4
6
8
Warp:
-- even or odd
8
Posterior distribution for cyclopean image
*
Obtain marginals for d--- check decomposition of p(z|I,d) and P(d)
*
-- robust estimate
Observation modelObservation density:
Generative form(matched)
Marginalise (step 1)
indept. Gaussians
(improper Gaussian)
uniform
Check decomposition of p(z|I,d) (for step 3)
9
Disparity prior
Decomposition (for step 2)
For example:
eveneven
... Disparity prior
m
n k
1
1-2q
q
q
eveneven
possibly occluded
10
Conventional forward algorithm?Time-stamped observations
Forward probabilities
Forward algorithm
But observations are not time-stamped – two streams:
observation history?
1 2 3 4
1
2
3
4
History for stereo observations
‘Past’ depends not only on time k but also on value dk
m
nk
d
11
Forward and backward probabilities
Define past observations:
and future observations:
(even hk)
Forward probabilities:
Backward probabilities:
(odd hk)
Forward step
Recall observation density
Forward step
for1 otherwise
12
...Forward step
m
n k
f(..)
1-2q
q
q
... Posterior distribution for cyclopean image
� Assume pointwise loss, so require only
for each
13
... Posterior distribution for cyclopean image
so
Posterior expectations of cyclopean intensity
m
n k
d
14
DijkstraFB
DijkstraFB
15
16
Dijkstra: c=0.005 FB: c=0.005, q=0.1 FB: c=0.005, q=0.02 FB: c=0.005, q=0.3
FB: disparity distributions
row 87row 90
disparity
17
FB vs. Dijkstracf. ground truth
Haloing around occlusions
18
Substituting background into occlusions
Cyclopean Smoothing --- open questions
• Non-cyclopean virtual views
• Regression to fronto-parallel
• Explicit occlusion labels/process
• Intra-scan-line constraints
• Robust intensity estimators
19
Right viewLeft view
(data: Scharstein & Szeliski 2002)
Labelling of occlusion
Occluded regions Disparity
End