the university of ontario cs 4487/6587 algorithms for image analysis correspondence
TRANSCRIPT
The University of
Ontario
CS 4487/6587
Algorithms for Image Analysis
Correspondence
The University of
Ontario
CS 4487/6587 Algorithms for Image Analysis Correspondence
Matching and Correspondence problems Stereo, motion, non-rigid registration,… From 1-dimensional to N-dimensional
optimization on graphs• Graph cuts vs. DP and Shortest Paths
Extra Reading: Cormen et.al. “Introduction to Algorithms”, Section 26
The University of
OntarioTemplate matching
• In matching we estimate “position” of a rigid template in the image
• “Position” includes global location parameters of a rigid template:
- translation, rotation, scale,…
Face templateimage
image
The University of
OntarioFlexible template matching
• In flexible template matching we estimate “position” of each rigid component of a template
The University of
Ontario
Term “correspondence” is normally used for “matching” of low-level features
Matching between two images on a pixel level• Each pixel has to find a corresponding one• Examples: non-rigid registration, stereo
More generally, matching on feature level• E.g. each model feature has to establish one
“corresponding” feature in the observed image
The University of
Ontario
More examples of correspondence:non-rigid image registration
Often need to find exact correspondences between image pixels
- All such correspondences specify non-rigid transformation (registration) between two images
p = (825,856)
q = (912,632)
q = T(p;a)
The University of
Ontario
More examples of correspondence:required in image morphing
How can we specify the warp?- First specify corresponding points- Then a morph can be computed (possibly later)
The University of
Ontario
More examples of correspondence:Motion (tracking)
beating heart
We have to establish correspondence between specific pointson the object boundary from frame to frame
- correspondences are visualized by
The University of
Ontario
More examples of correspondence:Stereo
knowncamera
viewpoints
Two views of the same scene from slightly different point of view
The University of
Ontario
Right imageLeft image
More examples of correspondence: Stereo
• Both images are very similar (like images that you see with your two eyes)
• all pixels in the left image are present in the right image (except for “few” occlusions) • Normally, images are horizontally aligned so that objects (pixels) in one image simply shift horizontally in the other image
The University of
Ontario
More examples of correspondence: Stereo
(After calibration) all correspondences are along the same horizontal scan lines
Right imageLeft image
The University of
OntarioStereo
Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923
• Correspondences are described by horizontal shifts (in x direction only)• Such shifts are also referred to as “disparities”
The University of
OntarioStereo
• If x-shifts (disparities) are known for all pixels in the left (or right) image then we can visualize them as a disparity map• Normally disparity relates to object depth (the larger the shift the closer the object )
Disparity map(Depth map)
Right imageLeft image
The University of
OntarioStereo Correspondence problem
Human vision can solve it(even for “random dot” stereograms)
Can computer vision solve it?
Maybe
see Middlebury Stereo Databasefor the state-of-the art results
http://cat.middlebury.edu/stereo/
The University of
OntarioStereo
Window based • Matching rigid windows around each pixel• Each window is matched independently
Scan-line based approach• Finding coherent correspondences for each scan-line• Scan-lines are independent
– DP, shortest paths
Muti-scan-line approach• Finding coherent correspondences for all pixels
– Graph cuts
The University of
Ontario
Right imageLeft image
Typical window cost function
Location with the best cost wins
C2 C3C1
f(x,y)x,y
Stereo Correspondence problemWindow based approach
The University of
OntarioFixed Windows Efficient implementation with “sliding column”,
Faugeras et. al.’1993 for window cost function
Running time is independent of window size
- +
f(x,y)x,y
The University of
OntarioProblems with Fixed Windows
small window large window
better at boundaries noisy in low texture areas
better in low texture areas blurred boundaries
The University of
OntarioVariable window size algorithms
Correspondences are still found independently at each pixel
All Window-based solution can be though of as “local” solutions• Fast
How to introduce spatial coherence• Introduce Energy function• Searching for “global” solutions
The University of
Ontario
Scan-line stereo • Try to match coherently pixels in each scan line• DP or shortest paths can be used (1D optimization)• Note: scan lines are still matched independently
– Streaking artifacts
Stereo Correspondence problemScan-line approach
The University of
Ontario
“Shortest paths” for Scan-line stereo
Left image
Right image
Actual implementation in OK’85 and C’96 uses DP algorithm
leftS
rightS
correspondence
q
p
Left
occ
lusi
on
t
Rightocclusion
s
e.g. Ohta&Kanade’85, Cox at.al.’96
2)( qp II
occlC
occlC
I
I
The University of
OntarioDP for scan-line stereo
Left image
Right image
rightSpleftS
2pd
pdp
Nqp
qp ddE},{
),(
Viterbi can handle this on non-loopy
graphs (e.g., scan-lines)
||pdpp II || 1 pp dd
= =
spatial coherencephoto consistency
I
I
Sp
ppSp
pp ddVdDE ),()()( 1d
}|{ Spd p drightS
Viterbi algorithm can be used to optimize the following energy of disparities of pixels p on a fixed scan-line
The University of
OntarioCoherent stereo on 2D grid?
Scan-line stereo generates streaking artifacts Can’t use DP (or Dijkstra) to find spatially
coherent disparities/correspondances on a 2D grid
Now we will learn about graph cuts algorithm
The University of
Ontario
Graph cuts for spatially coherent stereo on 2D grids
Consider a 2D grid G corresponding to
(right) image pixels
Nqp
qpGp
pp ddVdDE},{
),()()(d
||pdpp II || qppq ddw
= =
spatial coherencephoto consistency
We will globally minimize the same energy of disparities for pixels p on a grid G
}|{ Gpd p d
pqw weights of neighborhood
edges, n-links, may reflect (right)
image intensity gradients (static
cues)
The University of
Ontario
Multi-scan-line stereo with s-t graph cuts (Roy&Cox’98)
x
y
The University of
Ontario
Multi-scan-line stereo with s-t graph cuts (Roy&Cox’98)
s
t cut
L(p)
p
“cut”
x
y
labels
x
y
Dis
pari
ty lab
els
The University of
Ontario
Ishikawa&Geiger 98 What energy do we minimize this way?
Concentrate on one pair of neighboring pixels Nqp },{
p q
)5()2(),( qpqp DDddE
|3|pqw
cost of horizontal edges
cost of vertical edges
Dis
pari
ty lab
els
- 0
- 1
- 2
- 3
- 4
- 5
The University of
Ontario
Ishikawa&Geiger 98 What energy do we minimize this way?
Concentrate on one pair of neighboring pixels Nqp },{
)()(),( qqppqp dDdDddE
|| qppq ddw
cost of horizontal edges
cost of vertical edgesp q
Dis
pari
ty lab
els
- 0
- 1
- 2
- 3
- 4
- 5
The University of
Ontario
Ishikawa&Geiger 98 What energy do we minimize this way?
The combined energy over the entire grid G is
Gp
pp dDE )()(d
Nqp
qppq ddw},{
||
cost of horizontal edges(spatial consistency)
(photo consistency)cost of vertical edges
s
t cut
The University of
Ontario
Scan-line stereo vs. Multi-scan-line stereo
L(p)
p
“cut”
x
y
labels
labels
x
L(p)
p
Dynamic Programming(single scan line optimization)
s-t Graph Cuts(multi-scan-line optimization)
The University of
Ontario
Graph Cuts Basics (see Cormen’s book)Simple 2D example
s-t graph cut
A graph with two terminals S and T
“source”
S T
“sink”
• Cut cost is a sum of severed edge weights • Minimum cost s-t cut can be found in polynomial time
Goal: divide the graph into two parts separating red and blue nodes
The University of
Ontario
32
Graph cut is an old standard problemwith tons of applications outside vision
From Harris & Ross [1955]
The University of
Ontario
33
From Harris & Ross [1955]
Graph cut is an old standard problemwith tons of applications outside vision
The University of
Ontario
s/t min cut algorithms are widely studied (combinatorial optimization)
Augmenting paths [Ford & Fulkerson, 1962]
Push-relabel [Goldberg-Tarjan, 1986]
The University of
Ontario“Augmenting Paths”
Find a path from S to T along non-saturated edges
“source”
A graph with two terminals
S T
“sink” Increase flow along
this path until some edge saturates
The University of
Ontario“Augmenting Paths”
Find a path from S to T along non-saturated edges
“source”
A graph with two terminals
S T
“sink” Increase flow along
this path until some edge saturates
Find next path… Increase flow…
The University of
Ontario“Augmenting Paths”
Find a path from S to T along non-saturated edges
“source”
A graph with two terminals
S T
“sink” Increase flow along
this path until some edge saturates
Iterate until … all paths from S to T have at least one saturated edge
MAX FLOW MIN CUT
The University of
OntarioSome results from Roy&Cox
s
tminimum
cost s/t cut
multi scan line stereo(graph cuts)
single scan-line stereo (DP)
The University of
OntarioSome results from Roy&Cox
multi scan line stereo(graph cuts)
single scan-line stereo (DP)
s
tminimum
cost s/t cut
The University of
Ontario
Simple Examples:Stereo with only 2 depth layers
http://research.microsoft.com/vision/cambridge/i2i/bgsubstitution.html
i2i projects from
Microsoft ResearchCVPR 2005
Background Substitution
The University of
Ontario
Simple Examples:Stereo with only 2 depth layers
http://research.microsoft.com/vision/cambridge/i2i/default.html
i2i projects from
Microsoft ResearchCVPR 2005
Other Applications
The University of
Ontario
HW Assignment 3Stereo with only 2 depth layers Equivalent to Foreground/Background binary image segmentation Take one stereo pair
• first frame from from i2i video sequence (see “image samples”)• You can use MyImage library to read the images.
Use graph cuts to detect foreground and background layers in the right image.• Max-flow/min-cut graph library is provided (see “links”)• Foreground (s), e.g. includes large disparities 5,6,7,8,9,10• Background (t), e.g. includes small disparities 0,1,2,3,4
a cut
)(sDp
)(tDp
s
t
Nqp
qppqGp
pp llwlDE},{
||)(
},{ tslp ||min)(
105dpp
dp IIsD
||min)(40
dppd
p IItD