the university of ontario cs 4487/9687 algorithms for image analysis multi-label image analysis...
TRANSCRIPT
The University of
Ontario
CS 4487/9687
Algorithms for Image Analysis
Multi-Label Image Analysis Problems
The University of
Ontario
CS 4487/9687 Algorithms for Image Analysis Multi-label image analysis problems
Topic 1 From binary to multi-label problems:• Stereo, image restoration, texture synthesis, multi-object
segmentation• Ishikawa’s algorithm, total variation
Topic 2 Types of pair-wise pixel interactions• Convex interactions • Discontinuity preserving interactions
Topic 3 Energy minimization algorithms: • simulated annealing, ICM,• a-expansions
Extra Reading: …
The University of
Ontario
Graph cuts algorithms can minimize multi-label energies as well
The University of
Ontario
Multi-scan-line stereo with s-t graph cuts (Roy&Cox’98)
x
y
The University of
Ontario
Multi-scan-line stereo with s-t graph cuts (Roy&Cox’98)
s
t cut
L(p)
p
“cut”
x
y
labels
x
y
Dis
pari
ty lab
els
The University of
Ontario
s-t graph-cuts for multi-label energy minimization
Ishikawa 1998, 2000, 2003 Modification of construction by Roy&Cox 1998
V(dL)
dL=Lp-Lq
V(dL)
dL=Lp-Lq
Linear interactions “Convex” interactions
Npq
qpp
pp LLVLDLE ),()()( 1RLp
The University of
Ontario
Pixel interactions V:“convex” vs. “discontinuity-preserving”
V(dL)
dL=Lp-Lq
Potts model
Robust“discontinuity preserving”
Interactions V
V(dL)
dL=Lp-Lq
“Convex”Interactions V
V(dL)
dL=Lp-Lq
V(dL)
dL=Lp-Lq
“linear”
model
The University of
Ontario
Pixel interactions:“convex” vs. “discontinuity-preserving”
“linear” V
truncated “linear” V
The University of
Ontario
code
Robust interactions
NP-hard problem (3 or more labels) • two labels can be solved via s-t cuts
a-expansion approximation algorithm (Boykov, Veksler, Zabih 1998, 2001)
• guaranteed approximation quality (Veksler, 2001)– within a factor of 2 from the global minima (Potts model)
Many other (small or large) move making algorithms- a/b swap, jump moves, range moves, fusion moves, etc.
LP relaxations, message passing, e.g. (LBP, TRWS) Other MRF techniques (simulated annealing, ICM) Variational methods (e.g. multi-phase level-sets)
The University of
Ontario
other labelsa
a-expansion move
Basic idea is motivated by methods for multi-way cut problem
(similar to Potts model)
Break computation into a sequence of binary s-t cuts
The University of
Ontario
),( qppq SSE)( pp SE
a-expansion (binary move)optimizies sumbodular set function
expansions correspond to subsets
(shaded area)
S
Npq
qppqp
pp LLELESLESE)(
),()()()(ˆ
L current labeling
}{ pLp|
ppppp SLSSL )(
=
pS1
The University of
Ontario
),( qppq SSE)( pp SE
a-expansion (binary move)optimizies sumbodular set function
L current labeling
}{ pLp|
pS 0 1
)(pE)( pp LE
Npq
qppqp
pp LLELESLESE)(
),()()()(ˆ
The University of
Ontario
),( qppq SSE)( pp SE
a-expansion (binary move)optimizies sumbodular set function
L current labeling
}{ pLp|
pS 0 1
)( ,pqE
),( qppq LLE
qS
0
1
),( qpq LE
),( ppq LE
Npq
qppqp
pp LLELESLESE)(
),()()()(ˆ
The University of
Ontario
a-expansion (binary move)optimizies sumbodular set function
L current labeling
}{ pLp|
pS 0 1
)( ,pqE
),( qppq LLE
qS
0
1
),( qpq LE
),( ppq LE
)(SE
(1,0)(0,1)(0,0)(1,1) pqpqpqpq EEEE ˆˆˆˆ
Set function is submodular if
The University of
Ontario
a-expansion (binary move)optimizies sumbodular set function
L current labeling
}{ pLp|
pS 0 1
)( ,pqE
),( qppq LLE
qS
0
1
),( qpq LE
),( ppq LE
)(SE
),(),(),(),( qpqppqqppqpq LELELLEE Set function is submodular if
=
0 triangular inequality for ||a-b||=E(a,b)
The University of
Ontario
a-expansion (binary move)optimizies sumbodular set function
L current labeling
}{ pLp|
pS 0 1
)( ,pqE
),( qppq LLE
qS
0
1
),( qpq LE
),( ppq LE
),( baEpq
a-expansion moves are submodular if is a metric on the space of labels
[Boykov, Veksler, Zabih, PAMI 2001]
The University of
Ontarioa-expansion algorithm
1. Start with any initial solution2. For each label “a” in any (e.g. random) order
1. Compute optimal a-expansion move (s-t graph cuts)
2. Decline the move if there is no energy decrease
3. Stop when no expansion move would decrease energy
The University of
Ontarioa-expansion moves
initial solution
-expansion
-expansion
-expansion
-expansion
-expansion
-expansion
-expansion
In each a-expansion a given label “a” grabs space from other labels
For each move we choose expansion that gives the largest decrease in the energy: binary optimization problem
The University of
OntarioMulti-way graph cuts
stereo vision
original pair of “stereo” images
depth map
ground truthBVZ 1998KZ 2002
The University of
Ontario
normalized correlation,start for annealing, 24.7% err
simulated annealing, 19 hours, 20.3% err
a-expansions (BVZ 89,01)90 seconds, 5.8% err
0
20000
40000
60000
80000
100000
1 10 100 1000 10000 100000
Time in seconds
Sm
oo
thn
ess E
nerg
y
Annealing Our method
a-expansions vs. simulated annealing
The University of
Ontario
a-expansions:examples of metric interactions
Potts V
“noisy diamond”“noisy shaded diamond”
][, V
,V
Truncated “linear” V
The University of
OntarioMulti-way graph cuts
Graph-cut textures (Kwatra, Schodl, Essa, Bobick 2003)
similar to “image-quilting” (Efros & Freeman, 2001)
AB
C D
EF G
H I J
A B
G
DC
F
H I J
E
The University of
OntarioMulti-way graph cuts
Graph-cut textures (Kwatra, Schodl, Essa, Bobick 2003)
The University of
OntarioMulti-way graph cuts
Multi-object Extraction
Obvious generalization of binary object extraction technique(Boykov, Jolly, Funkalea 2004)
The University of
Ontario
Block-coordinate descent alternating a-expansion (for segmentation L) and
fitting colors Ii
Chan-Vese segmentation(multi-label case)
...)()(,...),,(1:
21
0:
2010
pp Lp
pLp
p IIIIIILE
Npq
qppq LLw}{
][ Potts model
The University of
Ontario
Chan-Vese segmentation(multi-label case)
Block-coordinate descent alternating a-expansion (for segmentation L) and
fitting colors Ii
...)()(,...),,(1:
21
0:
2010
pp Lp
pLp
p IIIIIILE
Npq
qppq LLw}{
][ Potts model
The University of
Ontario
Block-coordinate descent alternating a-expansion (for segmentation L) and
fitting colors Ii
Stereo via piece-wise constant plane fitting [Birchfield &Tomasi 1999]
Models T = parameters of affine transformations T(p)=a p + b
...)()(,...),,(1:
2)(
0:
2)(
10
10
pp Lp
ppTLp
ppT IIIITTLE
2x2 2x1
Npq
qppq LLw}{
][ Potts model
The University of
Ontario
Block-coordinate descent alternating a-expansion (for segmentation L) and
fitting colors Ii
Piece-wise smooth local plane fitting[Olsson et al. 2013]
...)()(,...),,(1:
2)(
0:
2)(
10
10
pp Lp
ppTLp
ppT IIIITTLE
Npq
qp LLw}{
truncated angle-differences
non-metric interactionsneed other optimization
The University of
Ontario
Block-coordinate descent alternating a-expansion (for segmentation L) and
fitting colors Ii
Signboard segmentation[Milevsky 2013]
Labels = planes in RGBXY space C(p) = a x + b
...))(())((,...),,(1:
21
0:
20
10
pp Lp
pLp
p IpCIpCCCLE
Npq
qppq LLw}{
][ Potts model
3x2 3x1
The University of
Ontario
Signboard segmentation[Milevsky 2013]
3x2 3x1Goal: detection of characters, then text line fitting and translation
The University of
OntarioMulti-label optimization
80% of computer vision and bio-medical image analysis are ill-posed labeling problems requiring optimization of regularization energies E(L)
Most problems are NP hard Optimization algorithms is area of active research
• Google, Microsoft, GE, Siemens, Adobe, etc.• LP relaxations [Schlezinger, Komodakis, Kolmogorov, Savchinsky,…]
• Message passing, e.g. LBP, TRWS [Kolmogorov]
• Graph Cuts (a-expanson, a/b-swap, fusion, FTR, etc) • Variational methods