efficient inference for fully-connected crfs with stationarity
DESCRIPTION
Efficient Inference for Fully-Connected CRFs with Stationarity. Yimeng Zhang, Tsuhan Chen CVPR 2012. Summary. Explore object-class segmentation with fully-connected CRF models Only restriction on pairwise terms is `spatial stationarity ’ (i.e. depend on relative locations) - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Efficient Inference for Fully-Connected CRFs with Stationarity](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681642e550346895dd5f6b7/html5/thumbnails/1.jpg)
Efficient Inference for Fully-Connected CRFs with Stationarity
Yimeng Zhang, Tsuhan ChenCVPR 2012
![Page 2: Efficient Inference for Fully-Connected CRFs with Stationarity](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681642e550346895dd5f6b7/html5/thumbnails/2.jpg)
Summary
• Explore object-class segmentation with fully-connected CRF models
• Only restriction on pairwise terms is `spatial stationarity’ (i.e. depend on relative locations)
• Show how efficient inference can be achieved by– Using a QP formulation– Using FFT to calculate gradients in complexity
(linear in) O(NlogN)
![Page 3: Efficient Inference for Fully-Connected CRFs with Stationarity](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681642e550346895dd5f6b7/html5/thumbnails/3.jpg)
Fully-connected CRF model
• General pairwise CRF model:
• Image I• Class labeling, X:• Label set, L: • V = set of pixels, N_i = neighbourhood of pixel i,
Z(I) = partition function, psi = potential functions
![Page 4: Efficient Inference for Fully-Connected CRFs with Stationarity](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681642e550346895dd5f6b7/html5/thumbnails/4.jpg)
Fully-connected CRF model
• General pairwise CRF model:
• In fully-connected CRF, for all i, N_i = V
![Page 5: Efficient Inference for Fully-Connected CRFs with Stationarity](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681642e550346895dd5f6b7/html5/thumbnails/5.jpg)
Unary Potential
• Unary potential generates a score for each object class per pixel (TextonBoost)
![Page 6: Efficient Inference for Fully-Connected CRFs with Stationarity](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681642e550346895dd5f6b7/html5/thumbnails/6.jpg)
Pairwise Potential
• Pairwise potential measures compatibility of the labels at each pair of pixels
• Combines spatial and colour contrast factors
![Page 7: Efficient Inference for Fully-Connected CRFs with Stationarity](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681642e550346895dd5f6b7/html5/thumbnails/7.jpg)
Pairwise Potential
• Colour contrast:
• Spatial term:
![Page 8: Efficient Inference for Fully-Connected CRFs with Stationarity](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681642e550346895dd5f6b7/html5/thumbnails/8.jpg)
Pairwise Potential
• Learning the spatial term
![Page 9: Efficient Inference for Fully-Connected CRFs with Stationarity](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681642e550346895dd5f6b7/html5/thumbnails/9.jpg)
MAP inference using QP relaxation• Introduce a binary indicator variable for each
pixel and label
• MAP inference expressed as a quadratic integer program, and relaxed to give the QP
![Page 10: Efficient Inference for Fully-Connected CRFs with Stationarity](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681642e550346895dd5f6b7/html5/thumbnails/10.jpg)
MAP inference using QP relaxation
• QP relaxation has been proved to be tight in all cases (Ravikumar ICML 2006 [24])
• Moreover, it is convex whenever matrix of edge-weights is negative-definite
• Additive bound for non-convex case• QP requires O(KN) variables, LP requires (K^2E)
![Page 11: Efficient Inference for Fully-Connected CRFs with Stationarity](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681642e550346895dd5f6b7/html5/thumbnails/11.jpg)
MAP inference using QP relaxation
• Gradient
• Derive fixed-point update by forming Lagrangian and setting its derivative to 0
![Page 12: Efficient Inference for Fully-Connected CRFs with Stationarity](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681642e550346895dd5f6b7/html5/thumbnails/12.jpg)
Illustration of QP updates
![Page 13: Efficient Inference for Fully-Connected CRFs with Stationarity](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681642e550346895dd5f6b7/html5/thumbnails/13.jpg)
Efficiently evaluating the gradient
• Required summation
• Would be a convolution without the color term• With color term is requires 5D-filtering• Can be approximated by clustering into C color
clusters, => C convolutions across
![Page 14: Efficient Inference for Fully-Connected CRFs with Stationarity](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681642e550346895dd5f6b7/html5/thumbnails/14.jpg)
Efficiently evaluating the gradient
• Hence, for the case x_i = x_j, we need to evaluate
• Instead, evaluate for C clusters (C = 10 to 15)
• where
• Finally, interpolate
![Page 15: Efficient Inference for Fully-Connected CRFs with Stationarity](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681642e550346895dd5f6b7/html5/thumbnails/15.jpg)
Update complexity• FFTs of each spatial filters can be calculated in
advance (K^2 filters)• At each update, we require C FFTs calculating,
O(CNlogN)• K^2 convolutions are needed, each requiring a
multiplication, O(K^2CN)• Terms can be added in Fourier domain, => only KC inverse FFTs needed, O(KCNlogN)
• Run-time per iteration < 0.1s for 213x320 pixels (+ downsampling by factor of 5)
![Page 16: Efficient Inference for Fully-Connected CRFs with Stationarity](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681642e550346895dd5f6b7/html5/thumbnails/16.jpg)
MSRC synthetic experiment
• Unary terms randomized• Spatial distributions set to ground-truth
![Page 17: Efficient Inference for Fully-Connected CRFs with Stationarity](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681642e550346895dd5f6b7/html5/thumbnails/17.jpg)
MSRC synthetic experiment
• Running times
![Page 18: Efficient Inference for Fully-Connected CRFs with Stationarity](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681642e550346895dd5f6b7/html5/thumbnails/18.jpg)
Sowerby synthetic experiment
![Page 19: Efficient Inference for Fully-Connected CRFs with Stationarity](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681642e550346895dd5f6b7/html5/thumbnails/19.jpg)
MSRC full experiment
• Use TextonBoost unary potentials• Compare with several other CRFs with same
unaries– Grid only– Grid + P^N (Kohli, CVPR 2008)– Grid + P^N + Cooccurrence (Ladickỳ, ECCV 2010)– Fully-connected + Gaussian spatial (Krähenbühl,
NIPS 2011)
![Page 20: Efficient Inference for Fully-Connected CRFs with Stationarity](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681642e550346895dd5f6b7/html5/thumbnails/20.jpg)
MSRC full experiment
• Qualitative comparison
![Page 21: Efficient Inference for Fully-Connected CRFs with Stationarity](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681642e550346895dd5f6b7/html5/thumbnails/21.jpg)
MSRC full experiment
• Quantitative comparison– Overall
– Per-class
– Timing: 2-8s per image