Robust Higher Order Potentials Robust Higher Order Potentials For Enforcing Label ConsistencyFor Enforcing Label Consistency

Pushmeet KohliPushmeet KohliMicrosoft Research CambridgeMicrosoft Research Cambridge

Lubor LadickyLubor Ladicky Philip TorrPhilip TorrOxford Brookes University, OxfordOxford Brookes University, Oxford

CVPR 2008CVPR 2008

Image labelling Problems

Image Denoising

Geometry Estimation

Object Segmentation

Assign a label to each image Assign a label to each image pixelpixel

Building

Sky

Tree Grass

Object Segmentation using CRFs

Unary potentials based on

Colour, Location and Texture

features

CRF Energy

(Shotton et al. ECCV 2006)

Encourages label consistency in adjacent pixels

Limitations of Pairwise CRFs

Image

Unary Potential

MAP-CRF

Solution

• Encourages short boundaries (Shrinkage bias)

• Can only enforce label consistency in adjacent pixels

• Inability to incorporate region based features

Label Consistency in Image Regions

• Pixels constituting some regions belong to- Same plane (Orientation)

(Hoiem, Efros, & Herbert, ICCV’05)- Same object

(Russel, Efros, Sivic, Freeman, & Zisserman, CVPR06)

Image (MSRC) Segmentation (Mean shift)

Image labelling using segments

Image Object Labelling

UnsupervisedSegmentation

• Geometric Context [Hoiem et al, ICCV05]

• Object Segmentation[He et al. ECCV06, Yang et al. CVPR07, Rabinovich et al. ICCV07, Batra et al. CVPR08]

• Interactive Video Segmentation[Wang, SIGGRAPH 2005 ]

Not robust to Inconsistent Segments!

Our Higher Order CRF Model

cMultiple Segmentation

s

Encourages label consistency in

regions

Label Consistency in Segments

• Encourages consistency within super-pixels

• Takes the form of a PN Potts model[Kohli et al. CVPR 2007]

c

Label Consistency in Segments

Cost: 0

c

• Encourages consistency within super-pixels

• Takes the form of a PN Potts model[Kohli et al. CVPR 2007]

Label Consistency in Segments

Cost: f (|c|)

c

• Encourages consistency within super-pixels

• Takes the form of a PN Potts model[Kohli et al. CVPR 2007]

• Encourages consistency within super-pixels

• Takes the form of a PN Potts model[Kohli et al. CVPR 2007]

Label Consistency in Segments

Cost: f (|c|)

c

Does not distinguish between Good/Bad

Segments !

Quality based Label Consistency

How to measure quality G(c)?[Ren and Malik ICCV03, Rabinovich et al. ICCV07, many others]

• Colour and Texture Similarity• Contour Energy

Label inconsistency cost depends on segment quality

Measure quality from variance in feature responses

Higher order generalization of contrast-sensitive pairwise potential

Quality based Label Consistency

MSRC imageMean shift

segmentation

Segment Quality(darker is better)

Robust Consistency Potentials

Inconsistent Pixels

max

0 1

PN Potts

Too Rigid!

max

0 1 T

Robust

Inconsistent Pixels

Robust Consistency Potentials

Number of Inconsistent

Pixels

SlopeMaximum

Inconsistency Cost

max

0 1 T

Robust

Inconsistent Pixels

Minimizing Higher order Energy Functions

• Message passing is computationally expensive– High runtime and space complexity - O(LN) – L = Number of Labels, N = Size of Clique

• Efficient BP for Higher Order MRFs [Lan et al. ECCV 06, Potetz CVPR 2007]

– 2x2 clique potentials for Image Denoising – Take minutes per iteration (Hours to

converge)

Minimizing Higher order Energy Functions

• Graph Cut based move making algorithm[Kohli et al. CVPR 2007]

– Can handle very high order energy functions – Extremely efficient: computation time in the order of

seconds– Only applicable to some classes of functions (PN Potts)– Cannot handle robust consistency potential

• This paper– Can minimize a much larger class of higher order energy

functions– Same time complexity as [Kohli et al. CVPR 2007]

Move making algorithms

Expansion and Swap move algorithms[Boykov Veksler and Zabih, PAMI 2001]

• Makes a series of changes to the solution (moves)• Each move results in a solution with smaller energy

Current SolutionCurrent Solution

Generate pseudo-boolean move

function

Generate pseudo-boolean move

function

Minimize move function to get optimal move

Minimize move function to get optimal move

Move to new

solution

Move to new

solution

How to minimize

move functions?

Minimizing Move Functions using Graph Cuts

st-mincut

(Positive weights)

st-mincut

(Positive weights)

Second orderPseudo-boolean

Function Minimization (submodular)

Second orderPseudo-boolean

Function Minimization (submodular)

Optimal moves can be found extremely efficiently using graphs cuts

Most pairwise CRF models used in Computer Vision lead to submodular

move functions

Minimizing Higher Order Energy Functions: Our results

• Result 1– We show that a large class of higher order potentials

lead to higher order submodular move functions– Can be minimized in polynomial time

• Submodular Function Minimization– Minimizing general submodular functions is

computationally expensive– Complexity O(n6)– Cannot handle large problems!

Details in Technical ReportDetails in Technical Report

Minimizing Higher Order Energy Functions: Our results

• Minimizing Higher order functions using Graph cuts– Higher order functions can be transformed to second

order functions by adding auxillary variables– Exponential number of auxillary variables needed in

general

• Result 2– Our higher order functions can be transformed to

second-order functions using ≤2 auxillary variables per potential.

– Can be minimized extremely efficiently – Complexity << O(n6) Details in Technical ReportDetails in Technical Report

Overview of our Method

Unary Potentials [Shotton et al. ECCV

2006]

+Contrast Sensitive Pairwise

Potentials

Higher Order Potentials

(Multiple Segmentations)

Segmentation Solution

+

Higher Order Energy

Energy Minimizatio

n

Experimental results

Datasets: MSRC (21), Sowerby (7) [Shotton et al. ECCV 2006] [He et al. CVPR 04]

Qualitative Results

Image(MSRC-

21)

Pairwise CRF

Higher order CRF

Ground Truth

SheepSheep

GrassGrass

Qualitative Results (Contd..)

Image(MSRC-

21)

Pairwise CRF

Higher order CRF

Ground Truth

Results can be improved using image specific colour models Rother et al. SIGGRAPH 2004 Shotton et al. ECCV

2006

Quantitative Results: Problems

Rough ground truth segmentations

Fine structures have small influence on overall pixel accuracy

Generating Accurate Segmentations

Image(MSRC-21)

OriginalSegmentation

NewSegmentation

• Generated accurate segmentation of 27 images• 30 minutes per image

Relationship between Qualitative and Quantitative

Results

Image(MSRC-21)

Pairwise CRF

Higher order CRF

Ground Truth

95.8%Overall Pixel

Accuracy

98.7%

Small changes in pixel accuracy can lead to large improvements in segmentation results.

Quantitative Accuracy• Measure accuracy in labelling boundary pixels.• Accuracy evaluated in boundary bands of

variable width

Image(MSRC-21)

Hand-labelled Segmentation

Trimap(8-pixels)

Trimap(16-pixels)

Quantitative Accuracy

• Measure accuracy in labelling boundary pixels.

• Accuracy evaluated in boundary bands of variable width

Conclusions

• Method to enforce label consistency in image regions

• Generalization of the commonly used Pairwise CRF model

• Allows integration of pixel and region level features for image labelling problems

Thanks

Running Time ResultsTim

e (

sec)

Number of Segmentations

Inconsistency Cost

Qualitative Results (Contd..)

Image(MSRC-

21)

Pairwise CRF

Higher order CRF

Ground Truth

Transformation to second order functions

cAuxiliary variables