deducing local influence neighbourhoods in images using graph cuts ashish raj, karl young and...

Deducing Local Influence Neighbourhoods in Images Using Graph Cuts

Ashish Raj, Karl Young and Kailash Thakur

Assistant Professor of RadiologyUniversity of California at San Francisco, ANDCenter for Imaging of Neurodegenerative

Diseases (CIND) San Francisco VA Medical Center

email: [email protected]

Webpage: http://www.cs.cornell.edu/~rdz/SENSE.htmhttp://www.vacind.org/faculty

CIND, UCSF Radiology 2

San Francisco, CA


Overview

We propose a new image structure called local influence neighbourhoods (LINs)

LINs are basically locally adaptive neighbourhoods around every voxel in image

Like “superpixels” Idea of LIN not new, but first principled cost

minimization approach Thus LINs allow us to probe the intermediate

structure of local features at various scales LINs were developed initially to address image

processing tasks like denoising and interpolation But as local image features they have wide

applications


Local neighbourhoods as intermediate image structures

Pixel-level Neighbourhood-level

12

3

Region-level

Low level High level

Too cumbersomeComputationally expensive

Not suited for pattern recognition

Prone to error propagationGreat for graph theoretic and pattern recognition

Good intermediaries between low and high levels?


Outline Intro to Local Influence Neighbourhoods How to compute LINs?

– Use GRAPH CUT energy minimzation Some examples of LINs in image filtering and denoising Other Applications:

– Segmentation– Using LINs for Fractal Dimension estimation– Use as features for tracking, registration


Local Influence Neighbourhoods A local neighbourhood around a voxel (x0, y0) is the set of voxels “close” to it

– closeness in geometric space– closeness in intensity

First attempt: use a “space-intensity box”

Definition of , arbitrary Produces disjoint, non-contiguous, “holey”, noisy neighbourhoods! Need to introduce prior expectations about contiguity We develop a principled probabilistic approach, using likelihood and prior distributions


Example: Binary image denoising

Suppose we receive a noisy fax:– Some black pixels in the original image

were flipped to white pixels, and some white pixels were flipped to black

We want to recover the original

input image

output image


Problem Constraints

Our Constraints:1. If a pixel is black (white) in the original image,

it is more likely to get the black (white) label2. Black labeled pixels tend to group together,

and white labeled pixels tend to group together

original image

good labeling bad labeling(constraint 1)

bad labeling(constraint 2)

likelihood

prior


Example of box vs. smoothness


A Better neighbourhood criterion1. Incorporate closeness, contiguity and smoothness assumptions2. Set up as a minimization problem3. Solve using everyone’s favourite minimization algorithm

– Simulated Annealing– (just kidding) - Graph Cuts!

A) Closeness: lets assume neighbourhoods follow Gaussian shapes around a voxel


A) Closeness criterion in action


B) Contiguity and smoothness

This is encoded via penalty terms between all neighbouring voxel pairs

p qG(x) = p,q V(xp, xq)

V(xp, xq) = distance metric

A) ClosenessB) Contiguity/smoothness

Define a binary field Fp around voxel p

s.t. 0 means not in LIN, 1 means in LIN

Bayesian interpretation: this is the log-prior for LINs


Markov Random Field Priors

Imposes spatial coherence (neighbouring pixels are similar)

G(x) = p,q V(xp, xq)

V(xp, xq) = distance metric

pq

Potential function is discontinuous, non-convex Potts metric is GOOD but very hard to minimize


Bottomline

Maximizing LIN prior corresponds to the minimization of

E(x) = Ecloseness(x) + Esmoothness(x)

MRF priors encode general spatial coherence properties of images

E(x) can be minimized using ANY available minimization algorithm

Graph Cuts can speedily solve cost functions involving MRF’s, sometimes with guaranteed global optimum.

Graph Cut based Energy Minimization


How to minimize E?

Graph cuts have proven to be a very powerful tool for minimizing energy functions like this one

First developed for stereo matching– Most of the top-performing algorithms for stereo rely

on graph cuts Builds a graph whose nodes are image pixels, and

whose edges have weights obtained from the energy terms in E(x)

Minimization of E(x) is reduced to finding the minimum cut of this graph


Minimum cut problem

Mincut/maxflow problem:– Find the cheapest way to

cut the edges so that the “source” is separated from the “sink”

– Cut edges going from source side to sink side

– Edge weights now represent cutting “costs”

a cut C

“source”

A graph with two terminals

S T

“sink”


Graph construction

Links correspond to terms in energy function Single-pixel terms are called t-links Pixel-pair terms are called n-links A Mincut is equivalent to a binary segmentation

I.e. mincut minimizes a binary energy function


Table1: Edge costs of induced graph

n-links

s

tt-

link

t-link


Graph Algorithm

Repeat graph mincut for each voxel p


Examples of Detected LINs


Results: Most Popular LINs


Filtering with LINs

Use LINs to restrict effect of filter– Convolutional filters:

– Rank order filter:

=


Maximum filter using LINs


Median filter using LINs


EM-style Denoising algorithm

Likelihood for i.i.d. Gaussian noise:

Image prior:

Maximize the posterior:

Noise model: O = I + n


Bayesian (Maximum a Posteriori) Estimate

Bayes Theorem:

Pr(x|y) = Pr(y|x) . Pr(x)

Pr(y)

likelihood

priorposterior

Here x is LIN, y is observed image Bayesian methods maximize the posterior probability:

Pr(x|y) Pr(y|x) . Pr(x)


EM-style image denoisingJoint maximization is challengingWe propose EM-style approach: Start with Iterate:

We show that


Results: LIN-based Image Denoising


Results: Bike image


Table1: Denoising Results

Other Applications of LINs

LINs can be used to probe scale-space of image data

– By varying scale parameters x and n Measuring fractal dimensions of brain images Hierarchical segmentation – “superpixel” concept Use LINs as feature vectors for

– image registration – Object recognition– Tracking


Hierarchical segmentation

Begin with LINs at fine scale Hierarchically fuse finer LINs to obtain coarser LINS

segmentation

How to measure Fractal Dimension using LINs?

How LINs vary with changing x and n depends on local image complexity Fractal dimension is a stable measure of complexity of multidimensional structuresThus LINs can be used to probe the multi-scale structure of image data


FD using LINs For each voxel p, for each value of x, n:

count the number N of voxels included in Bp

.CP1 CP2 ln x

ln N

extend to (x , n) plane

phase transition

Slope of each segment = local fractal dimension


Possible advantages of LIN over current techniques

LINs provide FD for each voxel Captures the FD of local regions as well as global Ideal for directional structures and oriented

features at various scales Far less susceptible to noise

– (due to explicit intensity scale n which can be tuned to the noise level)

Enables the probing of phase transitions


Possible Discriminators of Neurodegeneration

Fractal measures may provide better discriminators of neurodegeneration (Alzheimer’s Disease, Frontotemporal Dementia, Mild Cognitive Disorder, Normal Aging, etc)

Possibilities:– Mean (overall) FD -- D(0)– Critical points, phase transitions in (x, n) plane– More general Renyi dimensions D(q) for q ¸ 1– Summary image feature f() D(q)– Phase transitions in f()

Fractal structures can be characterized by dimensions D(q), summary f() and various associated critical points

These quantities may be efficiently probed by the Graph Cut –based local influence neighbourhoods

These fractal quantities may provide greater discriminability between normal, AD, FTD, etc.


Summary

We proposed a general method of estimating local influence neighbourhoods

Based on an “optimal” energy minimization approach

LINs are intermediaries between purely pixel-based and region-based methods

Applications include segmentation, denoising, filtering, recognition, fractal dimension estimation, …

… in other words, Best Thing Since Sliced Bread

Ashish RajCIND, UCSF

email: [email protected]

Webpage: http://www.cs.cornell.edu/~rdz/SENSE.htmhttp://www.vacind.org/faculty

Deducing Local Influence Neighbourhoods in Images Using Graph

Cuts

deducing local influence neighbourhoods in images using graph cuts ashish raj, karl young and...

Documents