JetStream: Probabilistic Contour Extraction with Particles

Patrick Perez, Andrew Blake, and Michel Gangnet, Microsoft Research, St George House, 1 Guildhall Street, Cambridge, CB2

3NH, UK

http://research.microsoft.com/vision

Presented by:

Vladan Radosavljevic

OutlineIntroduction

Image GradientRelated Work

Probabilistic contour trackingTracking framework

DynamicsMeasurement

Iterative computation of posterior - particlesModel Ingredients

Likelihood ratioDynamicsProposal Sampling Function

Experimental ResultsConclusion

Motivation• Contour extraction

– from segmenting images with closed contours

– to the extraction of linear structures of particular interest such as roads

Introduction - image gradient• The gradient of an image:

The gradient points in the direction of most rapid change in intensity

• The gradient direction is given by:

• how does this relate to the direction of the edge?

• The edge strength is given by the gradient magnitude

Introduction• Most approaches to contour extraction rely on some

minimal cost principle

• k is the curvature, s is the arc-length• y(r(s)) scalar or vector derived at location r(s) from

the raw image I• Often y(r(s)) is the gradient norm

• This function captures some kind of regularity on candidate curves, for example - rewarding, by a lower cost, the presence of large gradients along the contour

Introduction – related work• First approach - dynamic programming

• optimal curve in the form of chain pixels• unless optimality is abandoned, and huge storage resources

are available, there are tight restrictions on the form of cost function

• Second approach - growing a contour from the seed point according to cost function• given the current contour, a new segment is appended to it

according both to a shape prior (mainly smoothness) and to the evidence provided by the data at the location under concern

• deterministic: complete discontinuous curves provided by edge detectors

Introduction – related work• A probabilistic point of view:

• the contours are seen as the paths of a stochastic process• tracking problem

• We are interesting in the method that is able:• to avoid spurious distracting contours• to track the multiple off-springs starting at branching

contours• to interpolate over transient evidence “gaps”

• JetStream – a method with particle filtering

Probabilistic contour tracking• Tracking contours in still images is an unconventional

tracking problem because of the absence of a real notion of time

• The “time” is only associated with the growing of the contour in the image plane – consider a spatial chain as a temporal chain

• Contrary to standard tracking problems where data arrive one bit after another as time passes by, the whole set of data y is standing there at once

• There is no straightforward way of tuning the “speed”, or equivalently the length of successive moves

Tracking framework - dynamics• xi – points in the plane R2

• (x0, x1,...,xn) - curve in some standard way, e.g., the xi’s are the vertices of a polyline, starting from x0 then moving along the contour to xn

• dynamics:

• assuming second order dynamics with some kernel q:

• then a priori density is:

Tracking framework - measurement

• measurement y (observed image) conditioned on x0:n is considered as independent spatial process which is not true in reality but it is a reasonable approximation:

where Ώ is a discrete set of measurements locations in the image plane

• the conditional distributions depend only on whether or not point u is contained in contour x0:n

• in particular, for any two points u along the contour the corresponding conditional distributions are identical, similarly for any two points in the background

Tracking framework - measurement

• Therefore, each is either pon if u belongs to the contour x0:n, or poff if not:

• Finally:

where

Iterative computation of posterior• Function can be

considered as a cost function• Expressed as the minimization of this function, the

contour extraction problem then amounts to seeking the maximum a posteriori (MAP) estimate

• Posterior densities can be computed recursively:

but there is no closed form of the pi

• pi can be approximated by a finite set of M sample paths (the ‘particles’)

• Best path at step i can be which is a Monte Carlo approximation of posterior expectation

Iterative computation of posterior• Prediction: each path is grown one step

by sampling from the proposal density function f

• If the paths are samples from pi then the extended paths are samples from f*pi

• Since we want samples from distribution pi+1, extended paths are weighted according to ratio

• The resulting weighted path set now provides an approximation of the target distribution pi+1

• M paths are drawn with replacement from the weighted set

• The weights are:

Likelihood ratio l• Measurement: the norm of the luminance (or color)

gradient

poff pon

• poff – exponential distribution

• pon - complex mixture, better keep as less informative as possible

Likelihood ratio l

• The direction of the gradient also retains precious information that a data model based only on gradient norm neglects: the distribution of Ψ is symmetric, and it becomes tighter as the norm of the gradient increases

Likelihood ratio l• However, at corners, the norm of the gradient is

usually large but its direction cannot be accurately measured

• Using a standard corner detector, each pixel u is associated with a label c(u) = 1 if a corner is detected, and 0 otherwise

• Where a corner has been detected assumption is that distribution is uniform

Dynamics q• Because of the absence of natural time, it is better to

consider a dynamics with fixed step length d. The definition of second order dynamics then amounts to specifying an a priori probability distribution on direction change Θi

• Finally:

• To allow for abrupt direction changes at the locations where corners have been detected, the normal distribution is mixed with a small proportion of uniform distribution

Proposal sampling function f• With choice f = q, corners will be mostly ignored since

the expected number of particles undertaking drastic direction changes is vM, where typically ν = 0.01 and M = 100

• At locations where no corners are detected, the proposal density is the normal component of the dynamics.

• If location lies on a detected corner, the next proposed location is obtained by turning of an angle picked uniformly

• Therefore:

Proposal sampling function f

JetStream - iteration

Experimental Results – Interactive cut-out

• The extraction of a region of interest from one image

• In practice, JetStream is run for a fixed number n of steps (100 in our experiments) from initial conditions x0:1 chosen by the user.

• If the result is satisfactory, n more steps are undertaken.

• If not, a restart region within the particle flow, and an associated restart direction, can be chosen by the user

Experimental Results – Interactive cut-out

• More sophisticated user interaction• Provide the user with the facility to place one or more

dams, defined as regions Rk where:

Experimental Results – Interactive cut-out

Experimental Results – Road extraction• In the specific context of road extraction, one is in

fact interested in recovering “ribbons”• Using JetStream as defined previously results in

paths jumping from one side of the ribbon to the other

Experimental Results – Road extraction• The state space is extended to include a width

variable mi, which indicates the distance at step i between the two sides of the ribbon

• It expresses that xi+ and xi

- defined as

are on a contour while xi is not:

Experimental Results – Road extraction

• A simple first order dynamics is chosen for m:

Experimental Results – Road extraction

Conclusion• JetStream is applicable if there are not sharp corners

in the images • If sharp corners exist or the image’s intensity

distribution is complex JetStream doesn’t provide good results

References[1] M. Pérez, A. Blake, and M. Gangnet.

JetStream : Probabilistic contour extraction with particles. In Int. Conf. on Computer Vision, ICCV’ 2001, Vancouver, Canada, July 2001.