Image Segmentation of Multiple Objects and Their Compartments

Jerry L. PrinceImage Analysis and Communications

Laboratory (IACL)http://iacl.ece.jhu.edu

Johns Hopkins University© 2010

Acknowledgments• Chenyang Xu• Dzung Pham• Xiao Han• Duygu Tosun• Bai Ying• Daphne Yu• Kirsten Behnke• Xiaodong Tao• Sarah Ying• Xian Fan

• Susan Resnick• Mike Kraut• Maryam Rettmann• Christos Davatzikos• Nick Bryan• Aaron Carass• Ulisses Braga-Neto• Lotta Ellingsen• Pierre-Louis Bazin

Funding sources: NSF, NIH/NINDS, NIH/NIA

Outline• Introduction• Deformable models• TGDM: topology-preserving geometric

deformable model• MGDM: multi-object geometric deformable

model• Conclusion

Outline• Introduction• Deformable models• TGDM: topology-preserving geometric

deformable model• MGDM: multi-object geometric deformable

model• Conclusion

Conventional Structural Image

www.medical.philips.com

Segmentation of Brain Structures

Volumetric MR Data

Subcortical Structures

Cortex

TOADS CRUISE

Bazin and Pham, TMI, 2007

Bazin and Pham, MedIA, 2008

Xu et al., TMI, 1999

Han et al., NeuroImage, 2004

Tosun et al., NeuroImage, 2006

Thalamic nuclei

Cerebellar lobules

Multi-Compartment Anatomy

Cell counting

Circuit board inspection

Retinal examination Traffic camera

Satellite imageryAerial photographs

Other Multi-Object Scenarios

Outline• Introduction• Deformable models• TGDM: topology-preserving geometric

deformable model• MGDM: multi-object geometric deformable

model• Conclusion

Cortical Surface Segmentation

Partial Inflation

Ventricle Segmentation

• Parametric deformable models (PDMs)

─explicit parameterization

• Geometric deformable models (GDMs)– implicit

representation

Deformable Models

Parametric to Geometric[Osher & Sethian 1988]

Level Set PDE:

Contour Deformation:

Visual Concept of GDM

Properties of GDMs• Advantages:

– Produce closed, non-self-intersecting contours– Independent of contour parameterization– Easy to implement: numerical solution of PDEs on

regular computational grid– Stable computations– Automatically changes topology

• Potential disadvantage:– Does not maintain topology

• GDM cannot control topology• TGDM (ours) preserves topology

Topology Behavior

GDM: Standard GeometricDeformable Model

TGDM: Topology-preserving Geometric Deformable Model

Why Maintain Topology?

GDM: Standard GeometricDeformable Model

TGDM: Topology-preserving Geometric Deformable Model

Outline• Introduction• Deformable models• TGDM: topology-preserving geometric

deformable model• MGDM: multi-object geometric deformable

model• Conclusion

Marching Cubes Isosurface• Where is the boundary

defined by a level set function?

• Consider voxel values on corners of a cube

• Label as– above isovalue– below isovalue

• Determine position of triangular mesh surface passing through the cubes by linear interpolation

> 0.5

< 0.5

Voxel values

Digital Connectivity

• Consistent pairs: (foreground,background) → (6,18), (6,26), (18,6), (26,6)

6-connectivity

18-connectivity 26-connectivity

Digital Embedding of Contour Topology

White Points:

Black Points:

• Contour topology is determined by signs of the level set function at pixel locations

• Topology of the implicit contour is the same as the topology of the digital object

Connectivity Rule of Contour

• Topology of digital contour determined by connectivity rule

Same digital object, different topologies

Topology Preservation Principle

• Preserving surface topology is equivalent to maintaining the topology of the digital object

• The digital object can only change topology when the level set function changes sign at a grid point

• Which sign changes can be allowed, and which cannot?

• To prevent the digital object from changing topology, the level set function should only be allowed to change sign at simple points

[Han et al., PAMI, 2003]

Simple Point• Definition: a point is simple if adding or removing

the point from a binary object will not change the object topology

• Determination: can be characterized locally by the configuration of its neighborhood (8- in 2D, 26- in 3D) [Bertrand & Malandain 1994]

SimpleNon-

Simple

x is a Simple Point

0)( x

x

0)( x

xx

x is Not a Simple Point

0)( x 0)( xX

X

Topology Preserving Geometric Deformable Model (TGDM)

• Evolve level set function according to GDM• If level set function is going to change sign, check

whether the point is a simple point– If simple, permit the sign-change– If not simple, prohibit the sign-change (replace the grid value by epsilon with same sign)– (Roughly, this step adds 7% computation time.)

• Extract the final contour using a connectivity consistent isocontour algorithm

Ambiguous Faces

Two possible tilings:

Ambiguous Cubes

Two possible tilings:

Connectivity Consistent MC Algorithm

• (black,white)• (18,6) choose b, f• (26,6) choose b, e

(a) (b) (c)

(d) (e) (f)

AmbiguousFace

AmbiguousCube

• (6,18) choose c, f• (6,26) choose c, f

Nested Deformable Surfaces

Pial Surface

Inner Surface

Central Surface

TGDM-3

Initial WM Isosurface

TGDM-2TGDM-1

TGDM for Inner Surface

Initial WM Isosurface Evolving GM/WM Interface

[Han et al., NeuroImage, 2004]

IACL

TGDM for Central Surface

Initialize with GM/WM surface Evolving toward Central Surface

TGDM for Outer Surface

Evolving toward Outer SurfaceStart from Central Surface

Results—Visual Inspection

Sagittal

• surfaces overlaid on cross-sections of the original image

Axial

Outline• Introduction• Deformable models• TGDM: topology-preserving geometric

deformable model• MGDM: multi-object geometric deformable

model• Conclusion

Multiple Object Challenges1. Maintenance of

multiple level sets

2. Maintenance of object’s individual topologies and relationships between objects

Anatomical parcellation is not arbitrary

Prior Strategies• N level set functions for N objects

[Paragios00, Brox06, …]– Pros: Flexibility between objects– Cons: Objects might overlap or form gaps;

large memory and computational demands

• Multi-phase [Vese02] (4-color theorem)– Pros: Log(N) level set functions for N

objects; low computational complexity; no overlaps or gaps;

– Cons: Forces limited to region and length terms; little control over individual object forces; no 3D equivalent

Principle of MGDM• Simple point criterion can

be replaced by digital homeomorphism criterion

• Movement of collection of objects occurs primarily at: – edges between two objects

or – junctions between three

objects

• Higher-order relationships can be ignored

Objects are not digitally homeomorphic.

Fan et al., CVPR’08, MMBIA’08

Level Set Function Decomposition• N objects Oi, i=1,…N• Distance to objects:

• Label functions:

L0 = Object

L1 = Nearest neighbor

L2 = 2nd nearest neighbor

Distance and Level Set Functions• Distance-based functions:

• Reconstruction of level set functions:

0(x)

1(x)

2(x)

^

Approximation: valid assuming 3 objects max per junction

2D

3D

Evolution• Recall GDM:

• Required evolutions:

• Distance-based functions:

• Assume1. Compute forces2. Find “third” neighbor:

3. Compute:

If then setSet

MGDM Algorithm (2D case)4. Compute:

If then setand• Digital topology and

homeomorphism are readily added

Simulation Experiments• Compare algorithms:

– multiphase (MP)– coupled level sets (CLS)– ours (MGDM)

• Objective function (classic Mumford Shah energy; also Chan-Vese for GDM)

• Evaluate:– convergence, memory usage, computation

time, and misclassification percentage

Visual Comparison

Convergence ComparisonE

Iteration

Quantitative Results

Experiment I: Whole Brain SegmentationStructure memberships from TOADS [Bazin 07] (Topology-preserving, Anatomy-driven segmentation)

ii uf 5.0iu

Membership function:

Force:

Balloon force Smooth force

Sulcal CSF

Cerebral GM

Cerebral WM Cerebellar GM

Cerebellar WM

CaudateThalamus PutamenVentriclesBrainstem

Whole Brain Segmentation: 2D Visualization

(a) Original Image (c) No Topology or Smooth

(b) Result from Toads (d) No Topology but Smooth (f) Group Topology

(e) Single Topology

Topology is preserved with DHC

Whole Brain Segmentation: 3D Visualization

Object topology and relationships between objects can be preserved.

(a) No Topology or Smooth (c) Single Object Topology

(b) No Topology but Smooth (d) Group Objects Topology

Experiment II: Thalamic Nuclei Parcellation

MP-RAGE image

Thalamus Membership

TOADS

FA with PEV color map

co-registered

Homogeneous Orientation

Force for the thalamus boundary.

Force for the thalamus nuclei.

Apply to voxels whose label or the first neighbor is the background.

Apply to voxels whose label or the first neighbor belongs to thalamic nuclei.

Different forces applied to different parts of an object

F

Thalamic Nuclei Parcellation

iii xxxf )()(v)( V

Mean principal orientation from region i

)()(5.0)( xxuxf

The force for thalamus nuclei parcellation is designed based on the assumption that the orientations for each nuclei is homogeneous.

The force for thalamus boundary is a combination of balloon and smooth terms.

Membership function of thalamus at voxel x

Thalamic Nuclei Parcellation: Result

(b) Initialization (d) Result(c) Principal Orientation of Thalamus

(a) Membership Function for Thalamus

Front View Off Diagonal ViewOff Left ViewBack View

Outline• Introduction• Deformable models• TGDM: topology-preserving geometric

deformable model• MGDM: multi-object geometric deformable

model• Conclusion

General Principles• Object topology can be strictly preserved in

geometry deformable models: TGDM• Multiple objects can be simultaneously

segmented and– topology preserved– object relationships preserved– memory efficient– all conventional forces can be applied– guaranteed to have no overlaps or gaps

Remaining Concerns and the Future• How to establish initial object or collection?

– digital topology is not always preserved under simple transformations such as rotation

– recent work on manual skeleton is promising, but tedious

– automatic topology correction is known only for spherical topology, and it is not globally optimum

• Problem of objects getting “stuck”– not so bad in TGDM– much worse in MGDM

Questions?