Intensity-based deformable registration of 2D fluoroscopic X-

ray images to a 3D CT model

Aviv Hurvitz

Advisor: Prof. Leo Joskowicz

2

Navigation in orthopedic surgery

Applications

• Determine position of surgical tools relative to anatomy

• Position surgical robots

• Match pre-operative model to anatomy

[medtronic.com][hss.edu]

3

Navigation in orthopedic surgery

Related registration methods

• Registration to fiducials– Implanted fiducials– On skin

• Contact based registration (Point-cloud to surface registration)

• Registration of fluoroscopic X-ray to CT

4

Registration of fluoroscopic X-ray to CT

[LaRose 2001; Tomazevic 2003; Knaan 2003; Livyatan 2003; and others]

Preoperatively:1. Acquire CT of anatomy of interestIntraoperatively:1. Fix a tracking marker to the bone2. Acquire 2-5 fluoroscopic X-ray images from various camera

poses3. For each image, record:

• T(trackercamera) - camera position at acquisition time• T(trackerbone) - bone marker position at acquisition time

4. Obtain an initial transformation estimate T(boneCT)5. Estimate transformation T(boneCT) with the algorithm we

describe next

Tracker updates T(trackerbone) continuously, enabling real-time navigation.

5

Registration of fluoroscopic X-ray to CT

Input:1. For 2-5 X-ray images:

a. T(trackercamera) at acquisition timeb. T(trackerbone) at acquisition time

2. Initial T(boneCT) estimateRepeat until convergence:

1. For each camera position:a. Define a virtual camera positioned relative to CTb. Create DRR

2. Rate similarity of DRRs to X-ray images by a similarity metric

3. Determine the next T(boneCT) by an optimization algorithm

6

DRR creation

[Knaan et al., 03]

7

Is it possible to do the registration without a CT?

Advantages:

• Save time before operation

• Save costs

• Decrease radiation exposure

Disadvantages:

• A patient-specific CT provides more information

8

Is it possible to create a patient-specific model without a CT?

Approach• Use prior information – everyone’s bones are

similar• Deduce exact shapes of bones from the

fluoroscopic images

In absence of CT, need to search for both bone pose (6 d.o.f.) and the patient-specific bone shape

9

Image analysis-by-synthesisDefine a model of the bone which is parameterized by

pose and by shape.Repeat until convergence:

1. Simulate X-ray imaging process to create DRRs2. Rate similarity of DRRs to fluoroscopic images by similarity

metric3. Modify pose and shape parameters by optimization

algorithm

When the DRRs match the fluoroscopic images We found the pose and the shape of the bone We have registration: a transformation from every

point in the model to its corresponding point in the patient anatomy.

We can map data from the model to the patient anatomy. E.g., map the surface of the bone.

10

Active Appearance Model (AAM)[Cootes et al., 98; Matthews and Baker, 04]

• Model shape variations and appearance variations compactly (with few parameters)

• Can be trained from a dataset of samples • A generative model – can generate new images which

are similar to the training images• Image analysis-by-synthesis

[Edwards, Taylor and Cootes, 98]

11

• Each shape is represented by a vector of landmark coordinates

Shape• Manually label landmark points on training

images• Define a triangular mesh between the landmarks

[Matthews and Baker, 04]

12

• Procrustes analysis - align meshes to cancel global variations in translation, rotation and scale

• PCA on shape vectors

13

Appearance

• Warp each training image to the base mesh so. This creates a set of shape-normalized images.

• Represent pixels inside the base mesh so as vectors

• PCA on shape-normalized image vectors:

14

The resulting model

[Matthews and Baker, 04]

15

Instantiating the model

• Input:– Shape parameters – Appearance parameters

• Computation:1. Instance’s shape-normalized appearance

2. Instance’s shape mesh

16

Instantiating the model

[Matthews and Baker, 04]

17

Fitting AAMs to images

• Input:– Image I(x)– AAM

• Goal: find shape p and appearance λ that yield an AAM image similar to I(x).

• Minimize SSD over λ and p:

Sum differences over all pixels x in base mesh

18

• The optimization is nonlinear in p and linear in λ.• The parameters are found by iterative gradient

descent.• Standard gradient descent is slow because it

requires calculating the partial derivatives at every iteration.

• There are tricks to estimate gradient quickly [Cootes, 98] , or use a pre-computed gradient.[Matthews and Baker, 04].

• After improvements, algorithm is fast enough for real-time face tracking

19

Using AAMs for 2D-3D registration

• The training dataset is CTs (3D images). ROIs of femoral head approximately 100x100x100 pixels.

• The shapes are defined by tetrahedral meshes (approx. 3,000 nodes and 20,000 tetras).

• It is too difficult to find landmarks manually.

20

Finding landmarks automatically

• Select one CT arbitrarily as a template.• Define a tetrahedral mesh on the template.• Perform 3D-3D deformable registration of

each CT to the template. Defines a transformation

Ti:TemplateCTi

• Apply transformation to template mesh transfer mesh to the corresponding landmarks in each CT.

21

Finding landmarks automatically• 3D-3D registration performed with Elastix software (an

ITK wrapper) [Klein and Staring, 07]

• Registration procedure1. Preparation: match histogram to template2. Rigid registration:

• Initialize by aligning centerpoints of ROIs• Metric: Normalized Cross Correlation• Optimizer: Standard gradient descent

3. Deformable registration• Transform: 3D B-spline. (Grid spacing 8x8x8 voxels)• Metric: Sum of squared differences (SSD)• Optimizer: Conjugate gradient• Use image pyramids (3 levels)

• Unsupervised• Each registration runs approx. 10 mins.• Compute all registrations in parallel on bmos cluster

(mosix)

22

Building AAM of femur CTs• Input:

– 14 CTs and 14 corresponding meshes

• Output:– Mean shape mesh and 6 significant basis

vectors– Mean appearance image and 10 significant

basis vectors

23

Mean volume image - A0

24

Shape model

s0 + 2 std s1 s0 + 2 std s2 S0

25

Using the AAM in 2D-3D registration

Repeat until convergence:• Generate 2D DRRs out of AAM using current

pose, shape, and appearance parameter estimates.

• Rate similarity of DRRs to fluoroscopic images by similarity metric

• Modify AAM shape parameters and the pose parameters by optimization algorithm. (We ignore appearance variations)

26

Generating DRRs from AAM – Algorithm 1

1. Generate a 3D image of the AAM instancefor each pixel in 3D image:

– find tetrahedron containing pixel center (use KD-tree)

– find corresponding point in base mesh (an affine transform)

– read value from appearance image

2. Proceed with standard ray casting of 3D image

[Knaan et al., 03]

27

Generating DRRs from AAM – Algorithm 2

• Cast rays directly through tetrahedral mesh of AAM instance

• Sample points at uniform intervals on ray• Find corresponding points in base mesh• Read values from appearance image• Requires identifying all ray-tetrahedron intersection

points. We implemented algorithm of [Marmitt and Slusallek, 06]

Instance mesh

Base mesh Shape-normalized

volume image

28

Parameter search issues

• Accuracy and Robustness– Search from various initial locations– Genetic algorithm– Downhill simplex optimizer (ITK amoeba)

• Speed– Multi-resolution– Limit DRR generation to ROIs

29

Current status

• The naïve search converges to unsatisfactory result (local minimum)

• Need to improve both accuracy and speed– Search from multiple initialization points– ROIs

• Think about a smarter optimization algorithm

Future:– Experiment on real fluoroscopic images

30

Intensity-based registration of 2D X-ray images to a 3D deformable

model

31

Principal Component Analysis (PCA)