(rigid/affine) image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (rigid/afﬁne) image...

(Rigid/Affine)Image registration

Simon Rit12

1CREATIS laboratory

2Leon Berard center

Master EEAP / SI - Module 5 - 2012

Introduction Transformations Similarity measures Optimization Validation Conclusion

Module 5: Registration and Motion Estimation

RegistrationRigid/Affine: 4h, S. RitNon-rigid: 2h, D. SarrutPractical exercises: 2h, S. Rit and D. Sarrut

Motion estimationClassical approaches, discontinuities: 4h, P. ClarysseApplication to ultrasound imaging: 2h, H. LiebgottPractical exercises: 2h, P. Clarysse and S. Rit

Evaluation: practical exercises and test

Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 2


Outline

1 Introduction

2 Transformations

3 Similarity measures

4 Optimization

5 Validation

6 Conclusion



Image registration

DefinitionThe process of aligning images so that corresponding featurescan easily be related.

= Registration= Alignment= Geometrical correspondence= Matching= Motion estimation



Motivation

Many applicationsMedical imagingVisionetc.

Basic elements are used in many applicationsInterpolationSimilarity measureOptimizationetc.



Goal

Find a spatial transformation T that matches two images.

T

Image 1 Image 2



Example #1: panoramic photography



Example #2: satellite pictures (Google maps)



Example #3: multimodal 3D medical images

Registration is a prerequisite for adequate image fusion

MRI PET



Example #3: multimodal 3D medical images

Registration is a prerequisite for adequate image fusion

Before registration After registration



Example #4: image-guidance

CT scanner In-room cone-beam CT



Example #4: image-guidance

CT X-ray projection Cone-beam CT

3D/2D (one projection) or 3D/3D (reconstruction) registration



Summary

The input images may differ in

ResolutionTimeSpace (2D, 2D+t, 3D, 3D+t...)Modality (Photo, Radiography, CT, PET, MRI, US...)Subject (Inter-patient registration)...

There are many possible output transformations.

⇒ Many algorithms...



Formalism

T = arg maxT∈T

S(I , J,T )

NotationI = reference / fixed imageJ = moving / floating imageT = transformationT = search spaceS = similarity measurearg max = optimizationT = solution



Digital images

Discrete set of pixels/voxels

Pixels = 2D Voxels = 3D

We consider two images I(i), J(j) ∈ R.i , j ∈ Z3 are the spatial indices of the lattice.



Digital images

Spatial resolutionEx: 512× 512 pixels of 0.5× 0.5 mm2

Pixel valuesScalar (e.g. CT) or vector (e.g. RGB) or matrix (e.g.diffusion MRI) at each pointDigital number(s) (integer, float...) ⇒ limited precision

Visualization according to a color scaleDefined with window/level for linear gray scales

Coordinate systemOriginOrientation (e.g., with respect to anatomical orientation:cranio-caudal, left-right, antero-posterior).



Outline

1 Introduction

2 Transformations


4 Optimization

5 Validation

6 Conclusion



Formalism

T = arg maxT∈T

S(I , J,T )




Transformations

Cartesian coordinates: x = (x , y , z, t)T

Transformation :T : R4 → R4

x → T (x) = x ′



Search space T

One point of T is a transformation T . It depends on theDegrees of freedom = number n of parameters needed todescribe the transformation. n is the dimension of theoptimization problem.Boundaries of each parameter (if any).

The success of the registration requires thatThe solution T is in T .T is small enough with respect to the input data.



Categories

Linear registration: T (ax1 + x2) = aT (x1) + T (x2)(more exactly, composition of linear transformations withtranslations)

Rigid (translations + rotations)Rigid + ScalingAffine

Non-linear registrationNon-rigidDeformableElasticFluid



Rigid registration

A rigid transformation can be described with translations androtations.

The dimensionality can be adapted2D/2D→ 3 parameters (2 translations, 1 translation)3D/3D→ 6 parameters (3 rotations, 3 translations)2D/3D→ 6 parameters (3 rotations, 3 translations +projection operator).



Matrix representations

Rigid / affine transformations can be represented with matrices:

Tlinear (x) = Rx + t

where R is the rotation matrix and t the translation vector (R is3× 3 and t ∈ R3 in 3D).

For rigid transformations, R is constrained to have only 3parameters. If one uses all 9 parameters of R, T is an affinetransformation.



Homogeneous coordinates

For practicality, one can use a single matrix with homogeneouscoordinates:

Tlinear (x) = Mx

where x is the point in homogeneous coordinates and Mcombines R and t .

In 3D, we would have x = (x , y , z,1) and the matrix

M =

R00 R01 R02 t0R10 R11 R12 t1R20 R21 R22 t20 0 0 1

Also useful to apply projection transforms.



Matrix manipulation

Application: T (x) = Mx

Inverse (if invertible): T−1(x) = M−1x

Composition: T2(T1(x)) = (T2 T1)(x) = M2M1x

etc.



Parameterization

Affine = all possible matrices = 12 parameters in 3D

M =

R00 R01 R02 t0R10 R11 R12 t1R20 R21 R22 t20 0 0 1

For registration, one canOptimize the 12 parameters of the matrixDecompose the matrix in more meaningful parameters, e.g.translations, rotations, ...



Translations

In 2D, 2 translation parameters

M =

1 0 t00 1 t10 0 1

In 3D, 3 translation parameters

M =

1 0 0 t00 1 0 t10 0 1 t20 0 0 1



2D rotation around the origin

One parameter, the angle θ:

R =

cos θ − sin θ 0sin θ cos θ 0

0 0 1



2D rotation around another point

One can combine a translation (=change of coordinate system)with this rotation matrix to have a rotation around another pointp = (p0,p1):

R′ =

1 0 p00 1 p10 0 1

cos θ − sin θ 0sin θ cos θ 0

0 0 1

1 0 −p00 1 −p10 0 1



3D rotation

Rotation = 3× 3 matrix (Euler’s rotation theorem)

3 degrees of freedom (parameters)

Constraints: orthogonal matrix (RRT = Id) and det(R) = 1⇒ Inverse: R−1 = RT



3D rotation

Several decompositions from the 9 matrix values to 3parameters, e.g.

Euler anglesAxis-angleQuaternionsetc. (see http://en.wikipedia.org/wiki/Charts_on_SO(3) for others)

⇒ The parameters have a different meaning in each case


http://en.wikipedia.org/wiki/Charts_on_SO(3)


3D rotation – Euler angles

Composition of 3 rotations around the axis of thecoordinate system (parameters α, β, γ or θ, ϕ, ψ or Yaw,Pitch and Roll / Roulis, Tangage et Lacet):

RX =

1 0 00 cosα − sinα0 sinα cosα

RY =

cos β 0 sin β0 1 0

− sin β 0 cos β

RZ =

cos γ − sin γ 0sin γ cos γ 0

0 0 1

Order matters⇒ several conventions!Euler angles: ZXZ (XYX, XZX, YXY, YZY, ZYZ)Tait-Bryan angles: YXZ (XYZ, XZY, YZX, ZXY, ZYX)



3D rotation – The ZXY convention

http://en.wikipedia.org/wiki/Euler_angles


http://en.wikipedia.org/wiki/Euler_angles


3D rotation – Issues with Euler angles

Convention issuesGimbal lock = singularity

http://en.wikipedia.org/wiki/Gimbal_lock

Can be solved by changing the convention (other order) oradding a 4th gimbal.


http://en.wikipedia.org/wiki/Gimbal_lock


3D rotation – Axis-angle



3D rotation – Axis-angle

From Euler’s rotation theorem4 parameters

Unit vector n = (nx ,ny ,nz)T

Angle θ

but 3 degrees of freedom because unit vector (||n|| = 1).

Computation of the new position from parameters withRodrigues’ rotation formula:http://mathworld.wolfram.com/RodriguesRotationFormula.html

LimitationsRepresentation not minimalDifficult to combine two rotations

Euler parameters refer to the axis-angle representation andare different from the Euler angles...


http://mathworld.wolfram.com/RodriguesRotationFormula.html


3D rotation – Quaternions

4 parameters q = (q0,q1,q2,q3)T but 3 DOF |q|2 = 1q0q0 + q1q1 − q2q2 − q3q3 2(q1q2 − q0q3) 2(q1q3 + q0q2)

2(q2q1 + q0q3) q0q0 − q1q1 + q2q2 − q3q3 2(q2q3 − q0q1)2(q3q1 + q0q2) 2(q3q2 + q0q1) q0q0 − q1q1 − q2q2 + q3q3

Link with angle-axis: q =

cos(θ/2)

nx sin(θ/2)ny sin(θ/2)nz sin(θ/2)

Advantage: combine rotations, stability

Inconvenient: 4 non-independent parameters



Scaling

In 3D, 3 parameters sx , sy , sz

R =

sx 0 00 sy 00 0 sz

Useful if working in voxel coordinates, e.g.

Image #1: dxI × dyI × dzI mm3

Image #2: dxJ × dyJ × dzJ mm3

R =

dxIdxJ

0 00 dyI

dyJ0

0 0 dzIdzJ



Shear

In 3D, 6 parameters sxy , syx , sxz , szx , syz , szy

R =

1 syx szxsxy 1 szysxz syz 1



Skew

Idem as shear but antisymmetric (RT = −R)

⇒ In 3D, 3 parameters syx , szx , szy

R =

1 syx szx−syx 1 szy−szx −szy 1



Projection – Matrix

(n − 1)× n matrixParallel or perspective2D/1D perspective projection in homogeneouscoordinates:

M =

[SDD 0 0

0 1 SID

]SDD: Source to Detector DistanceSID: Source to Isocenter Distance



Summary

Linear matrices can be decomposed inhttp://mathworld.wolfram.org/

http://www.wikipedia.com/

Translation parameters: straightforward

Rotation parameters: need caution, several solutions

Scaling parameters: straightforward

Skew parameters: less used

+Projection for 2D/3D registration


http://mathworld.wolfram.org/

http://www.wikipedia.com/


Summary

Only a subset of the parameters can (should?) be optimized.

Rigid: 6 parameters (translations and rotations)

Rigid + global rescaling: 7 parameters

Rigid + independent rescaling: 9 parameters

Affine: 12 parameters (translations, rotations, scaling,skew)



Tips and tricks

Carefully chose the search space

As little parameters as possible...

... but make sure that the sought solution is in it!

Bound the search space, e.g. translations < 2 cm androtations < 10

Scale the parameters which are heterogeneous, e.g.translations in mm and rotations in radians or degrees



Outline

1 Introduction

2 Transformations


4 Optimization

5 Validation

6 Conclusion



Formalism

T = arg maxT∈T

S(I , J,T )




Similarity measure

Measure of the alignment quality of I and J with T .

Two families1 Feature-based registration2 Intensity-based registration



Features

Features are extracted from images prior to similarity measure

PointsLinesVectorsSurfacesVolumes...



Feature-based registration formalism

T = arg maxT∈T

S(FI , T FJ)

NotationFI = feature set of reference / fixed imageFJ = feature set of moving / floating imageT = transformationT = search spaceS = similarity measurearg max = optimizationT = solution



Geometric primitives – Manual identification

Landmarks (recognizable points) clicked by an expert.



Geometric primitives – Manual identification

Manual contours.



Geometric primitives – Automatic identification

Segmentation⇒ Another problem which can be very difficult...

Unpaired features⇒ Unpaired similarity measure, e.g. pair to closest feature in

other image⇒ Or find an algorithm to pair features (another problem...)



Manual vs auto: pros and cons

Manual+ Reference- Long- Expensive (requires time of an expert, e.g. physician)- Inter- and intra- observer variability

Automatic+ Fast and cheap- Paired or unpaired?? Reproducible? Robust



(Dis-)similarity measures for features

Sum of distances

L1-norm: |x |1 =n∑

r=1

|xr |

L2-norm: |x |2 =

√√√√ n∑r=1

x2r

Quadratic sum (faster):n∑

r=1

x2r

Other distances...



Distance map

Map of the distance between each point and the closest feature⇒ Fast computation of distances for unpaired features

Image Features = bones Distance map

Computed only once for the reference image.Fast computations: Chamfer distance (not Euclidian),separable algorithms [Coeurjolly, PAMI, 2007].



S(FI , T FJ) or S(FJ , T FI)?

This is not identical sincewith the distance map dI

S(I, J,T ) =∑

F∈FJ

dI T (F )

with the distance map dJ

S(I, J,T ) =∑

F∈FI

dJ T (F )

⇒ Use the distance map of the image which has featureseasier to extract



Summary of feature-based registration

- Preliminary step to identify features- Pairing algorithm maybe required+ Generally fast (depending on the feature type and size)+ Registration of some features only...

There is currently a renewed interest in feature-basedregistration, particularly for non-rigid registration...



Limitation of feature based registration

Multimodal images

⇒ Intensity-based registration



Intensity-based registration formalism

T = arg maxT∈T

S(I , J T )




Intensity-based similarity measure

S(I , J T ) is a measure which assumes a func-tional dependence between the pixel intensities

Intensity-based registration = iconic registrationChoice depends on the modalities

= Functional dependence



Forward and backward warping

Required for computing J T

Image flottante

Image de référence

Zone d’interpolation

Forward mapping

Backward mapping



Forward and backward warping

Forward mapping requires an additional weight map+ potential holes

⇒ Backward is usually preferred. Not a problem:T is easily invertible for affine registrationOne can always optimize T−1 and invert it at the end

The resolution of the reference image is used

More: Wolberg, Digital image warping, 1990



Warping – Fast computation

∀ voxel x ∈ ΩI :

M x =

a b c de f g hi j k l0 0 0 1

×

xyz1

=

x ′ = ax + by + cz + dy ′ = ex + fy + gz + hz ′ = ix + jy + kz + l

1

Use increment in innermost loop (along x here):

x ′n+1 = x ′n + a and x ′0 = by + cz + dy ′n+1 = y ′n + e and y ′0 = fy + gz + hz ′n+1 = z ′n + i and z ′0 = jy + kz + l



Interpolation

Nearest neighbor

Linear

More accurate: spline, sinc...

⇒ Compromise between speed and accuracy



2D/3D: Digitally Reconstructed Radiograph

Based on the Beer-Lambert law:Φ(u, v) = Φ0 exp

−∫

Lβ,u,vf (x)dx



Intensity-based similarity measures

Sum of Squared Difference

SSD(I, J) =∑x∈Ω

(I(x)− J(x))2

Correlation coefficient

CC(I, J) = N∑

(I(x)−mI)(J(x)−mJ)∑√I(x)−mI

∑√J(x)−mJ

Mutual information

MI(I, J) =∑i,j

pij logpij

pi pj

Correlation ratio

CR(I, J) = η2(I|J) = 1− 1σ2

I

∑j

pjσ2I|j

Many others...



Sum of Squared Difference

SSD(I, J) =∑x∈Ω

(I(x)− J(x))2

Dissimilarity measureFastNeeds to be normalized to Ω size (in voxels)Functional dependence between intensities: identity

Optimal if images only differ by Gaussian noise(monomodal)Not robust otherwise (multimodal)

Similar: Sum of Absolute Difference (SAD)



(Pearson product-moment) Correlation coefficient

CC(I, J) = ρ(I, J) =cov(I, J)

σI σJ= N

∑(I(x)−mI)(J(x)−mJ)∑√I(x)−mI

∑√J(x)−mJ

Range [−1,1]

1 is perfect increasing linear relationship-1 is perfect decreasing linear relationship0 if independent

⇒ Similarity measure: |CC(I, J)|Fast (with computational tricks)Linear dependence: I(x) = a J(T (x)) + b ∀x a,b ∈ R

More robust than SSD, e.g. for 2 imaging systems of asame modalityNot good for multimodal



Functional dependence between intensities

SSD : I(x) = J(T (x)) ∀xCC : I(x) = a J(T (x)) + b ∀x

⇒ Other functional relationship? ⇒ Information theory

Mutual informationCoefficient ratio

Both require joint histograms.



Histogram

0 50 100 150 200 2500

0.5

1

1.5

2

2.5

3x 10

4

Pixel intensity

Nu

mb

er

of

pix

els



Joint histogram – Definition

J colors

i

j nij

∑j nij

∑i nij ni

nj

I colors

i , j : (binned) colorsnij : number of pixels withcolor i in I and j in Jni , nj : marginal values, i.e.,histograms of I and J



Joint histogram



Joint histogram

2D distribution of the pair of intensities at each voxellocation

SSD and CC can also be understood with joint histograms

. . . but it is useless to build it for SSD and CC

Recomputed at each step of the optimization



Joint histogram – Example #2

From [Roche, PhD, 2001]



Joint histogram – Computation

Two steps:

1 Count the nij (accounting for T )Compute the warped image J T on I gridCompute the joint histogram of J T and I

Or directly update the histogram pixel-by-pixel

2 Derive the probability pij



Joint histogram



Mutual information – Entropy

Measure of information as a registration metric⇒ Entropy (Shannon-Wiener):

H =∑

i

pi log1pi

= −∑

i

pi log pi

i

p(i

)

Entropy=3.00

i

p(i

)

Entropy=2.99

i

p(i

)

Entropy=2.11

i

p(i

)

Entropy=−0.00



Mutual information – Joint entropy

Two images⇒ two symbols at each grid point

H = −∑i,j

pij log pij

⇒ Joint entropy: the more similar the distributions, the lowerthe joint entropy compared to the sum of individualentropies

H(I, J) ≤ H(I) + H(J)



Mutual information

MI(I, J) =∑i,j

pij logpij

pi pj

MI(I, J) = H(I)− H(J|IJ) = H(J)− H(I|J) = H(I) + H(J)− H(I, J)

PositiveSymmetric MI(I, J) = MI(J, I)H(J|I) = H(I, J)− H(I): conditional entropyExample [Pluim, TMI, 2003]:



Mutual information is a reference

First articles: June 1995Maes in BelgiumViola in US

First journal articles: 1997

Currently: #1 similarity measureMultimodalRobust

Among the most cited papers of the domain...



MI implementation

For mutual information, implementation matters (as usual!).Survey of methods: [Pluim, TMI, 2003].Specifically:

InterpolationBinningProbability computationNormalization (less sensitive to overlap changes)e.g. Normalized Mutual Information [Studholm, PatternRecognition, 1999]



MI implementation – Interpolation

Conventional: nearest neighbor, linear...

Partial volume interpolation (Collignon)

The difference lies in the smoothness of the optimizedfunction



MI implementation – Interpolation

NN LIN

PV



MI implementation – Number of bins



MI implementation – Probability

Frequential approach (simplest): pij =nijN

Parzen window: pij =∑

k

ϕ(nij − nk )

ϕ =Gaussian [Viola, PhD, 1995]ϕ =Cubic spline [Thevenaz, 2000]

Also Bayesian approach...



Correlation ratio

RC(I, J) = η2(I|J) = 1− 1σ2

I

∑j

pjσ2I|j

Based on joint histogram as wellAssume a functional dependence between intensitiesVariance instead of entropyResults comparable to MI, not symmetricftp://ftp.inria.fr/INRIA/publication/RR/RR-3980.pdf


ftp://ftp.inria.fr/INRIA/publication/RR/RR-3980.pdf


Summary of intensity-based similarity measures

The measures can be classified with respect to hiddenvariables [Malandain, HDR, 2006]

SSD, SAD: 0 variableCC: 2 variables, slope and interceptCR: size of one side of the joint histMI: product of the size of the sides of the joint hist

⇒ MI is more general, hence its success⇒ But it is also more susceptible to fail in simple situations

where SSD is sufficient, e.g., time series



Local affine registration

One can always use a region-of-interest (ROI) of the image forregistration (e.g. rectangular box or shaped volume of interest)

Applied to the reference and/or the target image depending onthe similarity measure.



Outline

1 Introduction

2 Transformations


4 Optimization

5 Validation

6 Conclusion



Formalism

T = arg maxT∈T

S(I , J,T )




Challenge

Find the global minimum... as fast as possible



1D illustration



Optimization

Optimization = maximize or minimize

Search for global minimum⇒ if the max is sought, thenminimize −f

Optimization 1D / nD (up to n = 12 for affine registration)

With or without a gradient of fThe solution respects ∇f (x) = 0 (as well as other extrema)

Multi-resolution



Optimization

Iterative process: iteration xk / f (xk+1) < f (xk )

Stopping criterion|f (xk+1)− f (xk )| < ε∇f (x) = 0

One iteration⇒ search δxk / xk+1 = xk + δxk

Initialization x0IdentityAlign image centersAlign mass centers. . .



1D optimization: Golden search

Without derivatives

Based on the bisection method (root search)Based on the intermediate value theorem

Bracketed solutionIf a < b < c such that f (b) < f (a) and f (b) < f (c) then fhas a minimum in the interval (a, c).

⇒ 4 values at each iteration (position based on Goldennumber)a < b < x < c ora < x < b < c

⇒ x replaces either a or c depending on where is theminimum.



1D optimization: Brent’s method

Without derivatives

Based on Dekker’s method, 1969Bisection / Golden search methodSecant methodLinear interpolation

Brent’s method, 1973+ Inverse quadratic interpolation+ Additional tests⇒ 6 values at each step



Brent’s method - Illustration

Three points bracket the intervalInverse quadratic interpolationIterate and stop when f (x3)− f (x1) < ε




Three points bracket the interval

Inverse quadratic interpolationIterate and stop when f (x3)− f (x1) < ε




Three points bracket the intervalInverse quadratic interpolation

Iterate and stop when f (x3)− f (x1) < ε




Three points bracket the intervalInverse quadratic interpolationIterate and stop when f (x3)− f (x1) < ε



nD optimization

Without gradient With gradientSimplex Gradient descent δk+1 = −∇f (xk )Powell Conjugate gradient δk+1 = −∇f (xk ) + βk dk

(Newton) δk+1 = −H−1 .∇f (xk )Quasi-Newton δk+1 = −A−1 .∇f (xk )Levenberg-Marquardt SD/QN



Downhill simplex

Nelder and Meade, 1965Simplex = geometric object(nondegenerate)n + 1 verticesBased on pseudo-derivatives

Not to be mixed with the simplexmethod in linear programming

1 Compute the function (similaritymeasure) at each vertex

2 Conditional deformation (see next slide)



Downhill simplex

https://wiki.ece.cmu.edu/ddl/index.php/Downhill_Simplex




Downhill simplex

AlgorithmSort f (x i) with i the vertex numberTake worst and replace with (if better)

1 Reflection2 Expansion3 Contraction4 Shrink

Iterate until convergence...



Downhill simplex - Examples





Line search along unit vectors

(Numerical Recipes)



Line search

Used by many nD strategies= 1D optimization in a direction of the nD space

⇒ Given a set of parameters x ∈ Rn and a direction y ∈ Rn,find the best α ∈ R so that x + αy is minimal.

Typically: Brent’s method



Powell’s method

Powell’s conjugate gradient descent method (1964)Set of n directions, e.g. the unit vectorsMove along one direction until minimum

Line search

Loop over directions until minimumUpdate set of directions

Displacement vector becomes a (conjugate) directionRemove direction which contributed the most

Order matters, e.g. [Maes, 1999] (tx , ty , φz , tz , φx , φy ).Converge in n(n + 1) iterations for quadratic functions



Conjugate directions

Directions which will not spoil previous minimizations

Let δk and δk+1 be two successive search directions

f has been minimized in the δk direction

⇒ The projection of the gradient on δk is ~0

⇒ Chose a perpendicular direction

⇒ δk+1 will not interfere with the minimization along δk .



Conjugate directions properties

If f is quadratic, one pass through the set of directionsresults in the exact minimumElse, quadratic convergence

Powell’s method constructs a conjugate direction at eachiteration



Powell - Example

http://www.mathworks.com/matlabcentral/fileexchange/authors/26509


http://www.mathworks.com/matlabcentral/fileexchange/authors/26509


nD optimization

Without gradient With gradientSimplex Gradient descent δk+1 = −∇f (xk )Powell Conjugate gradient δk+1 = −∇f (xk ) + βk dk

(Newton) δk+1 = −H−1 .∇f (xk )Quasi-Newton δk+1 = −A−1 .∇f (xk )Levenberg-Marquardt SD/QN



Gradient optimization

Gradient vector:

∇f =

∂f∂x1∂f∂x2

. . .

∈ Rn

Hessian matrix:

H =

∂2f∂x2

1

∂2f∂x1∂x2

. . .

∂2f∂x2∂x1

∂2f∂x2

2. . .

......

. . .

∈ Rn×n



Gradient optimization

Expected gain: O(n) instead of O(n2)

Problem: computing partial derivatives⇒ Is it worth the effort?

But:there might be redundancies in ∇fline search might be faster



Gradient optimization - 1D example



Computing the gradient in nD

Feature-based registration: almost impossibleDepends on the segmentation...

⇒ Iterative Closest Point [Zhang, IJCV, 1994]?

Intensity-based registration: composition of image andtransform, e.g. J T ⇒ use chain rule.In 1D: f g′(x) = f ′(g(x)) · g′(x). In our case:

∂J T (x)

∂p=∂J(T (x))

∂x∂T∂p

Result is a combination ofImage gradientDerivatives of the transform for each parameter in eachdirection (m × n Jacobian matrix in mD for n parameters)



Image gradient

Finite difference(s) (approximate), e.g. (f (x + h)− f (x))/hDeriche recursive gaussian filters and its derivativeshttp://hal.inria.fr/inria-00074778/en/


http://hal.inria.fr/inria-00074778/en/


Derivation of similarity measures

Measure specific

SSD : [Thevenaz, IEEE IP, 1998]

MI : [Viola, PhD, 1995], [Thevenaz, IEEE IP, 2000]

CR : [Cachier, PhD, 2002]



Jacobian matrix examples

3 translations in 3D∂Tx∂t0

∂Tx∂t1

∂Tx∂t2

∂Ty∂t0

∂Ty∂t1

∂Ty∂t2

∂Tz∂t0

∂Tz∂t1

∂Tz∂t2

=

t0 0 0

0 t1 0

0 0 t2

2 translations, 1 rotation in 2D∂Tx∂t0

∂Tx∂t1

∂Tx∂θ

∂Ty∂t0

∂Ty∂t1

∂Ty∂θ

=

[t0 0 −x sin θ − y cos θ

0 t1 x cos θ − y sin θ

]



Gradient descent

= steepest (gradient) descent = Cauchy, 1847Minimize f along the gradient, i.e.

δk+1 = −∇f (xk )

Step size?Fixed α ∈ R+

Function of gradient norm, e.g. α/||∇f (xk )||Line search

⇒ ∇f (xk ).∇f (xk−1) = 0⇒ Orthogonal steps



Gradient descent - Examples



Gradient descent - Examples

Full Zoom



Gradient descent - Examples (Wikipedia)



Conjugate gradient

Line search in the direction of the gradient

Construct a conjugate direction to previous gradient

δk+1 = ∇f (xn+1) + αkδk

If f quadratic, δk+1 are mutually conjugate⇒ Converge in n iterations



Conjugate gradient

Let gk = −∇f (xk ).

Fletcher and Reeves

αk =gk+1.gk+1

gk .gk=||gk+1||2

||gk ||2

Polak and Ribiere (faster)

αk =(gk+1 − gk ).gk+1

gk .gk

Stopping criterion: gk+1 = 0



Newton’s method in 1D

We seek the step a to go to the optimal position. Taylorserie:

f (x + a) = f (x) + f ′(x) · a +12

f ′′(x) · a2 + ...

Extremum if quadratic (right part)

f ′(x) + f ′′(x)a = 0

⇒ Go to the optimum in one step (if quadratic)

x = xn −f ′(x)

f ′′(x)



Newton’s method in nD

If quadratic, go to optimum in one step

x = xn − H−1 · ∇f (x)

⇒ The search direction is δk = H−1 · ∇f (x)

In practice, not quadratic⇒ step α

xn+1 = xn − αH−1 · ∇f (x)



Newton’s method in nD

Hessian = matrix of second derivatives

If quadratic, converges in one iteration

But computing the Hessian matrix costs too muchand it is not quadratic...

⇒ Approximate Hessian⇒ Use as search direction



Quasi-Newton

Also called variable metric methods

Stores more information than conjugate gradient, e.g.n × n instead of n

Use f (xn) and ∇f (xn) to approximate Hessian

Different solutionsDFP (Davidon-Fletcher-Powell)BFGS (Broyden-Fletcher-Goldfarb-Shanno) (superior?)



Levenberg-Marquardt

Least-Squares methodVaries smoothly between

Steepest-descent far from minimumQuasi-Newton close to minimum

Solve at each iteration Hessian approximation

(Hk + λk I) · δxk

with I the identityFar from minimum : λk large→ Hk + λk I ≈ IClose to minimum : λk small→ Hk + λk I ≈ Hk



Summary

Without gradient:SimplexPowell (conjugate directions)

With gradient:Gradient descentConjugate gradientQuasi-NewtonLevenberg-Marquardt

Also:Genetic algorithmsSimulated annealing. . .



Multiresolution

ConvergenceLarge deformationsRobustness to noise andlocal minima(smooth upper levels)



Comparison [Maes, MedIA, 1999]

High resolution images (256× 256× 100), < 1mm

Mutual information

Partial volume interpolation

3 resolutions

1260 registrations



Comparison [Maes, MedIA, 1999]



Comparison

Full resolutionIterations:SMP, POW, LM, CG < SD, QNTime:POW, SMP < CG < LM < SD, QN

With multiresolutionIterations:CG, SMP, LM < POW, SD, QNTime:SMP < CG, LM, POW < SD, QN

Acceleration factor: 2 to 6 (5− 15min)



Reference

Numerical recipes: www.nr.com

... with proper care, sometimes outdated or bugged, always double-check validity


www.nr.com


Outline

1 Introduction

2 Transformations


4 Optimization

5 Validation

6 Conclusion



Validation

Clinical data Cadavers Physical phantoms Realisticsimulations

Numericalsimulations

EasyControl − −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−→ +

ClinicalRealism +←−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− −



Validation - Anthropomorphic phantom



Online datasets / Challenges / Comparison studies

Example: Retrospective Image Registration EvaluationVanderbilt university (Fitzpatrick)

Multimodal PET / CT / MRI

Head

Markers removed from images during registration



Outline

1 Introduction

2 Transformations


4 Optimization

5 Validation

6 Conclusion



Time to implement - www.itk.org?

T = arg maxT∈T

S(I , J,T )

...or use existing solutions (elastix)



Potential research in each compartment

Optimization

Similarity measure

Parametrization

Implementation

. . .


(rigid/affine) image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (rigid/afﬁne) image...

Documents