![Page 1: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/1.jpg)
(Rigid/Affine)Image registration
Simon Rit12
1CREATIS laboratory
2Leon Berard center
Master EEAP / SI - Module 5 - 2012
![Page 2: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/2.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Module 5: Registration and Motion Estimation
RegistrationRigid/Affine: 4h, S. RitNon-rigid: 2h, D. SarrutPractical exercises: 2h, S. Rit and D. Sarrut
Motion estimationClassical approaches, discontinuities: 4h, P. ClarysseApplication to ultrasound imaging: 2h, H. LiebgottPractical exercises: 2h, P. Clarysse and S. Rit
Evaluation: practical exercises and test
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 2
![Page 3: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/3.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Outline
1 Introduction
2 Transformations
3 Similarity measures
4 Optimization
5 Validation
6 Conclusion
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 3
![Page 4: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/4.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Image registration
DefinitionThe process of aligning images so that corresponding featurescan easily be related.
= Registration= Alignment= Geometrical correspondence= Matching= Motion estimation
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 4
![Page 5: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/5.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Motivation
Many applicationsMedical imagingVisionetc.
Basic elements are used in many applicationsInterpolationSimilarity measureOptimizationetc.
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 5
![Page 6: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/6.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Goal
Find a spatial transformation T that matches two images.
T
Image 1 Image 2
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 6
![Page 7: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/7.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Example #1: panoramic photography
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 7
![Page 8: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/8.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Example #1: panoramic photography
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 7
![Page 9: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/9.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Example #1: panoramic photography
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 7
![Page 10: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/10.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Example #2: satellite pictures (Google maps)
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 8
![Page 11: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/11.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Example #2: satellite pictures (Google maps)
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 8
![Page 12: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/12.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Example #2: satellite pictures (Google maps)
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 8
![Page 13: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/13.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Example #2: satellite pictures (Google maps)
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 8
![Page 14: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/14.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Example #3: multimodal 3D medical images
Registration is a prerequisite for adequate image fusion
MRI PET
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 9
![Page 15: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/15.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Example #3: multimodal 3D medical images
Registration is a prerequisite for adequate image fusion
Before registration After registration
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 9
![Page 16: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/16.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Example #4: image-guidance
CT scanner In-room cone-beam CT
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 10
![Page 17: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/17.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Example #4: image-guidance
CT X-ray projection Cone-beam CT
3D/2D (one projection) or 3D/3D (reconstruction) registration
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 11
![Page 18: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/18.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Summary
The input images may differ in
ResolutionTimeSpace (2D, 2D+t, 3D, 3D+t...)Modality (Photo, Radiography, CT, PET, MRI, US...)Subject (Inter-patient registration)...
There are many possible output transformations.
⇒ Many algorithms...
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 12
![Page 19: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/19.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Formalism
T = arg maxT∈T
S(I , J,T )
NotationI = reference / fixed imageJ = moving / floating imageT = transformationT = search spaceS = similarity measurearg max = optimizationT = solution
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 13
![Page 20: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/20.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Digital images
Discrete set of pixels/voxels
Pixels = 2D Voxels = 3D
We consider two images I(i), J(j) ∈ R.i , j ∈ Z3 are the spatial indices of the lattice.
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 14
![Page 21: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/21.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Digital images
Spatial resolutionEx: 512× 512 pixels of 0.5× 0.5 mm2
Pixel valuesScalar (e.g. CT) or vector (e.g. RGB) or matrix (e.g.diffusion MRI) at each pointDigital number(s) (integer, float...) ⇒ limited precision
Visualization according to a color scaleDefined with window/level for linear gray scales
Coordinate systemOriginOrientation (e.g., with respect to anatomical orientation:cranio-caudal, left-right, antero-posterior).
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 15
![Page 22: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/22.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Outline
1 Introduction
2 Transformations
3 Similarity measures
4 Optimization
5 Validation
6 Conclusion
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 16
![Page 23: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/23.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Formalism
T = arg maxT∈T
S(I , J,T )
NotationI = reference / fixed imageJ = moving / floating imageT = transformationT = search spaceS = similarity measurearg max = optimizationT = solution
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 17
![Page 24: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/24.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Transformations
Cartesian coordinates: x = (x , y , z, t)T
Transformation :T : R4 → R4
x → T (x) = x ′
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 18
![Page 25: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/25.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Search space T
One point of T is a transformation T . It depends on theDegrees of freedom = number n of parameters needed todescribe the transformation. n is the dimension of theoptimization problem.Boundaries of each parameter (if any).
The success of the registration requires thatThe solution T is in T .T is small enough with respect to the input data.
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 19
![Page 26: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/26.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Categories
Linear registration: T (ax1 + x2) = aT (x1) + T (x2)(more exactly, composition of linear transformations withtranslations)
Rigid (translations + rotations)Rigid + ScalingAffine
Non-linear registrationNon-rigidDeformableElasticFluid
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 20
![Page 27: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/27.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Rigid registration
A rigid transformation can be described with translations androtations.
The dimensionality can be adapted2D/2D→ 3 parameters (2 translations, 1 translation)3D/3D→ 6 parameters (3 rotations, 3 translations)2D/3D→ 6 parameters (3 rotations, 3 translations +projection operator).
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 21
![Page 28: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/28.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Matrix representations
Rigid / affine transformations can be represented with matrices:
Tlinear (x) = Rx + t
where R is the rotation matrix and t the translation vector (R is3× 3 and t ∈ R3 in 3D).
For rigid transformations, R is constrained to have only 3parameters. If one uses all 9 parameters of R, T is an affinetransformation.
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 22
![Page 29: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/29.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Homogeneous coordinates
For practicality, one can use a single matrix with homogeneouscoordinates:
Tlinear (x) = Mx
where x is the point in homogeneous coordinates and Mcombines R and t .
In 3D, we would have x = (x , y , z,1) and the matrix
M =
R00 R01 R02 t0R10 R11 R12 t1R20 R21 R22 t20 0 0 1
Also useful to apply projection transforms.
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 23
![Page 30: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/30.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Matrix manipulation
Application: T (x) = Mx
Inverse (if invertible): T−1(x) = M−1x
Composition: T2(T1(x)) = (T2 T1)(x) = M2M1x
etc.
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 24
![Page 31: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/31.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Parameterization
Affine = all possible matrices = 12 parameters in 3D
M =
R00 R01 R02 t0R10 R11 R12 t1R20 R21 R22 t20 0 0 1
For registration, one canOptimize the 12 parameters of the matrixDecompose the matrix in more meaningful parameters, e.g.translations, rotations, ...
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 25
![Page 32: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/32.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Translations
In 2D, 2 translation parameters
M =
1 0 t00 1 t10 0 1
In 3D, 3 translation parameters
M =
1 0 0 t00 1 0 t10 0 1 t20 0 0 1
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 26
![Page 33: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/33.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
2D rotation around the origin
One parameter, the angle θ:
R =
cos θ − sin θ 0sin θ cos θ 0
0 0 1
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 27
![Page 34: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/34.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
2D rotation around another point
One can combine a translation (=change of coordinate system)with this rotation matrix to have a rotation around another pointp = (p0,p1):
R′ =
1 0 p00 1 p10 0 1
cos θ − sin θ 0sin θ cos θ 0
0 0 1
1 0 −p00 1 −p10 0 1
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 28
![Page 35: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/35.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
3D rotation
Rotation = 3× 3 matrix (Euler’s rotation theorem)
3 degrees of freedom (parameters)
Constraints: orthogonal matrix (RRT = Id) and det(R) = 1⇒ Inverse: R−1 = RT
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 29
![Page 36: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/36.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
3D rotation
Several decompositions from the 9 matrix values to 3parameters, e.g.
Euler anglesAxis-angleQuaternionsetc. (see http://en.wikipedia.org/wiki/Charts_on_SO(3) for others)
⇒ The parameters have a different meaning in each case
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 30
![Page 37: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/37.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
3D rotation – Euler angles
Composition of 3 rotations around the axis of thecoordinate system (parameters α, β, γ or θ, ϕ, ψ or Yaw,Pitch and Roll / Roulis, Tangage et Lacet):
RX =
1 0 00 cosα − sinα0 sinα cosα
RY =
cos β 0 sin β0 1 0
− sin β 0 cos β
RZ =
cos γ − sin γ 0sin γ cos γ 0
0 0 1
Order matters⇒ several conventions!Euler angles: ZXZ (XYX, XZX, YXY, YZY, ZYZ)Tait-Bryan angles: YXZ (XYZ, XZY, YZX, ZXY, ZYX)
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 31
![Page 38: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/38.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
3D rotation – The ZXY convention
http://en.wikipedia.org/wiki/Euler_angles
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 32
![Page 39: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/39.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
3D rotation – Issues with Euler angles
Convention issuesGimbal lock = singularity
http://en.wikipedia.org/wiki/Gimbal_lock
Can be solved by changing the convention (other order) oradding a 4th gimbal.
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 33
![Page 40: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/40.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
3D rotation – Axis-angle
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 34
![Page 41: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/41.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
3D rotation – Axis-angle
From Euler’s rotation theorem4 parameters
Unit vector n = (nx ,ny ,nz)T
Angle θ
but 3 degrees of freedom because unit vector (||n|| = 1).
Computation of the new position from parameters withRodrigues’ rotation formula:http://mathworld.wolfram.com/RodriguesRotationFormula.html
LimitationsRepresentation not minimalDifficult to combine two rotations
Euler parameters refer to the axis-angle representation andare different from the Euler angles...
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 35
![Page 42: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/42.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
3D rotation – Quaternions
4 parameters q = (q0,q1,q2,q3)T but 3 DOF |q|2 = 1q0q0 + q1q1 − q2q2 − q3q3 2(q1q2 − q0q3) 2(q1q3 + q0q2)
2(q2q1 + q0q3) q0q0 − q1q1 + q2q2 − q3q3 2(q2q3 − q0q1)2(q3q1 + q0q2) 2(q3q2 + q0q1) q0q0 − q1q1 − q2q2 + q3q3
Link with angle-axis: q =
cos(θ/2)
nx sin(θ/2)ny sin(θ/2)nz sin(θ/2)
Advantage: combine rotations, stability
Inconvenient: 4 non-independent parameters
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 36
![Page 43: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/43.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Scaling
In 3D, 3 parameters sx , sy , sz
R =
sx 0 00 sy 00 0 sz
Useful if working in voxel coordinates, e.g.
Image #1: dxI × dyI × dzI mm3
Image #2: dxJ × dyJ × dzJ mm3
R =
dxIdxJ
0 00 dyI
dyJ0
0 0 dzIdzJ
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 37
![Page 44: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/44.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Shear
In 3D, 6 parameters sxy , syx , sxz , szx , syz , szy
R =
1 syx szxsxy 1 szysxz syz 1
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 38
![Page 45: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/45.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Skew
Idem as shear but antisymmetric (RT = −R)
⇒ In 3D, 3 parameters syx , szx , szy
R =
1 syx szx−syx 1 szy−szx −szy 1
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 39
![Page 46: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/46.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Projection – Matrix
(n − 1)× n matrixParallel or perspective2D/1D perspective projection in homogeneouscoordinates:
M =
[SDD 0 0
0 1 SID
]SDD: Source to Detector DistanceSID: Source to Isocenter Distance
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 40
![Page 47: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/47.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Summary
Linear matrices can be decomposed inhttp://mathworld.wolfram.org/
http://www.wikipedia.com/
Translation parameters: straightforward
Rotation parameters: need caution, several solutions
Scaling parameters: straightforward
Skew parameters: less used
+Projection for 2D/3D registration
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 41
![Page 48: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/48.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Summary
Only a subset of the parameters can (should?) be optimized.
Rigid: 6 parameters (translations and rotations)
Rigid + global rescaling: 7 parameters
Rigid + independent rescaling: 9 parameters
Affine: 12 parameters (translations, rotations, scaling,skew)
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 42
![Page 49: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/49.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Tips and tricks
Carefully chose the search space
As little parameters as possible...
... but make sure that the sought solution is in it!
Bound the search space, e.g. translations < 2 cm androtations < 10
Scale the parameters which are heterogeneous, e.g.translations in mm and rotations in radians or degrees
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 43
![Page 50: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/50.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Outline
1 Introduction
2 Transformations
3 Similarity measures
4 Optimization
5 Validation
6 Conclusion
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 44
![Page 51: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/51.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Formalism
T = arg maxT∈T
S(I , J,T )
NotationI = reference / fixed imageJ = moving / floating imageT = transformationT = search spaceS = similarity measurearg max = optimizationT = solution
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 45
![Page 52: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/52.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Similarity measure
Measure of the alignment quality of I and J with T .
Two families1 Feature-based registration2 Intensity-based registration
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 46
![Page 53: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/53.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Features
Features are extracted from images prior to similarity measure
PointsLinesVectorsSurfacesVolumes...
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 47
![Page 54: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/54.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Feature-based registration formalism
T = arg maxT∈T
S(FI , T FJ)
NotationFI = feature set of reference / fixed imageFJ = feature set of moving / floating imageT = transformationT = search spaceS = similarity measurearg max = optimizationT = solution
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 48
![Page 55: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/55.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Geometric primitives – Manual identification
Landmarks (recognizable points) clicked by an expert.
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 49
![Page 56: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/56.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Geometric primitives – Manual identification
Manual contours.
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 50
![Page 57: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/57.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Geometric primitives – Automatic identification
Segmentation⇒ Another problem which can be very difficult...
Unpaired features⇒ Unpaired similarity measure, e.g. pair to closest feature in
other image⇒ Or find an algorithm to pair features (another problem...)
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 51
![Page 58: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/58.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Manual vs auto: pros and cons
Manual+ Reference- Long- Expensive (requires time of an expert, e.g. physician)- Inter- and intra- observer variability
Automatic+ Fast and cheap- Paired or unpaired?? Reproducible? Robust
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 52
![Page 59: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/59.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
(Dis-)similarity measures for features
Sum of distances
L1-norm: |x |1 =n∑
r=1
|xr |
L2-norm: |x |2 =
√√√√ n∑r=1
x2r
Quadratic sum (faster):n∑
r=1
x2r
Other distances...
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 53
![Page 60: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/60.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Distance map
Map of the distance between each point and the closest feature⇒ Fast computation of distances for unpaired features
Image Features = bones Distance map
Computed only once for the reference image.Fast computations: Chamfer distance (not Euclidian),separable algorithms [Coeurjolly, PAMI, 2007].
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 54
![Page 61: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/61.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
S(FI , T FJ) or S(FJ , T FI)?
This is not identical sincewith the distance map dI
S(I, J,T ) =∑
F∈FJ
dI T (F )
with the distance map dJ
S(I, J,T ) =∑
F∈FI
dJ T (F )
⇒ Use the distance map of the image which has featureseasier to extract
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 55
![Page 62: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/62.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Summary of feature-based registration
- Preliminary step to identify features- Pairing algorithm maybe required+ Generally fast (depending on the feature type and size)+ Registration of some features only...
There is currently a renewed interest in feature-basedregistration, particularly for non-rigid registration...
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 56
![Page 63: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/63.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Limitation of feature based registration
Multimodal images
⇒ Intensity-based registration
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 57
![Page 64: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/64.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Intensity-based registration formalism
T = arg maxT∈T
S(I , J T )
NotationI = reference / fixed imageJ = moving / floating imageT = transformationT = search spaceS = similarity measurearg max = optimizationT = solution
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 58
![Page 65: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/65.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Intensity-based similarity measure
S(I , J T ) is a measure which assumes a func-tional dependence between the pixel intensities
Intensity-based registration = iconic registrationChoice depends on the modalities
= Functional dependence
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 59
![Page 66: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/66.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Forward and backward warping
Required for computing J T
Image flottante
Image de référence
Zone d’interpolation
Forward mapping
Backward mapping
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 60
![Page 67: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/67.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Forward and backward warping
Forward mapping requires an additional weight map+ potential holes
⇒ Backward is usually preferred. Not a problem:T is easily invertible for affine registrationOne can always optimize T−1 and invert it at the end
The resolution of the reference image is used
More: Wolberg, Digital image warping, 1990
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 61
![Page 68: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/68.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Warping – Fast computation
∀ voxel x ∈ ΩI :
M x =
a b c de f g hi j k l0 0 0 1
×
xyz1
=
x ′ = ax + by + cz + dy ′ = ex + fy + gz + hz ′ = ix + jy + kz + l
1
Use increment in innermost loop (along x here):
x ′n+1 = x ′n + a and x ′0 = by + cz + dy ′n+1 = y ′n + e and y ′0 = fy + gz + hz ′n+1 = z ′n + i and z ′0 = jy + kz + l
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 62
![Page 69: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/69.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Interpolation
Nearest neighbor
Linear
More accurate: spline, sinc...
⇒ Compromise between speed and accuracy
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 63
![Page 70: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/70.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
2D/3D: Digitally Reconstructed Radiograph
Based on the Beer-Lambert law:Φ(u, v) = Φ0 exp
−∫
Lβ,u,vf (x)dx
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 64
![Page 71: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/71.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Intensity-based similarity measures
Sum of Squared Difference
SSD(I, J) =∑x∈Ω
(I(x)− J(x))2
Correlation coefficient
CC(I, J) = N∑
(I(x)−mI)(J(x)−mJ)∑√I(x)−mI
∑√J(x)−mJ
Mutual information
MI(I, J) =∑i,j
pij logpij
pi pj
Correlation ratio
CR(I, J) = η2(I|J) = 1− 1σ2
I
∑j
pjσ2I|j
Many others...
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 65
![Page 72: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/72.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Sum of Squared Difference
SSD(I, J) =∑x∈Ω
(I(x)− J(x))2
Dissimilarity measureFastNeeds to be normalized to Ω size (in voxels)Functional dependence between intensities: identity
Optimal if images only differ by Gaussian noise(monomodal)Not robust otherwise (multimodal)
Similar: Sum of Absolute Difference (SAD)
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 66
![Page 73: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/73.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
(Pearson product-moment) Correlation coefficient
CC(I, J) = ρ(I, J) =cov(I, J)
σI σJ= N
∑(I(x)−mI)(J(x)−mJ)∑√I(x)−mI
∑√J(x)−mJ
Range [−1,1]
1 is perfect increasing linear relationship-1 is perfect decreasing linear relationship0 if independent
⇒ Similarity measure: |CC(I, J)|Fast (with computational tricks)Linear dependence: I(x) = a J(T (x)) + b ∀x a,b ∈ R
More robust than SSD, e.g. for 2 imaging systems of asame modalityNot good for multimodal
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 67
![Page 74: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/74.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Functional dependence between intensities
SSD : I(x) = J(T (x)) ∀xCC : I(x) = a J(T (x)) + b ∀x
⇒ Other functional relationship? ⇒ Information theory
Mutual informationCoefficient ratio
Both require joint histograms.
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 68
![Page 75: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/75.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Histogram
0 50 100 150 200 2500
0.5
1
1.5
2
2.5
3x 10
4
Pixel intensity
Nu
mb
er
of
pix
els
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 69
![Page 76: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/76.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Joint histogram – Definition
J colors
i
j nij
∑j nij
∑i nij ni
nj
I colors
i , j : (binned) colorsnij : number of pixels withcolor i in I and j in Jni , nj : marginal values, i.e.,histograms of I and J
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 70
![Page 77: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/77.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Joint histogram
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 71
![Page 78: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/78.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Joint histogram
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 72
![Page 79: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/79.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Joint histogram
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 73
![Page 80: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/80.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Joint histogram
2D distribution of the pair of intensities at each voxellocation
SSD and CC can also be understood with joint histograms
. . . but it is useless to build it for SSD and CC
Recomputed at each step of the optimization
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 74
![Page 81: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/81.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Joint histogram – Example #2
From [Roche, PhD, 2001]
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 75
![Page 82: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/82.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Joint histogram – Computation
Two steps:
1 Count the nij (accounting for T )Compute the warped image J T on I gridCompute the joint histogram of J T and I
Or directly update the histogram pixel-by-pixel
2 Derive the probability pij
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 76
![Page 83: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/83.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Joint histogram
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 77
![Page 84: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/84.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Mutual information – Entropy
Measure of information as a registration metric⇒ Entropy (Shannon-Wiener):
H =∑
i
pi log1pi
= −∑
i
pi log pi
i
p(i
)
Entropy=3.00
i
p(i
)
Entropy=2.99
i
p(i
)
Entropy=2.11
i
p(i
)
Entropy=−0.00
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 78
![Page 85: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/85.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Mutual information – Joint entropy
Two images⇒ two symbols at each grid point
H = −∑i,j
pij log pij
⇒ Joint entropy: the more similar the distributions, the lowerthe joint entropy compared to the sum of individualentropies
H(I, J) ≤ H(I) + H(J)
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 79
![Page 86: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/86.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Mutual information
MI(I, J) =∑i,j
pij logpij
pi pj
MI(I, J) = H(I)− H(J|IJ) = H(J)− H(I|J) = H(I) + H(J)− H(I, J)
PositiveSymmetric MI(I, J) = MI(J, I)H(J|I) = H(I, J)− H(I): conditional entropyExample [Pluim, TMI, 2003]:
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 80
![Page 87: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/87.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Mutual information is a reference
First articles: June 1995Maes in BelgiumViola in US
First journal articles: 1997
Currently: #1 similarity measureMultimodalRobust
Among the most cited papers of the domain...
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 81
![Page 88: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/88.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
MI implementation
For mutual information, implementation matters (as usual!).Survey of methods: [Pluim, TMI, 2003].Specifically:
InterpolationBinningProbability computationNormalization (less sensitive to overlap changes)e.g. Normalized Mutual Information [Studholm, PatternRecognition, 1999]
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 82
![Page 89: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/89.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
MI implementation – Interpolation
Conventional: nearest neighbor, linear...
Partial volume interpolation (Collignon)
The difference lies in the smoothness of the optimizedfunction
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 83
![Page 90: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/90.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
MI implementation – Interpolation
NN LIN
PV
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 84
![Page 91: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/91.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
MI implementation – Number of bins
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 85
![Page 92: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/92.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
MI implementation – Probability
Frequential approach (simplest): pij =nijN
Parzen window: pij =∑
k
ϕ(nij − nk )
ϕ =Gaussian [Viola, PhD, 1995]ϕ =Cubic spline [Thevenaz, 2000]
Also Bayesian approach...
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 86
![Page 93: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/93.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Correlation ratio
RC(I, J) = η2(I|J) = 1− 1σ2
I
∑j
pjσ2I|j
Based on joint histogram as wellAssume a functional dependence between intensitiesVariance instead of entropyResults comparable to MI, not symmetricftp://ftp.inria.fr/INRIA/publication/RR/RR-3980.pdf
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 87
![Page 94: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/94.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Summary of intensity-based similarity measures
The measures can be classified with respect to hiddenvariables [Malandain, HDR, 2006]
SSD, SAD: 0 variableCC: 2 variables, slope and interceptCR: size of one side of the joint histMI: product of the size of the sides of the joint hist
⇒ MI is more general, hence its success⇒ But it is also more susceptible to fail in simple situations
where SSD is sufficient, e.g., time series
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 88
![Page 95: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/95.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Local affine registration
One can always use a region-of-interest (ROI) of the image forregistration (e.g. rectangular box or shaped volume of interest)
Applied to the reference and/or the target image depending onthe similarity measure.
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 89
![Page 96: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/96.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Outline
1 Introduction
2 Transformations
3 Similarity measures
4 Optimization
5 Validation
6 Conclusion
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 90
![Page 97: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/97.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Formalism
T = arg maxT∈T
S(I , J,T )
NotationI = reference / fixed imageJ = moving / floating imageT = transformationT = search spaceS = similarity measurearg max = optimizationT = solution
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 91
![Page 98: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/98.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Challenge
Find the global minimum... as fast as possible
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 92
![Page 99: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/99.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
1D illustration
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 93
![Page 100: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/100.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Optimization
Optimization = maximize or minimize
Search for global minimum⇒ if the max is sought, thenminimize −f
Optimization 1D / nD (up to n = 12 for affine registration)
With or without a gradient of fThe solution respects ∇f (x) = 0 (as well as other extrema)
Multi-resolution
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 94
![Page 101: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/101.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Optimization
Iterative process: iteration xk / f (xk+1) < f (xk )
Stopping criterion|f (xk+1)− f (xk )| < ε∇f (x) = 0
One iteration⇒ search δxk / xk+1 = xk + δxk
Initialization x0IdentityAlign image centersAlign mass centers. . .
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 95
![Page 102: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/102.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
1D optimization: Golden search
Without derivatives
Based on the bisection method (root search)Based on the intermediate value theorem
Bracketed solutionIf a < b < c such that f (b) < f (a) and f (b) < f (c) then fhas a minimum in the interval (a, c).
⇒ 4 values at each iteration (position based on Goldennumber)a < b < x < c ora < x < b < c
⇒ x replaces either a or c depending on where is theminimum.
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 96
![Page 103: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/103.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
1D optimization: Brent’s method
Without derivatives
Based on Dekker’s method, 1969Bisection / Golden search methodSecant methodLinear interpolation
Brent’s method, 1973+ Inverse quadratic interpolation+ Additional tests⇒ 6 values at each step
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 97
![Page 104: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/104.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Brent’s method - Illustration
Three points bracket the intervalInverse quadratic interpolationIterate and stop when f (x3)− f (x1) < ε
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 98
![Page 105: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/105.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Brent’s method - Illustration
Three points bracket the interval
Inverse quadratic interpolationIterate and stop when f (x3)− f (x1) < ε
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 98
![Page 106: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/106.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Brent’s method - Illustration
Three points bracket the intervalInverse quadratic interpolation
Iterate and stop when f (x3)− f (x1) < ε
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 98
![Page 107: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/107.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Brent’s method - Illustration
Three points bracket the intervalInverse quadratic interpolation
Iterate and stop when f (x3)− f (x1) < ε
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 98
![Page 108: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/108.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Brent’s method - Illustration
Three points bracket the intervalInverse quadratic interpolationIterate and stop when f (x3)− f (x1) < ε
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 98
![Page 109: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/109.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
nD optimization
Without gradient With gradientSimplex Gradient descent δk+1 = −∇f (xk )Powell Conjugate gradient δk+1 = −∇f (xk ) + βk dk
(Newton) δk+1 = −H−1 .∇f (xk )Quasi-Newton δk+1 = −A−1 .∇f (xk )Levenberg-Marquardt SD/QN
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 99
![Page 110: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/110.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Downhill simplex
Nelder and Meade, 1965Simplex = geometric object(nondegenerate)n + 1 verticesBased on pseudo-derivatives
Not to be mixed with the simplexmethod in linear programming
1 Compute the function (similaritymeasure) at each vertex
2 Conditional deformation (see next slide)
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 100
![Page 111: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/111.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Downhill simplex
https://wiki.ece.cmu.edu/ddl/index.php/Downhill_Simplex
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 101
![Page 112: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/112.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Downhill simplex
AlgorithmSort f (x i) with i the vertex numberTake worst and replace with (if better)
1 Reflection2 Expansion3 Contraction4 Shrink
Iterate until convergence...
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 102
![Page 113: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/113.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Downhill simplex - Examples
https://wiki.ece.cmu.edu/ddl/index.php/Downhill_Simplex
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 103
![Page 114: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/114.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Downhill simplex - Examples
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 104
![Page 115: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/115.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Line search along unit vectors
(Numerical Recipes)
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 105
![Page 116: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/116.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Line search
Used by many nD strategies= 1D optimization in a direction of the nD space
⇒ Given a set of parameters x ∈ Rn and a direction y ∈ Rn,find the best α ∈ R so that x + αy is minimal.
Typically: Brent’s method
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 106
![Page 117: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/117.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Powell’s method
Powell’s conjugate gradient descent method (1964)Set of n directions, e.g. the unit vectorsMove along one direction until minimum
Line search
Loop over directions until minimumUpdate set of directions
Displacement vector becomes a (conjugate) directionRemove direction which contributed the most
Order matters, e.g. [Maes, 1999] (tx , ty , φz , tz , φx , φy ).Converge in n(n + 1) iterations for quadratic functions
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 107
![Page 118: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/118.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Conjugate directions
Directions which will not spoil previous minimizations
Let δk and δk+1 be two successive search directions
f has been minimized in the δk direction
⇒ The projection of the gradient on δk is ~0
⇒ Chose a perpendicular direction
⇒ δk+1 will not interfere with the minimization along δk .
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 108
![Page 119: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/119.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Conjugate directions properties
If f is quadratic, one pass through the set of directionsresults in the exact minimumElse, quadratic convergence
Powell’s method constructs a conjugate direction at eachiteration
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 109
![Page 120: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/120.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Powell - Example
http://www.mathworks.com/matlabcentral/fileexchange/authors/26509
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 110
![Page 121: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/121.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
nD optimization
Without gradient With gradientSimplex Gradient descent δk+1 = −∇f (xk )Powell Conjugate gradient δk+1 = −∇f (xk ) + βk dk
(Newton) δk+1 = −H−1 .∇f (xk )Quasi-Newton δk+1 = −A−1 .∇f (xk )Levenberg-Marquardt SD/QN
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 111
![Page 122: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/122.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Gradient optimization
Gradient vector:
∇f =
∂f∂x1∂f∂x2
. . .
∈ Rn
Hessian matrix:
H =
∂2f∂x2
1
∂2f∂x1∂x2
. . .
∂2f∂x2∂x1
∂2f∂x2
2. . .
......
. . .
∈ Rn×n
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 112
![Page 123: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/123.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Gradient optimization
Expected gain: O(n) instead of O(n2)
Problem: computing partial derivatives⇒ Is it worth the effort?
But:there might be redundancies in ∇fline search might be faster
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 113
![Page 124: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/124.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Gradient optimization - 1D example
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 114
![Page 125: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/125.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Gradient optimization - 1D example
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 114
![Page 126: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/126.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Gradient optimization - 1D example
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 114
![Page 127: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/127.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Gradient optimization - 1D example
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 114
![Page 128: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/128.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Gradient optimization - 1D example
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 114
![Page 129: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/129.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Gradient optimization - 1D example
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 114
![Page 130: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/130.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Gradient optimization - 1D example
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 114
![Page 131: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/131.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Gradient optimization - 1D example
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 114
![Page 132: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/132.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Gradient optimization - 1D example
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 114
![Page 133: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/133.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Computing the gradient in nD
Feature-based registration: almost impossibleDepends on the segmentation...
⇒ Iterative Closest Point [Zhang, IJCV, 1994]?
Intensity-based registration: composition of image andtransform, e.g. J T ⇒ use chain rule.In 1D: f g′(x) = f ′(g(x)) · g′(x). In our case:
∂J T (x)
∂p=∂J(T (x))
∂x∂T∂p
Result is a combination ofImage gradientDerivatives of the transform for each parameter in eachdirection (m × n Jacobian matrix in mD for n parameters)
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 115
![Page 134: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/134.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Image gradient
Finite difference(s) (approximate), e.g. (f (x + h)− f (x))/hDeriche recursive gaussian filters and its derivativeshttp://hal.inria.fr/inria-00074778/en/
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 116
![Page 135: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/135.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Derivation of similarity measures
Measure specific
SSD : [Thevenaz, IEEE IP, 1998]
MI : [Viola, PhD, 1995], [Thevenaz, IEEE IP, 2000]
CR : [Cachier, PhD, 2002]
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 117
![Page 136: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/136.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Jacobian matrix examples
3 translations in 3D∂Tx∂t0
∂Tx∂t1
∂Tx∂t2
∂Ty∂t0
∂Ty∂t1
∂Ty∂t2
∂Tz∂t0
∂Tz∂t1
∂Tz∂t2
=
t0 0 0
0 t1 0
0 0 t2
2 translations, 1 rotation in 2D∂Tx∂t0
∂Tx∂t1
∂Tx∂θ
∂Ty∂t0
∂Ty∂t1
∂Ty∂θ
=
[t0 0 −x sin θ − y cos θ
0 t1 x cos θ − y sin θ
]
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 118
![Page 137: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/137.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Gradient descent
= steepest (gradient) descent = Cauchy, 1847Minimize f along the gradient, i.e.
δk+1 = −∇f (xk )
Step size?Fixed α ∈ R+
Function of gradient norm, e.g. α/||∇f (xk )||Line search
⇒ ∇f (xk ).∇f (xk−1) = 0⇒ Orthogonal steps
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 119
![Page 138: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/138.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Gradient descent - Examples
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 120
![Page 139: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/139.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Gradient descent - Examples
Full Zoom
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 121
![Page 140: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/140.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Gradient descent - Examples (Wikipedia)
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 122
![Page 141: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/141.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Conjugate gradient
Line search in the direction of the gradient
Construct a conjugate direction to previous gradient
δk+1 = ∇f (xn+1) + αkδk
If f quadratic, δk+1 are mutually conjugate⇒ Converge in n iterations
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 123
![Page 142: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/142.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Conjugate gradient
Let gk = −∇f (xk ).
Fletcher and Reeves
αk =gk+1.gk+1
gk .gk=||gk+1||2
||gk ||2
Polak and Ribiere (faster)
αk =(gk+1 − gk ).gk+1
gk .gk
Stopping criterion: gk+1 = 0
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 124
![Page 143: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/143.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Newton’s method in 1D
We seek the step a to go to the optimal position. Taylorserie:
f (x + a) = f (x) + f ′(x) · a +12
f ′′(x) · a2 + ...
Extremum if quadratic (right part)
f ′(x) + f ′′(x)a = 0
⇒ Go to the optimum in one step (if quadratic)
x = xn −f ′(x)
f ′′(x)
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 125
![Page 144: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/144.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Newton’s method in nD
If quadratic, go to optimum in one step
x = xn − H−1 · ∇f (x)
⇒ The search direction is δk = H−1 · ∇f (x)
In practice, not quadratic⇒ step α
xn+1 = xn − αH−1 · ∇f (x)
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 126
![Page 145: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/145.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Newton’s method in nD
Hessian = matrix of second derivatives
If quadratic, converges in one iteration
But computing the Hessian matrix costs too muchand it is not quadratic...
⇒ Approximate Hessian⇒ Use as search direction
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 127
![Page 146: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/146.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Quasi-Newton
Also called variable metric methods
Stores more information than conjugate gradient, e.g.n × n instead of n
Use f (xn) and ∇f (xn) to approximate Hessian
Different solutionsDFP (Davidon-Fletcher-Powell)BFGS (Broyden-Fletcher-Goldfarb-Shanno) (superior?)
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 128
![Page 147: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/147.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Levenberg-Marquardt
Least-Squares methodVaries smoothly between
Steepest-descent far from minimumQuasi-Newton close to minimum
Solve at each iteration Hessian approximation
(Hk + λk I) · δxk
with I the identityFar from minimum : λk large→ Hk + λk I ≈ IClose to minimum : λk small→ Hk + λk I ≈ Hk
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 129
![Page 148: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/148.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Summary
Without gradient:SimplexPowell (conjugate directions)
With gradient:Gradient descentConjugate gradientQuasi-NewtonLevenberg-Marquardt
Also:Genetic algorithmsSimulated annealing. . .
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 130
![Page 149: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/149.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Multiresolution
ConvergenceLarge deformationsRobustness to noise andlocal minima(smooth upper levels)
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 131
![Page 150: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/150.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Comparison [Maes, MedIA, 1999]
High resolution images (256× 256× 100), < 1mm
Mutual information
Partial volume interpolation
3 resolutions
1260 registrations
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 132
![Page 151: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/151.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Comparison [Maes, MedIA, 1999]
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 133
![Page 152: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/152.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Comparison [Maes, MedIA, 1999]
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 133
![Page 153: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/153.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Comparison
Full resolutionIterations:SMP, POW, LM, CG < SD, QNTime:POW, SMP < CG < LM < SD, QN
With multiresolutionIterations:CG, SMP, LM < POW, SD, QNTime:SMP < CG, LM, POW < SD, QN
Acceleration factor: 2 to 6 (5− 15min)
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 134
![Page 154: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/154.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Reference
Numerical recipes: www.nr.com
... with proper care, sometimes outdated or bugged, always double-check validity
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 135
![Page 155: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/155.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Outline
1 Introduction
2 Transformations
3 Similarity measures
4 Optimization
5 Validation
6 Conclusion
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 136
![Page 156: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/156.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Validation
Clinical data Cadavers Physical phantoms Realisticsimulations
Numericalsimulations
EasyControl − −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−→ +
ClinicalRealism +←−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− −
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 137
![Page 157: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/157.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Validation - Anthropomorphic phantom
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 138
![Page 158: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/158.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Online datasets / Challenges / Comparison studies
Example: Retrospective Image Registration EvaluationVanderbilt university (Fitzpatrick)
Multimodal PET / CT / MRI
Head
Markers removed from images during registration
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 139
![Page 159: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/159.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Outline
1 Introduction
2 Transformations
3 Similarity measures
4 Optimization
5 Validation
6 Conclusion
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 140
![Page 160: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/160.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Time to implement - www.itk.org?
T = arg maxT∈T
S(I , J,T )
...or use existing solutions (elastix)
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 141
![Page 161: (Rigid/Affine) Image registrationsrit/tete/2012_master_eeap_si_m5.pdf · (Rigid/Affine) Image registration Simon Rit12 1CREATIS laboratory 2Leon B´erard center Master EEAP / SI](https://reader030.vdocuments.net/reader030/viewer/2022040908/5e7f15d4f75eb37f285cfbfe/html5/thumbnails/161.jpg)
Introduction Transformations Similarity measures Optimization Validation Conclusion
Potential research in each compartment
Optimization
Similarity measure
Parametrization
Implementation
. . .
Image registration Master EEAP / SI - Module 5 - 2012 Simon Rit 142