CSci 6971: Image Registration Lecture 5: Feature-Base Regisration

January 27, 2004

CSci 6971: Image Registration Lecture 5: Feature-Base Regisration

January 27, 2004

Prof. Chuck Stewart, RPIDr. Luis Ibanez, KitwareProf. Chuck Stewart, RPIDr. Luis Ibanez, Kitware

Image Registration Lecture 5 2

OverviewOverview

What is feature-based (point-based) registration?

Feature points The correspondence problem Solving for the transformation estimate Putting it all together: ICP Discussion and conclusion

What is feature-based (point-based) registration?

Feature points The correspondence problem Solving for the transformation estimate Putting it all together: ICP Discussion and conclusion

Image Registration Lecture 5 3

What is Feature-Based Registration?What is Feature-Based Registration?

Images are described as discrete sets of point locations associated with a geometric measurement Locations may have additional properties

such as intensities and orientations Registration problem involves two parts:

Finding correspondences between features Estimating the transformation parameters

based on these correspondences

Images are described as discrete sets of point locations associated with a geometric measurement Locations may have additional properties

such as intensities and orientations Registration problem involves two parts:

Finding correspondences between features Estimating the transformation parameters

based on these correspondences

Image Registration Lecture 5 4

Feature Examples: Range DataFeature Examples: Range Data

Range image points: (x,y,z) values Triangulated mesh Surface normals are

sometimes computed Notice:

Some information (locations) is determined directly by the sensor (“raw data”)

Some information is inferred from the data

Range image points: (x,y,z) values Triangulated mesh Surface normals are

sometimes computed Notice:

Some information (locations) is determined directly by the sensor (“raw data”)

Some information is inferred from the data

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Image Registration Lecture 5 5

Feature Examples: Vascular LandmarksFeature Examples: Vascular Landmarks

Branching points pulmonary images: Lung vessels Airway branches Retinal image

branches and cross-over points

Typically augmented (at least) with orientations of vessels meeting to form landmarks

Branching points pulmonary images: Lung vessels Airway branches Retinal image

branches and cross-over points

Typically augmented (at least) with orientations of vessels meeting to form landmarks

Image Registration Lecture 5 6

Points Along Centers of Vessels and AirwaysPoints Along Centers of Vessels and Airways

Airways and vessels modeled as tubular structures

Sample points spaced along center of tubes Note that the entire

tube is rarely used as a unit

Augmented descriptions: Orientation Radius

Airways and vessels modeled as tubular structures

Sample points spaced along center of tubes Note that the entire

tube is rarely used as a unit

Augmented descriptions: Orientation Radius

Image Registration Lecture 5 7

“Interest” Points“Interest” Points

Locations of strong intensity variation in all directions

Augmented with summary descriptions (moments) of surrounding intensity structures

Recent work in making these invariant to viewpoint and illumination.

We’ll discuss interest points during Lectures 16 and 17

Locations of strong intensity variation in all directions

Augmented with summary descriptions (moments) of surrounding intensity structures

Recent work in making these invariant to viewpoint and illumination.

We’ll discuss interest points during Lectures 16 and 17

Brown and Lowe, Int. Conf. On Computer Vision, 2003

Image Registration Lecture 5 8

Feature Points: DiscussionFeature Points: Discussion

Many different possible features Problem is reliably extracting features in all

images This is why more sophisticated features are

not used Feature extraction methods do not use all

intensity values Use of features dominates range-image

registration techniques where “features” are provided by the sensor

Many different possible features Problem is reliably extracting features in all

images This is why more sophisticated features are

not used Feature extraction methods do not use all

intensity values Use of features dominates range-image

registration techniques where “features” are provided by the sensor

Image Registration Lecture 5 9

Preamble to Feature-Based Registration: NotationPreamble to Feature-Based Registration: Notation

Set of moving image features

Set of fixed image features

Each feature must include a point location in the coordinate system of its image. It may include more

Set of correspondences

Set of moving image features

Set of fixed image features

Each feature must include a point location in the coordinate system of its image. It may include more

Set of correspondences

Image Registration Lecture 5 10

Error objective function depends on unknown transformation parameters and unknown feature correspondences Each may depend on the other!

Transformation may include mapping of more than just locations

Distance function, D, could be as simple as the Euclidean distance between location vectors.

We are using the forward transformation model.

Error objective function depends on unknown transformation parameters and unknown feature correspondences Each may depend on the other!

Transformation may include mapping of more than just locations

Distance function, D, could be as simple as the Euclidean distance between location vectors.

We are using the forward transformation model.

Mathematical FormulationMathematical Formulation

Image Registration Lecture 5 11

Correspondence ProblemCorrespondence Problem

Determine correspondences before estimating transformation parameters Based on rich description of features Error prone

Determine correspondences at the same time as estimation of parameters “Chicken-and-egg” problem

For the next few minutes we will assume a set of correspondences is given and proceed to the estimation of parameters Then we will return to the correspondence

problem

Determine correspondences before estimating transformation parameters Based on rich description of features Error prone

Determine correspondences at the same time as estimation of parameters “Chicken-and-egg” problem

For the next few minutes we will assume a set of correspondences is given and proceed to the estimation of parameters Then we will return to the correspondence

problem

Image Registration Lecture 5 12

Example: Estimating ParametersExample: Estimating Parameters

2d point locations:

Similarity transformation:

Euclidean distance:

2d point locations:

Similarity transformation:

Euclidean distance:

Image Registration Lecture 5 13

Putting This TogetherPutting This Together

Image Registration Lecture 5 14

What Do We Have?What Do We Have?

Least-squares objective function Quadratic function of each parameter We can

Take the derivative with respect to each parameter

Set the resulting gradient to 0 (vector) Solve for the parameters through matrix

inversion We’ll do this in two forms: component and

matrix/vector

Least-squares objective function Quadratic function of each parameter We can

Take the derivative with respect to each parameter

Set the resulting gradient to 0 (vector) Solve for the parameters through matrix

inversion We’ll do this in two forms: component and

matrix/vector

Image Registration Lecture 5 15

Component Derivative (a)Component Derivative (a)

Image Registration Lecture 5 16

Component Derivative (b)Component Derivative (b)

At this point, we’ve dropped the leading factor of 2. It will be eliminated when this is set to 0.

Image Registration Lecture 5 17

Component Derivatives tx and tyComponent Derivatives tx and ty

Image Registration Lecture 5 18

GatheringGathering

Setting each of these equal to 0 we obtain a set of 4 linear equations in 4 unknowns. Gathering into a matrix we have:

Setting each of these equal to 0 we obtain a set of 4 linear equations in 4 unknowns. Gathering into a matrix we have:

Image Registration Lecture 5 19

SolvingSolving

This is a simple equation of the form

Provided the 4x4 matrix X is full-rank (evaluate SVD) we easily solve as

This is a simple equation of the form

Provided the 4x4 matrix X is full-rank (evaluate SVD) we easily solve as

Image Registration Lecture 5 20

Matrix VersionMatrix Version

We can do this in a less painful way by rewriting the following intermediate expression in terms of vectors and matrices:

We can do this in a less painful way by rewriting the following intermediate expression in terms of vectors and matrices:

Image Registration Lecture 5 21

Matrix Version (continued)Matrix Version (continued)

This becomes

Manipulating:

This becomes

Manipulating:

Image Registration Lecture 5 22

Matrix Version (continued)Matrix Version (continued)

Taking the derivative of this wrt the transformation parameters (we didn’t cover vector derivatives, but this is fairly straightforward):

Setting this equal to 0 and solving yields:

Taking the derivative of this wrt the transformation parameters (we didn’t cover vector derivatives, but this is fairly straightforward):

Setting this equal to 0 and solving yields:

Image Registration Lecture 5 23

Comparing the Two VersionsComparing the Two Versions

Final equations are identical (if you expand the symbols)

Matrix version is easier (once you have practice) and less error prone

Sometimes efficiency requires hand-calculation and coding of individual terms

Final equations are identical (if you expand the symbols)

Matrix version is easier (once you have practice) and less error prone

Sometimes efficiency requires hand-calculation and coding of individual terms

Image Registration Lecture 5 24

Resetting the StageResetting the Stage

What we have done: Features Error function of transformation parameters

and correspondences Least-squares estimate of transformation

parameters for fixed set of correspondences

Next: ICP: joint estimation of correspondences

and parameters

What we have done: Features Error function of transformation parameters

and correspondences Least-squares estimate of transformation

parameters for fixed set of correspondences

Next: ICP: joint estimation of correspondences

and parameters

Image Registration Lecture 5 25

Iterative Closest Points (ICP) AlgorithmIterative Closest Points (ICP) Algorithm

Given an initial transformation estimate 0

t = 0 Iterate until convergence:

Establish correspondences: For fixed transformation parameter estimate, t,

apply the transformation to each moving image feature and find the closest fixed image feature

Estimate the new transformation parameters, For the resulting correspondences, estimate

t+1

Given an initial transformation estimate 0

t = 0 Iterate until convergence:

Establish correspondences: For fixed transformation parameter estimate, t,

apply the transformation to each moving image feature and find the closest fixed image feature

Estimate the new transformation parameters, For the resulting correspondences, estimate

t+1

ICP algorithm was developed almost simultaneous by at least 5 research groups in the early 1990’s.

Image Registration Lecture 5 26

Finding CorrespondencesFinding Correspondences

Map feature into coordinate system of If

Find closest point

Map feature into coordinate system of If

Find closest point

Image Registration Lecture 5 27

Finding Correspondences (continued)Finding Correspondences (continued)

Enforce unique correspondences Avoid trivial minima of objective function

due to having no correspondences Spatial data structures needed to make

search for correspondences efficient K-d trees Digital distance maps More during lectures 11-15…

Enforce unique correspondences Avoid trivial minima of objective function

due to having no correspondences Spatial data structures needed to make

search for correspondences efficient K-d trees Digital distance maps More during lectures 11-15…

Image Registration Lecture 5 28

Initialization and ConvergenceInitialization and Convergence

Initial estimate of transformation is again crucial because this is a minimization technique

Determining correspondences and estimating the transformation parameters are two separate processes With Euclidean distance metrics you can show

they are working toward the same minimum In general this is not true

Convergence in practice is sometimes problematic and the correspondences oscillate between points.

Initial estimate of transformation is again crucial because this is a minimization technique

Determining correspondences and estimating the transformation parameters are two separate processes With Euclidean distance metrics you can show

they are working toward the same minimum In general this is not true

Convergence in practice is sometimes problematic and the correspondences oscillate between points.

Image Registration Lecture 5 29

2d Retinal Example2d Retinal Example

White = vessel centerline points from one image

Black = vessel centerline points from second image

Yellow line segments drawn between corresponding points

Because of the complexity of the structure, initialization must be fairly accurate

White = vessel centerline points from one image

Black = vessel centerline points from second image

Yellow line segments drawn between corresponding points

Because of the complexity of the structure, initialization must be fairly accurate

Image Registration Lecture 5 30

ComparisonComparison

For a given transformation estimate, we can only find a new, better estimate, not the best estimate, based on the gradient step.

We then need to update the constraints and re-estimate

For a given transformation estimate, we can only find a new, better estimate, not the best estimate, based on the gradient step.

We then need to update the constraints and re-estimate

Intensity-Based Feature-Based

For given set of correspondences, we can directly (least-squares) estimate the best transformation

BUT, the transformation depends on the correspondences, so we generally need to re-establish the correspondences.

For given set of correspondences, we can directly (least-squares) estimate the best transformation

BUT, the transformation depends on the correspondences, so we generally need to re-establish the correspondences.

Image Registration Lecture 5 31

SummarySummary

Feature-based registration Feature types and properties Correspondences Least-squares estimate of parameters based

on correspondences ICP Comparison

Feature-based registration Feature types and properties Correspondences Least-squares estimate of parameters based

on correspondences ICP Comparison

Image Registration Lecture 5 32

Looking Ahead to Lecture 6Looking Ahead to Lecture 6

Introduction to ITK and the ITK registration framework.

Introduction to ITK and the ITK registration framework.