deformable image registration - ulisboa
TRANSCRIPT
Deformable Image Registration
Adrien Bartoli
ALCoV – ISIT
Université d’Auvergne
Clermont-Ferrand, France
Fourth Tutorial on Computer Vision in a Nonrigid World ICCV’11, Barcelona, Spain – octobre 6, 2011
Lourdes Agapito, Adrien Bartoli, Alessio Del Bue
Deformable Image Registration
The warp 𝜑𝑖: Ω → ℝ2 is a function that transfers pixels between images
Warp visualization grid
→ To relate the content of at least two images
𝜑1 𝜑2
𝜑3 ROI Ω ∈ ℕ2
Difficulty: Variation of the Imaged Appearance
External occlusions, color Self-occlusion Field of view + lighting
Scene geometry Surface appearance Imaging conditions
• External occlusions • Self-occlusions • Wrinkles and foldings • Temporal continuity • Extensibility • Etc.
• Lighting • Lack of / repeating texture • Reflectance • Specularities • Transparency • Etc.
• Pose • Field of view • Affine vs perspective • Optical and motion blur • Auto-exposure, saturation • Etc.
Template
The Warp Function 𝜑
The warp 𝜑:Ω → ℝ2 is continuous and piecewise ‘smooth’
3D deformation Ψ ∈ 𝐶0
Warp 𝜑 ∈ 𝐶0
Image Pair Versus Video Registration
𝜑1
𝜑2 𝜑3
Video registration
Image pair registration
𝜑
min𝜑∈𝐶0ℰ[𝜑]
A variational optimization problem
ℰ is the cost functional
Computational Warp Representation
A warp 𝜑 is an interpolant between pairs of fixed/moving driving features
source target
fixed
moving
Computational Warp Representation
Encompasses the Flow-Field, Mesh Interpolation, Radial Basis Functions (and so the Thin-Plate Spline), Tensors Product (and so the Cubic B-Spline Free-Form Deformation), etc.
[Brunet et al., IJCV’11 ; Bartoli et al., IJCV’10]
The Cost Functional
Data term • Relates the warp to the image content • Bayesian likelihood
Regularization term • Measures closeness to prior knowledge • Bayesian prior
Regularization weight • Hyperparameter • Trades-off data-regularization • ≈ Bayesian noise variance
Overfitting About fine Underfitting Underfitting
Measuring Unsmoothness
This energy is Linear Least Squares (and so convex) for Linear Basis Expansion warps
Partial derivatives of order 1 and 2 are commonly used:
[Bookstein, PAMI’89]
Measuring Data Fitting
1 – Feature-based registration
• Wide-baseline
• Keeps only geometric features
• Less accurate than pixel-based
• Small-baseline
• Uses the whole image content
• More accurate than feature-based
2 –Pixel-based registration
Warp Estimation from Keypoint Matches
Property: for Linear Basis Expansion warps the above cost function is Linear Least Squares
Keypoint Detection
• Locations where the image changes significantly – Repeatable (stable over appearance variation)
– Accurately localised
– Discriminable
• Literature is massive (Harris, SIFT, SURF, etc.)
Robust Estimation
Local consistency → strong evidence of correctness 1 – Keep all locally consistent matches as strong correct matches 2 – Add matches locally consistent with strong correct matches
[Pizarro et al, IJCV, to appear]
• Global methods (RANSAC, M-estimators, etc.) are not adapted • Local methods do much better
Self-Occlusions
Current warp profile
Desired warp profile
Self-occlusion boundary
Self-occlusions
Self-occlusion
Visible
Self-Occlusions
Differential properties:
Let 𝜂:ℝ2 → ℝ give the warp’s Jacobian (𝜂 =𝜕𝜑
𝜕𝐪)
𝜂 𝐪 ≤ 0 means point q is self-occluded
𝜂 𝐪 > 0 is not conclusive
Current warp profile
Self-occlusions
Differential properties
Desired warp profile
𝜂 = 0
𝜂 ≥ 0
Self-Occlusion Estimation Results
Initial warp Partial self-occlusion map Self-occlusion resistant warp
1 2 3
Measuring Data Fitting
1 – Feature-based registration
• Wide-baseline
• Keeps only geometric features
• Less accurate than pixel-based
• Small-baseline
• Uses the whole image content
• More accurate than feature-based
2 –Pixel-based registration
Warp Estimation from the Brightness Constancy Assumption
The above cost function is Nonlinear Least Squares → Iterative local minimization (gradient descent-like)
Explicit Photometric Transformation
Video: no photometric model Video: affine photometric model
Template Image
[Bartoli, PAMI’08]
Light-Invariant Registration
Video: no photometric model
Light-invariant registration
Template Image
[Pizarro et al, SCIA’07, CVPR’08]
Robustification
Idea: inhibitate the data terms that get `too large'
The square loss function is replaced by an M-estimator ½
Self-Occlusion Detection Results
Videos (with self-occlusion): visualization grid and template with self-occluded pixels in white
Self-Occlusion Detection Results
Videos (with external and self-occlusion): visualization grid and template with self-occluded pixels in white and external occluded pixels in blue
Self-Occlusion Detection Results
Videos (with self-occlusion): visualization grid and template with self-occluded pixels in white
Registration Workflow for an Image Pair
1. Feature-based initialization
– Convex optimizations (Linear Least Squares)
– Wide baseline, less accurate
– Partial detection of self-occlusions
2. Pixel-based refinement
– Non-convex optimization
– Short baseline, more accurate
– Complete detection of self-occlusions
Registering a Video
𝜑1 𝜑2
𝜑3
The warps 𝜑1, … , 𝜑𝑛 are temporally coherent
min𝜑1,…,𝜑𝑛 ℰ𝑑[𝜑1, … , 𝜑𝑛] + 𝜆𝑠ℰ𝑠[𝜑1, … , 𝜑𝑛] +𝜆𝑡ℰ𝑡[𝜑1, … , 𝜑𝑛]
data Spatial smoothness
Temporal smoothness
Data (L1 norm)
Spatial smoothness
Temporal smoothness
Subspace Trajectory Constraints
[Garg et al, EMMCVPR’11]
𝜑𝑖 ≈ 𝑢𝑖 Frame 1
Frame 𝑖 Frame 𝑛 Fixed 2D trajectory basis
Coefficients
• PCA of sparse tracks • Low frequency
components of DCT • B-Splines
𝑢𝑖 𝐪 = B𝑖𝛼(𝐪)
min𝜑1,…,𝜑𝑛𝛼 𝐪 , 𝐪∈Ω
ℰ𝑑[𝜑𝑖]
𝑛
𝑖=1
+ 𝜆𝑡 𝑢𝑖 − 𝜑𝑖 2
𝑛
𝑖=1
+ 𝜆𝑠𝜕𝛼
𝜕𝐪1
Ω
𝐪
Subspace Trajectory Constraints
[Garg et al, EMMCVPR’11]
For 𝑖 = 1,… , 𝑛
min𝜑𝑖ℰ𝑑 𝜑𝑖 + 𝜆𝑡 𝑢𝑖 − 𝜑𝑖 2
Initialize all warps to identity
min𝛼 𝐪 , 𝐪∈Ω
𝜆𝑡 𝑢𝑖 − 𝜑𝑖 2
𝑛
𝑖=1
+ 𝜆𝑠𝜕𝛼
𝜕𝐪1
Solved point-wise
Can be solved independently for each basis
loop
1 2
1
2
This is embedded in a coarse-to-fine approach
Data (L1 norm)
Spatial smoothness
Temporal smoothness
min𝜑1,…,𝜑𝑛𝛼 𝐪 , 𝐪∈Ω
ℰ𝑑[𝜑𝑖]
𝑛
𝑖=1
+ 𝜆𝑡 𝑢𝑖 − 𝜑𝑖 2
𝑛
𝑖=1
+ 𝜆𝑠𝜕𝛼
𝜕𝐪1
→ Show « vid_flag_orig.mp4 »
Video Registration Results
Video: registration error of video registration
From left to right • Subspace trajectory constraints, PCA basis [Garg et al, EMMCVPR’11] • Subspace trajectory constraints, DCT basis [Garg et al, EMMCVPR’11] • Large Displacement Optical Flow [Brox et al, ECCV’10] • Baseline method (no motion basis) [Wedel et al, 2009] • Pairwise B-Spline pixel-based estimation [Pizarro et al, IJCV]